Industrial robot multi-station intelligent sorting and stacking method based on visual guidance

Abstract

The invention provides an industrial robot multi-station intelligent sorting and stacking method based on visual guidance, which relates to the technical field of robot control, and comprises the following steps: determining a target station by acquiring multi-station real-time state information, calculating an optimal grabbing time window by using a graph neural network state predictor, and obtaining a target grabbing time window; and the placement strategy is dynamically adjusted based on the visual image and the force sensing data in the article stacking process. According to the multi-station intelligent stacking system, the sorting efficiency in a multi-station scene can be improved, the grabbing precision of dynamic objects is enhanced, and stable and reliable intelligent stacking operation is achieved.

Description

Technical Field

[0001] The present invention relates to robot control technology, and in particular to a multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance. Background Art

[0002] Traditional industrial robot sorting and palletizing systems usually use fixed procedures to perform tasks according to preset workstation sequences and time schedules, which have some obvious defects and shortcomings.

[0003] There is a lack of a dynamic priority sorting mechanism in multi-station scenarios. Existing technologies usually adopt a fixed station order or a simple first-come, first-served strategy. They are unable to dynamically adjust the operation sequence according to the real-time station status, material backlog and production demand, resulting in uneven resource allocation and low overall system efficiency.

[0004] The grasping accuracy for moving objects is insufficient. Existing technologies often plan grasping based on static or slowly moving objects. For objects that are moving at high speeds or in a constantly changing state, it is difficult to accurately predict their trajectory and future state, resulting in a low grasping success rate. This is especially true when the objects are in varying shapes or irregularly stacked.

[0005] The palletizing process lacks intelligent self-adaptation. Traditional palletizing technology uses a pre-set, fixed placement strategy that cannot be adjusted in real time based on the actual state of the items and the stability of the stack. When dealing with items of varying materials, shapes, and weights, it cannot guarantee stacking stability and efficiency, which can easily lead to palletizing failures or stack collapse, impacting the continuity and safety of the production line. Summary of the Invention

[0006] The embodiments of the present invention provide a multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance, which can solve the problems in the prior art.

[0007] A first aspect of an embodiment of the present invention provides a multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance, comprising:

[0008] Acquire real-time status information for multiple workstations, including workstation backlogs, waiting times, and the urgency of subsequent process requirements. Establish a multi-dimensional workstation status assessment method based on this real-time status information, predict the material arrival frequency and processing time for each workstation based on historical data, and calculate a workstation priority score. Determine the target workstation based on the workstation priority score.

[0009] Acquire multi-perspective images of stacked items at the target workstation within a preset time window, analyze the position and posture changes of the items in the multi-perspective images, establish a graph neural network state predictor, and calculate the optimal grasping time window based on the graph neural network state predictor; within the optimal grasping time window, predict the position and posture information of the items at the time of grasping execution based on the movement characteristics of the items, and generate a grasping path based on the prediction;

[0010] Controlling the industrial robot to perform an object grabbing operation according to the grabbing path;

[0011] During the item palletizing process, continuous visual image sequences and force sensing data are collected. The displacement and deformation of the stack are detected based on the visual image sequences. The magnitude and distribution of the contact force during the placement process are detected based on the force sensing data, and changes in the item state are predicted. Based on the predicted results of the item state changes, compensation parameters are calculated in real time to dynamically adjust the industrial robot's placement strategy, which includes placement speed, placement posture, and point of force.

[0012] In an optional embodiment,

[0013] Acquire real-time status information for multiple workstations, including workstation backlog, waiting time, and urgency of subsequent process requirements. Establish a multi-dimensional workstation status assessment method based on this real-time status information. Calculate workstation priority scores based on historical data to predict material arrival frequency and processing time for each workstation. The steps include:

[0014] Construct a workstation relationship graph and extract inter-workstation correlation features through a graph convolutional network. The inter-workstation correlation features and the real-time status information are input into a multi-head attention mechanism fusion network to obtain global fusion features. Based on historical time series data, the Transformer model is used to predict material arrival frequency and material processing time.

[0015] Establish a multi-objective constraint evaluation system, including efficiency index, balance index and urgency index. The efficiency index is calculated based on the material processing time and the workstation backlog. The balance index is calculated by the workstation backlog deviation and the waiting time deviation. The urgency index is calculated by the urgency of the subsequent process demand and the material arrival frequency.

[0016] Establish a weight adaptive mechanism, construct a loss function through the decision effect evaluation value, and dynamically update the weight coefficient based on the gradient information of the loss function;

[0017] The efficiency index, balance index and urgency index are combined with the global fusion feature, and weighted calculation is performed with the dynamically updated weight coefficient to obtain an initial priority score. The initial priority score is multiplied by the correction term of the workstation's historical performance to obtain the workstation priority score; a score feedback correction mechanism is established based on the performance improvement index and the efficiency improvement index, and the workstation priority score is corrected by the adaptive adjustment coefficient to obtain the final workstation priority score.

[0018] In an optional embodiment,

[0019] The steps of obtaining multi-view images of stacked objects at the target workstation within a preset time window, analyzing the position and posture change patterns of the objects in the multi-view images, establishing a graph neural network state predictor, and calculating the optimal grasping time window based on the graph neural network state predictor include:

[0020] Collecting multi-view images of the target area and preprocessing them to obtain a depth confidence image; extracting a feature response map of the depth confidence image;

[0021] Calculating the spatial correlation between items based on the feature response graph, the spatial correlation is obtained by a weighted combination of feature similarity scores and depth continuity scores of overlapping areas of items, and constructing a spatial relationship graph including an item node set and an associated edge set based on the spatial correlation;

[0022] Inputting the spatial relationship graph into a graph attention mechanism network, updating the node features through node feature transformation and attention vector, performing object segmentation based on the updated node features, and obtaining segmented object regions;

[0023] Based on the segmented object area, an object state vector including position, attitude angle, velocity, angular velocity, and spatial relationship state is established, and the object state vector and the spatial relationship graph are input into a graph neural network predictor to obtain a state prediction value and prediction uncertainty at the next moment;

[0024] Calculating stability metrics of the object's posture, motion, and spatial relationship based on the state prediction value, performing confidence weighting on each stability metric in combination with the prediction uncertainty, combining the weighted stability metrics to obtain a stability score, and combining the stability score with the accessibility score and the visibility score to obtain a comprehensive decision target;

[0025] The optimal grasping time is determined based on the comprehensive decision-making target, and the time window adjustment amount and range are calculated according to the state prediction covariance matrix to obtain the optimal grasping time window centered on the optimal grasping time.

[0026] In an optional embodiment,

[0027] The steps of predicting the position and posture information of the object at the time of grasping execution according to the motion characteristics of the object within the optimal grasping time window and generating a grasping path based on the prediction include:

[0028] Based on the state prediction value and prediction uncertainty output by the graph neural network state predictor, perform a Kalman filter update to obtain a filtered item state vector and a filtered state prediction covariance matrix;

[0029] Calculating a predicted motion trajectory of the object based on the filtered object state vector and the filtered state prediction covariance matrix, and generating a target pose matrix based on the predicted motion trajectory;

[0030] Verifying the target pose matrix based on velocity, acceleration, angular acceleration, workspace, and joint angle constraints to determine a reachable target pose that satisfies the constraints;

[0031] A fifth-order polynomial trajectory planning method is used to generate a position path point sequence, wherein the polynomial coefficients of the fifth-order polynomial trajectory planning method are obtained by solving the start and end positions, velocity and acceleration boundary conditions, and a spherical linear interpolation method is used to generate a posture path point sequence;

[0032] Calculating the motion execution time from the current position to the achievable target pose based on the position path point sequence and the posture path sequence, and determining the grasping execution time in combination with the clock synchronization error and the execution delay;

[0033] At the grasping execution moment, the position and posture errors during the path execution process are calculated, the correction amount is calculated based on the position and posture errors, and the grasping path is compensated and adjusted in real time, and the robotic arm is driven to execute the compensated grasping path.

[0034] In an optional embodiment,

[0035] The steps of detecting the displacement and deformation of the stack according to the visual image sequence, detecting the magnitude and distribution of the contact force during the placement process according to the force sensing data, and predicting the change in the state of the item include:

[0036] Adaptively meshing the surface of the object according to the object's surface curvature, and calculating the mesh density using the curvature gradient; establishing a local coordinate system based on the principal curvature direction based on the adaptive mesh, analyzing changes in mesh node positions based on the visual image sequence, projecting the object's surface deformation onto the principal curvature direction and decomposing the deformation pattern, calculating a deformation gradient tensor and a strain tensor including the displacement field, and obtaining deformation data representing the stacking displacement and deformation;

[0037] Extracting visual features from the visual image sequence and force features from the force sensing data, the visual features including object contours, surface textures, and correspondences between feature points between adjacent frames, and the force features including contact point positions and corresponding normal and tangential force components, performing intra-modal attention calculations on the visual features and force features to obtain intra-modal features, and performing cross-modal attention calculations on the visual features and force features to obtain inter-modal features; calculating information entropy and signal-to-noise ratio based on the visual image sequence and force sensing data, and determining dynamic fusion weights for the visual and force features based on the information entropy and signal-to-noise ratio;

[0038] The intra-modal features, inter-modal features and dynamic fusion weights are input into a temporal attention memory network, and the temporal attention weights are obtained based on the historical state information; the visual features, force features and historical state features are weightedly fused according to the dynamic fusion weights and the temporal attention weights to obtain multimodal fusion features; and the state change of the object is predicted based on the multimodal fusion features and the deformation data.

[0039] In an optional embodiment,

[0040] Based on the prediction results of the object state change, the compensation parameters are calculated in real time to dynamically adjust the placement strategy of the industrial robot. The placement strategy includes the steps of placement speed, placement posture and force point.

[0041] Establishing an item state prediction matrix based on the prediction results of the item state change, wherein the item state prediction matrix includes real-time state vectors of multiple time steps, wherein the real-time state vectors include position and posture components, velocity components, and torque components; calculating a risk index based on the item state prediction matrix, wherein the risk index is obtained by weighted calculation of a displacement evaluation index, a strain evaluation index, and a force distribution evaluation index;

[0042] Generating a compensation parameter vector based on the risk index; constructing a reinforcement learning state space, wherein the reinforcement learning state space includes a current real-time state vector, the risk index, and historical execution errors; and constructing a dual network structure of a policy network and a value network based on the reinforcement learning state space;

[0043] The placement strategy is comprehensively optimized through the dual network structure, a velocity compensation strategy is generated according to the velocity component and the risk index, predictive posture compensation is performed on the robot end effector based on the position posture component and the compensation parameter vector, and predictive adjustment is performed on the position of the force point and the contact force using the torque component and the force control compensation amount in the compensation parameter vector;

[0044] The real-time state vector, action vector, reward value and next state vector during the execution process are stored in a priority experience pool; based on the sample data in the priority experience pool, a priority sampling method is used to update the parameters of the policy network and the weight coefficient of the risk index to adaptively optimize the placement strategy.

[0045] In an optional embodiment,

[0046] The steps of overall optimizing the placement strategy through the dual network structure include:

[0047] Construct a policy network based on the attention mechanism and a value network based on distributed evaluation. The policy network includes a multi-head self-attention module and a cross-attention module, and the value network includes a quantile regression module and a risk perception module.

[0048] The multi-head self-attention module performs weight mapping on the state features to generate a state representation vector; the cross-attention module performs feature interaction on the state representation vector and the historical action sequence to generate an action attention matrix;

[0049] constructing an action distribution based on the action attention matrix and the risk index, wherein the action distribution includes a speed strategy distribution, a posture strategy distribution, and a force control strategy distribution; and dynamically adjusting the exploration-exploitation balance according to the risk level by using a soft strategy optimization method with adaptive temperature parameter adjustment.

[0050] The state representation vector is input into the quantile regression module to generate a value distribution; the risk index is input into the risk perception module to calculate a risk weight coefficient; the action values ​​are evaluated and ranked based on the value distribution and the risk weight coefficient, and the optimal placement strategy is selected.

[0051] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0055] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0056] The present invention realizes the intelligent scheduling and allocation of multi-station resources through a workstation priority evaluation mechanism based on multi-dimensional real-time status information, effectively solves the problem that the traditional fixed-order sorting method cannot cope with the dynamic production environment, and significantly improves the overall operating efficiency and production flexibility of the system.

[0057] The present invention uses a graph neural network state predictor to analyze the dynamic changes in the stacking state of objects, accurately predict the optimal grasping time window and the object state at the execution moment, and solves the technical problems of low grasping accuracy and poor stability of traditional visual systems for moving objects, greatly improving the success rate of grasping irregularly stacked and high-speed moving objects.

[0058] The present invention innovatively combines visual image sequences and force sensing data to predict item status and dynamically adjust placement strategies, achieving adaptive palletizing of items of different shapes and materials, effectively preventing stacking and item damage. At the same time, the intelligent compensation parameter calculation mechanism ensures the stability and efficiency of high-density palletizing, with strong adaptability and high reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance according to an embodiment of the present invention;

[0060] Figure 2 Schematic diagram of performance comparison analysis for dual network placement strategy optimization. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0063] Figure 1 FIG. 1 is a flow chart of a multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] Acquire real-time status information for multiple workstations, including workstation backlogs, waiting times, and the urgency of subsequent process requirements. Establish a multi-dimensional workstation status assessment method based on this real-time status information, predict the material arrival frequency and processing time for each workstation based on historical data, and calculate a workstation priority score. Determine the target workstation based on the workstation priority score.

[0065] Acquire multi-perspective images of stacked items at the target workstation within a preset time window, analyze the position and posture changes of the items in the multi-perspective images, establish a graph neural network state predictor, and calculate the optimal grasping time window based on the graph neural network state predictor; within the optimal grasping time window, predict the position and posture information of the items at the time of grasping execution based on the movement characteristics of the items, and generate a grasping path based on the prediction;

[0066] Controlling the industrial robot to perform an object grabbing operation according to the grabbing path;

[0067] During the item palletizing process, continuous visual image sequences and force sensing data are collected. The displacement and deformation of the stack are detected based on the visual image sequences. The magnitude and distribution of the contact force during the placement process are detected based on the force sensing data, and changes in the item state are predicted. Based on the predicted results of the item state changes, compensation parameters are calculated in real time to dynamically adjust the industrial robot's placement strategy, which includes placement speed, placement posture, and point of force.

[0068] In an optional embodiment, real-time status information of multiple workstations is obtained, including the workstation backlog, waiting time, and urgency of subsequent process requirements. Based on the real-time status information, a multi-dimensional workstation status assessment method is established to predict the material arrival frequency and processing time of each workstation based on historical data. The steps of calculating the workstation priority score include:

[0069] The real-time status information includes the workstation backlog, waiting time, and urgency of subsequent process requirements;

[0070] Construct a workstation relationship graph and extract inter-workstation correlation features through a graph convolutional network. The inter-workstation correlation features and the real-time status information are input into a multi-head attention mechanism fusion network to obtain global fusion features. Based on historical time series data, the Transformer model is used to predict material arrival frequency and material processing time.

[0071] Establish a multi-objective constraint evaluation system, including efficiency index, balance index and urgency index. The efficiency index is calculated based on the material processing time and the workstation backlog. The balance index is calculated by the workstation backlog deviation and the waiting time deviation. The urgency index is calculated by the urgency of the subsequent process demand and the material arrival frequency.

[0072] Establish a weight adaptive mechanism, construct a loss function through the decision effect evaluation value, and dynamically update the weight coefficient based on the gradient information of the loss function;

[0073] The efficiency index, balance index and urgency index are combined with the global fusion feature, and weighted calculation is performed with the dynamically updated weight coefficient to obtain an initial priority score. The initial priority score is multiplied by the correction term of the workstation's historical performance to obtain the workstation priority score; a score feedback correction mechanism is established based on the performance improvement index and the efficiency improvement index, and the workstation priority score is corrected by the adaptive adjustment coefficient to obtain the final workstation priority score.

[0074] For example, real-time status information for workstations can be obtained, including the workstation backlog, waiting time, and the urgency of subsequent process requirements. For example, in a manufacturing plant environment, workstation A currently has a backlog of 15 pending items, the oldest item has been waiting for 35 minutes, and the urgency of the subsequent workstation B is 0.85 (out of a maximum score of 1).

[0075] A workstation relationship graph is constructed to represent the process flow relationship between each workstation. Assume that in a manufacturing environment, there are workstations A, B, C, D, and E, where materials from workstation A can flow to workstations B and C, materials from workstation B flow to workstation D, and materials from workstations C and D flow to workstation E. The workstation relationship is represented by an adjacency matrix. If workstation i has a direct material flow relationship with workstation j, the corresponding position in the matrix is ​​1, otherwise it is 0. Based on the constructed workstation relationship graph, the correlation features between workstations are extracted through a graph convolutional network. The graph convolutional network contains two convolutional layers, each consisting of 32 convolution kernels, and uses ReLU as the activation function. For example, for the relationship graph of the five workstations mentioned above, the extracted correlation feature dimension is a 32-dimensional feature vector for each workstation, representing the structural characteristics of the workstation in the entire production network.

[0076] The multi-head attention fusion network uses eight attention heads, each with an output dimension of 64. Ultimately, it generates a 512-dimensional global fusion feature vector that contains both workstation association and status information. For example, the global fusion feature for workstation A not only includes its own backlog and wait time information, but also incorporates the status influence of its connected workstations B and C.

[0077] The Transformer model predicts material arrival frequency and material processing time based on historical time series data. The Transformer model uses a 6-layer encoder structure, 8 attention heads, and an internal dimension of 512. It takes into account the past 30 days of workstation operation data. For workstation A, the model predicts a material arrival frequency of 12 per hour over the next four hours, with an average material processing time of 4.5 minutes.

[0078] The efficiency index is calculated based on material processing time and workstation backlog. For example, workstation A currently has a backlog of 15 materials, each processing time is 4.5 minutes, and the total processing time is 67.5 minutes. The efficiency index is 0.72. The balance index is calculated by the backlog deviation and waiting time deviation. Assuming the average backlog for the entire production line is 10 materials, the average waiting time is 25 minutes, the backlog deviation at workstation A is 5, and the waiting time deviation is 10 minutes, the calculated balance index value is 0.65. The urgency index is calculated by the urgency of the subsequent process demand and the frequency of material arrival. The demand urgency of the subsequent process B at workstation A is 0.85, and the material arrival frequency is 12 per hour, resulting in an urgency index value of 0.78.

[0079] A weight-adaptive mechanism was established to dynamically adjust the weights of efficiency, balance, and urgency indicators. Initial weights were set to 0.4, 0.3, and 0.3, respectively. The overall efficiency improvement of the production line after each decision was recorded, a loss function was constructed, and the weight coefficients were updated using gradient descent. For example, if the efficiency improvement of the production line was not significant after a priority score, the weight of the efficiency indicator was adjusted to 0.45, and the weights of the balance and urgency indicators were adjusted to 0.25 and 0.3, respectively.

[0080] The efficiency, balance, and urgency indicators are combined with the global fusion features and mapped to a 128-dimensional space using a fully connected network. The results are then weighted with the dynamic weight coefficients to generate an initial priority score. For example, if workstation A has three indicators of 0.72, 0.65, and 0.78, respectively, with weights of 0.45, 0.25, and 0.3, the calculated initial priority score is 0.722.

[0081] The historical performance modifier is based on the workstation's processing efficiency over the past seven days. If the workstation's historical performance is good, the modifier is 1.1; if it's average, the modifier is 1.0; if it's poor, the modifier is 0.9. Assuming workstation A has good historical performance and the modifier is 1.1, the priority score is 0.722 × 1.1 = 0.7942.

[0082] In establishing a scoring feedback correction mechanism, the performance improvement index is calculated based on the percentage increase in the workstation's production capacity after the priority decision, and the efficiency improvement index is calculated based on the percentage increase in the efficiency of the entire production line. Assuming that after prioritizing workstation A, its production capacity increased by 15%, and the efficiency of the entire production line increased by 8%, the adaptive adjustment coefficient is calculated to be 1.05. The final workstation priority score is obtained by multiplying the workstation priority score by the adaptive adjustment coefficient. The final priority score for workstation A is 0.7942 × 1.05 = 0.8339.

[0083] All workstation priority scores are calculated and sorted, with workstations with higher priority scores receiving higher resource allocation priority. In practical applications, when resource competition arises on a production line—for example, when resources such as manpower, equipment, or materials are insufficient to simultaneously meet the needs of all workstations—the final priority score determines the order of resource allocation, thereby achieving overall optimization of production efficiency.

[0084] The present invention constructs a workstation relationship graph and combines it with a graph convolutional network to extract the correlation features between workstations. It also cooperates with a multi-head attention mechanism to fuse the network and the Transformer model to establish a multi-objective constraint evaluation system that includes efficiency, balance, and urgency indicators. It also designs a weight adaptive mechanism to dynamically update the weight coefficient, thereby achieving accurate assessment of workstation priority and improving the overall operational efficiency of sorting and the rationality of resource allocation.

[0085] In an optional embodiment, obtaining multi-view images of stacked objects at the target workstation within a preset time window, analyzing the position and posture change patterns of the objects in the multi-view images, establishing a graph neural network state predictor, and calculating the optimal grasping time window based on the graph neural network state predictor includes the following steps:

[0086] Collecting multi-view images of the target area and preprocessing them to obtain a depth confidence image; extracting a feature response map of the depth confidence image;

[0087] Calculating the spatial correlation between items based on the feature response graph, the spatial correlation is obtained by a weighted combination of feature similarity scores and depth continuity scores of overlapping areas of items, and constructing a spatial relationship graph including an item node set and an associated edge set based on the spatial correlation;

[0088] Inputting the spatial relationship graph into a graph attention mechanism network, updating the node features through node feature transformation and attention vector, performing object segmentation based on the updated node features, and obtaining segmented object regions;

[0089] Based on the segmented object area, an object state vector including position, attitude angle, velocity, angular velocity, and spatial relationship state is established, and the object state vector and the spatial relationship graph are input into a graph neural network predictor to obtain a state prediction value and prediction uncertainty at the next moment;

[0090] Calculating stability metrics of the object's posture, motion, and spatial relationship based on the state prediction value, performing confidence weighting on each stability metric in combination with the prediction uncertainty, combining the weighted stability metrics to obtain a stability score, and combining the stability score with the accessibility score and the visibility score to obtain a comprehensive decision target;

[0091] The optimal grasping time is determined based on the comprehensive decision-making target, and the time window adjustment amount and range are calculated according to the state prediction covariance matrix to obtain the optimal grasping time window centered on the optimal grasping time.

[0092] For example, when obtaining multi-perspective images of stacked items at a target workstation within a preset time window, three high-definition cameras can be used to capture image sequences of the target area from different angles. These cameras are installed at 45 degrees above the workstation, 60 degrees to the side, and 30 degrees in front of the workstation. The camera resolution is 1920×1080 pixels, the frame rate is 60fps, the acquisition duration is 5 seconds, and a total of 300 frames of multi-perspective images are acquired. After the acquisition is completed, the image is preprocessed, including distortion correction, color balancing, and lighting uniformity processing. A structured light depth camera is used to calculate the scene depth information and generate a depth confidence image with a resolution of 640×480. Each pixel contains a depth value and its confidence score.

[0093] To extract depth confidence image features, a ResNet-50 network architecture is used, with the input being a preprocessed RGB-D image (three RGB channels plus a depth channel). After forward propagation, the network extracts multi-scale feature maps from the third and fourth convolutional blocks and fuses them through a feature pyramid network to produce a 256-channel feature response map that is one-quarter the size of the original image. This feature response map preserves the object's shape, texture, and spatial position, providing the foundation for subsequent spatial correlation calculations.

[0094] The spatial correlation between items is calculated using a dual-index evaluation method. For the stacking area, the feature vectors of two adjacent items are extracted, and the cosine similarity is calculated to obtain the feature similarity score. At the same time, the continuity of the depth map at the object boundary is evaluated, and the smoothness of the depth gradient is calculated to obtain the depth continuity score. The feature similarity score and the depth continuity score are weighted and combined in a ratio of 0.6:0.4 to obtain the final spatial correlation. Based on this correlation, a spatial relationship graph G = (V, E) is constructed, where the node set V represents the detected items and the edge set E represents the spatial correlation between items. When the spatial correlation between two items exceeds the threshold of 0.75, a connecting edge is established between the nodes, and the edge weight is the corresponding spatial correlation value.

[0095] The constructed spatial relationship graph is input into a graph attention mechanism network, which consists of three layers of graph convolutional layers, with output dimensions of 128, 64, and 32, respectively. For each node, its representation is updated by aggregating the feature information of its neighboring nodes. The attention vector is calculated by passing the node features through two fully connected layers (with dimensions of 64 and 1) and is used to determine the influence weight of the neighboring nodes on the central node. After three rounds of message passing, the node features contain local spatial relationship information. Based on the updated node features, a dense prediction network is used to generate object segmentation masks with a mask resolution of the original image size.

[0096] For each segmented object, a 15-dimensional state vector is constructed, consisting of position coordinates (x, y, z), Euler angles (α, β, γ), linear velocities (vx, vy, vz), angular velocities (ωα, ωβ, ωγ), and spatial relationship variables (contact area, contact angle, and relative pressure). The input layer of the predictor consists of two branches: a state encoding branch and a graph structure branch. The state encoding branch uses a three-layer MLP (64-128-64) to map the object state vector into a high-dimensional feature space. Simultaneously, the graph structure branch utilizes an edge-conditioned GNN to process the spatial relationship graph, performing three rounds of message passing. In each round of message passing, node information is updated interactively with neighboring nodes via the EdgeConv operator, capturing the dynamic coupling between objects. Features from these two branches are fused using an attention mechanism to generate context-aware features. The fused features are fed into a prediction head consisting of a temporal convolutional network and a global attention pooling layer. Temporal convolution extracts short-term dynamic features, while attention pooling performs weighted aggregation of global information. Finally, the dual-output fully connected layer predicts the mean and covariance matrix of the state vector at the next time step (t+100ms), respectively. The covariance matrix represents the uncertainty of the prediction. The predictor is trained using a combination of loss functions, including the mean square error (MSE) loss for state prediction, the negative log-likelihood loss for uncertainty estimation, and the temporal consistency loss, ensuring accurate predictions even in scenarios with object interactions and occlusions.

[0097] When calculating the stability metric for an item based on its predicted state, it is evaluated from three dimensions: posture stability, motion stability, and spatial relationship stability. Posture stability is calculated using the rate of change of the posture angle and the center of gravity offset; motion stability is assessed based on the changing trends of velocity and acceleration; and spatial relationship stability considers the relative position changes between adjacent items. Each stability metric is multiplied by its corresponding confidence weight (derived from the inverse calculation of the prediction covariance matrix) and then combined in a ratio of 0.4:0.3:0.3 to produce a comprehensive stability score ranging from 0 to 1. The accessibility score is obtained through simulation of the robot's motion planning, and the visibility score is calculated based on the visible proportion of the item's surface area in multi-view images. The three scores are combined in a weighted ratio of 0.5:0.3:0.2 to form a comprehensive decision-making target.

[0098] The optimal grasping time is determined using a sliding time window evaluation method. In the forecast sequence, the comprehensive decision target value is evaluated at each time point with a step size of 50ms. The time point with the highest decision target value is determined as the optimal grasping time (denoted as t_opt). To account for forecast uncertainty, the time window adjustment is calculated based on the eigenvalues ​​of the state forecast covariance matrix, typically ±150ms. The final optimal grasping time window is [t_opt - 150ms, t_opt + 150ms].

[0099] Existing dynamic grasping technologies are usually based on single-view images and simple motion models to predict object trajectories. When faced with complex stacked objects, these methods have a significantly reduced grasping success rate due to the lack of modeling of the spatial relationship between objects and accurate state prediction capabilities, especially when objects move quickly or occlude each other. The present invention introduces multi-view image acquisition and deep confidence analysis, constructs a method for calculating the spatial correlation between objects, and solves the problem of accurate segmentation of stacked objects; innovatively applies graph neural networks as a state predictor, captures the dynamic coupling relationship between objects through a message passing mechanism, and achieves high-precision prediction of the state of objects in complex environments; designs a multi-dimensional comprehensive decision-making target that includes stability, accessibility, and visibility, combines prediction uncertainty for confidence weighting, and determines the optimal grasping time window, which significantly improves the grasping success rate in dynamic environments.

[0100] In an optional embodiment, within the optimal grasping time window, predicting the position and posture information of the object at the grasping execution time according to the motion characteristics of the object, and generating a grasping path based on the prediction includes:

[0101] Based on the state prediction value and prediction uncertainty output by the graph neural network state predictor, perform a Kalman filter update to obtain a filtered item state vector and a filtered state prediction covariance matrix;

[0102] Calculating a predicted motion trajectory of the object based on the filtered object state vector and the filtered state prediction covariance matrix, and generating a target pose matrix based on the predicted motion trajectory;

[0103] Verifying the target pose matrix based on velocity, acceleration, angular acceleration, workspace, and joint angle constraints to determine a reachable target pose that satisfies the constraints;

[0104] A fifth-order polynomial trajectory planning method is used to generate a position path point sequence, wherein the polynomial coefficients of the fifth-order polynomial trajectory planning method are obtained by solving the start and end positions, velocity and acceleration boundary conditions, and a spherical linear interpolation method is used to generate a posture path point sequence;

[0105] Calculating the motion execution time from the current position to the achievable target pose based on the position path point sequence and the posture path sequence, and determining the grasping execution time in combination with the clock synchronization error and the execution delay;

[0106] At the grasping execution moment, the position and posture errors during the path execution process are calculated, the correction amount is calculated based on the position and posture errors, and the grasping path is compensated and adjusted in real time, and the robotic arm is driven to execute the compensated grasping path.

[0107] Exemplarily, after determining the optimal grasping time window, a Kalman filter update is performed based on the state prediction value and prediction uncertainty output by the graph neural network state predictor. The state vector of the Kalman filter contains 15 components, which are the same as the aforementioned object state vector, including position coordinates (x, y, z), Euler angles (α, β, γ), linear velocity (vx, vy, vz), angular velocity (ωα, ωβ, ωγ), and spatial relationship variables such as contact area, contact angle, and relative pressure. The state transfer matrix is ​​constructed using a physical kinematic model, taking into account the influence of velocity on position, the influence of angular velocity on posture angle, and the interaction between adjacent objects.

[0108] The Kalman filter update process consists of a prediction step and a correction step. The prediction step uses the previous state estimate and state transition matrix to predict the current state. The correction step fuses the prediction result with the output of the graph neural network predictor. The fusion weights are dynamically adjusted based on their respective uncertainties, and the covariance matrix of the graph neural network predictor is directly used as the measurement noise covariance. After the Kalman filter update, the filtered object state vector is obtained, whose uncertainty is reduced by approximately 30% compared to the original prediction, improving the position prediction accuracy to the millimeter level and the attitude prediction accuracy to the angle level.

[0109] Based on the filtered object state vector and the state prediction covariance matrix, the predicted trajectory of the object over the next 500ms is calculated. In the implementation, the state transition equation is iteratively applied at 20ms intervals to generate a state prediction sequence of 25 time points. For each predicted time point, a 4×4 pose matrix of the target object is calculated. This matrix consists of a rotation matrix and a translation vector. The rotation matrix is ​​converted from Euler angles, and the translation vector is the position coordinate. For example, if the current position of the target object is (120mm, 85mm, 30mm), the attitude angle is (15°, 30°, 45°), the linear velocity is (5mm / s, 8mm / s, 0mm / s), and the angular velocity is (0° / s, 0° / s, 10° / s), it is predicted that the object's position will be (120.5mm, 85.8mm, 30mm) and the attitude angle will be (15°, 30°, 46°) 100ms later.

[0110] The generated target pose matrix is ​​constrained and verified to ensure that the robot arm's motion capabilities are met. Constraint verification includes the following aspects: the speed constraint limits the maximum linear velocity of the end effector to no more than 500mm / s, and the angular velocity of each joint to no more than 180° / s; the acceleration constraint limits the maximum linear acceleration of the end effector to no more than 2000mm / s 2 , the angular acceleration of each joint does not exceed 360° / s 2 ; Angular acceleration constraint limits joint angular acceleration to no more than 720° / s 3; The workspace constraint ensures that the target position is within the working radius of the robot arm (such as 800mm) and is not near the singularity; the joint angle constraint ensures that each joint angle is within the allowable range (such as ±170°).

[0111] Constraint verification adopts a step-by-step screening method. First, the workspace constraints are verified and the postures that exceed the working range are eliminated; then the inverse kinematics algorithm is used to calculate the joint angles and verify the joint angle constraints; finally, the movement process of the robot arm from the current position to the target posture is simulated to check the speed, acceleration and angular acceleration constraints. For example, when the predicted object posture is position (650mm, 420mm, 150mm) and attitude (30°, 45°, 60°), the 6 joint angles are calculated to be (25°, 40°, -30°, 65°, 35°, 80°). All joint angles are within the limit range, and the maximum joint speed during the movement is 95° / s, and the maximum joint acceleration is 290° / s. 2 , all satisfy the constraints, so this pose is determined to be the reachable target pose.

[0112] In order to generate a smooth motion trajectory, a fifth-order polynomial trajectory planning method is used. For the position trajectory, the three coordinate components x, y, and z are planned separately. The fifth-order polynomial expression contains 6 coefficients, and the polynomial coefficients are solved by setting the six boundary conditions of the starting and ending positions, velocity, and acceleration. For example, the starting position of the x-axis is 100mm, the velocity is 0mm / s, and the acceleration is 0mm / s. 2 ; The end position is 200mm, the speed is 0mm / s, and the acceleration is 0mm / s 2 The movement time is 2 seconds. The coefficients of the solved quintic polynomial are [100, 0, 0, 37.5, -18.75, 3.125], which means the trajectory equation is x(t) = 100 + 37.5t 3 -18.75t 4 +3.125t 5 , where t∈[0, 2].

[0113] For the pose trajectory, the pose expressed in Euler angles is first converted to a quaternion representation. Then, the spherical linear interpolation (SLERP) method is used to generate a pose pathpoint sequence. For example, if the starting pose quaternion is (0.9239, 0.1678, 0.2278, 0.2586) and the ending pose quaternion is (0.8660, 0.0868, 0.2165, 0.4419), 20 interpolation points are generated, of which the quaternion of the fifth interpolation point is (0.9131, 0.1487, 0.2244, 0.3025). This method ensures smoothness and the shortest path characteristics of the pose transformation process.

[0114] Based on the generated position path point sequence and posture path point sequence, the motion execution time from the current position to the achievable target posture is calculated. The calculation method considers three factors: path length, maximum speed limit, and maximum acceleration limit. For a path with a total length of 300mm, the maximum speed limit is 200mm / s and the maximum acceleration limit is 500mm / s. 2 The calculated minimum execution time is 2.14 seconds. Furthermore, clock synchronization errors (typically ±5ms) and execution delays (e.g., controller processing delay of 15ms and mechanical response delay of 25ms) are also considered. These errors and delays are added to the execution time to determine the final grasp execution time.

[0115] At the moment of grasp execution, the position and attitude errors during path execution are continuously calculated. The position error is calculated by comparing the actual position of the robot end effector (calculated by forward kinematics) with the target position of the planned path point; the attitude error is calculated by calculating the angular difference between the actual attitude quaternion and the target attitude quaternion. When a position error exceeding 5mm or an attitude error exceeding 3° is detected, a correction is calculated based on the current error. The correction is calculated using a proportional-differential control strategy with a proportional coefficient set to 0.8 and a differential coefficient set to 0.2. For example, when the x-axis position error is 8mm and the rate of change is 2mm / s, the calculated x-axis correction is 8×0.8+2×0.2=6.8mm.

[0116] The calculated correction is applied to subsequent waypoints, compensating the grasping path in real time. This compensation uses a progressive strategy, distributing the correction exponentially attenuating the correction across the next 500ms to avoid mechanical vibration caused by sudden changes. Finally, the adjusted waypoints are converted to target angles in joint space, and the controller drives the robotic arm to execute the compensated grasping path.

[0117] This method uses a graph neural network state predictor combined with Kalman filtering technology to accurately predict object states. Constraint verification ensures the reachability of the target pose. It uses quintic polynomial trajectory planning and spherical linear interpolation to generate a smooth path. Furthermore, it incorporates a real-time compensation adjustment mechanism to dynamically correct the grasping path. This comprehensive and optimized grasping execution method significantly improves the accuracy and stability of grasping moving objects.

[0118] In an optional embodiment, detecting the displacement and deformation of the stack based on the visual image sequence, detecting the magnitude and distribution of the contact force during placement based on the force sensing data, and predicting the change in the state of the item includes:

[0119] Adaptively meshing the surface of the object according to the object's surface curvature, and calculating the mesh density using the curvature gradient; establishing a local coordinate system based on the principal curvature direction based on the adaptive mesh, analyzing changes in mesh node positions based on the visual image sequence, projecting the object's surface deformation onto the principal curvature direction and decomposing the deformation pattern, calculating a deformation gradient tensor and a strain tensor including the displacement field, and obtaining deformation data representing the stacking displacement and deformation;

[0120] Extracting visual features from the visual image sequence and force features from the force sensing data, the visual features including object contours, surface textures, and correspondences between feature points between adjacent frames, and the force features including contact point positions and corresponding normal and tangential force components, performing intra-modal attention calculations on the visual features and force features to obtain intra-modal features, and performing cross-modal attention calculations on the visual features and force features to obtain inter-modal features; calculating information entropy and signal-to-noise ratio based on the visual image sequence and force sensing data, and determining dynamic fusion weights for the visual and force features based on the information entropy and signal-to-noise ratio;

[0121] The intra-modal features, inter-modal features and dynamic fusion weights are input into a temporal attention memory network, and the temporal attention weights are obtained based on the historical state information; the visual features, force features and historical state features are weightedly fused according to the dynamic fusion weights and the temporal attention weights to obtain multimodal fusion features; and the state change of the object is predicted based on the multimodal fusion features and the deformation data.

[0122] For example, adaptive meshing based on the curvature of the object surface is the basis of deformation detection. Collect three-dimensional point cloud data of the object surface and calculate the principal curvature and curvature direction of each point. For areas with larger curvature (such as areas with curvature values ​​greater than 0.5), the mesh density is set higher and the mesh side length is 3mm; for areas with gentle curvature (such as areas with curvature values ​​less than 0.1), the mesh density can be reduced and the mesh side length is expanded to 10mm. The curvature gradient (i.e., the curvature change rate) is used to control the smooth transition of the mesh density change. When the curvature gradient exceeds the threshold of 0.2, the change rate of adjacent mesh sizes is controlled within 1.5 to ensure mesh quality. This meshing method can provide a more detailed depiction of the deformation of the object surface while ensuring computational efficiency.

[0123] For each grid node on the surface of the object, the direction of the maximum principal curvature of the point is taken as the x-axis, the direction of the minimum principal curvature is taken as the y-axis, and the normal vector is taken as the z-axis to form a right-handed coordinate system. After the visual image sequence is collected, the position changes of the grid nodes between adjacent frames are tracked by feature matching technology. For areas with rich textures, SIFT or ORB feature points are extracted with an error within 1.5 pixels; for areas with less texture, the optical flow method is used for tracking, and the average tracking accuracy can reach 2.2 pixels. The node displacement is projected into the local coordinate system, and the tensile / compressive deformation component in the direction of the principal curvature and the tangential shear deformation component can be decomposed. The deformation gradient tensor is calculated based on the displacement field, and the deformation data characterizing the stacking displacement and deformation are obtained through the length ratio and angle change of the grid units before and after deformation.

[0124] In terms of visual features, the object outline (using the Canny algorithm), surface texture (using LBP or Gabor filters, with a texture feature dimension of 128), and the correspondence between feature points between adjacent frames (using the KLT tracking algorithm, with 256 feature points) are extracted from the visual image sequence. In terms of force features, the contact point position and the corresponding normal force (measuring range 0-50N, resolution 0.01N) and tangential force component (measuring range 0-25N, resolution 0.01N) are collected through a six-dimensional force / torque sensor installed at the end of the robotic arm. The force data collected at each time step constitutes a force feature vector with a dimension of 18 (3 contact points × 6 degrees of freedom).

[0125] In visual features, a self-attention mechanism is used to calculate correlations between different feature regions. An attention matrix is ​​constructed for 256 feature points, with weights assigned based on correlation, retaining the top 20% of strongly correlated connections. Force features utilize self-attention within a time window (with a window size of 10 frames) to highlight force variation patterns at key moments. The resulting intra-modal features effectively capture spatiotemporal dependencies within a single modality.

[0126] Intermodal feature fusion is achieved through a cross-modal attention mechanism. A bidirectional attention mapping is constructed, from vision to force and from force to vision, to calculate the correlation distribution of the two modal features. For example, when a significant deformation (deformation exceeding 3mm) is detected in a certain area of ​​the object's surface, the force data weight corresponding to that area is automatically increased. Conversely, when an abnormal contact force (exceeding a preset threshold of 15N) is detected, the visual feature weight of the corresponding contact area is increased. This bidirectional mapping can mine the complementary information between the two modalities.

[0127] Dynamic fusion weight calculation based on information entropy and signal-to-noise ratio ensures the adaptability of feature fusion. The information entropy (typical values ​​between 4.5-6.2) and signal-to-noise ratio (typical values ​​between 12-25dB) of visual features, as well as the information entropy (typical values ​​between 2.8-4.5) and signal-to-noise ratio (typical values ​​between 15-30dB) of force features are calculated. When visual conditions are poor (such as the signal-to-noise ratio is lower than 15dB), the weight of the force feature can be increased to 0.7; when the force sensing data is unstable (such as the signal-to-noise ratio is lower than 18dB), the weight of the visual feature can be increased to 0.8. This dynamic adjustment mechanism ensures robustness under different environmental conditions.

[0128] The temporal attention memory network consists of 64 LSTM units and a temporal attention layer, capable of storing state information for the past 30 time steps. It calculates the correlation between the current state and the historical state and generates temporal attention weights. For example, when an object is detected to be beginning to deform, the attention weight for the initial deformation phase is increased (up to 0.4) to capture the deformation trend.

[0129] The final state prediction is achieved through weighted fusion of multimodal features and deformation data. Based on the dynamic fusion weights and temporal attention weights, the visual features, force features, and historical state features are weighted and summed to produce a multimodal fusion feature vector of dimension 256. This feature vector, along with the deformation data, is input into a prediction network consisting of three fully connected layers (with 128, 64, and 32 nodes, respectively), which outputs the position offset and deformation prediction of the object at the next time step.

[0130] Existing technologies typically use fixed-density grid partitioning schemes and basic convolutional networks to process perception data. These methods lack adaptability to objects of different materials and are unable to effectively handle the complementary and redundant relationships between multimodal data, resulting in insufficient prediction accuracy in complex environments. In particular, this method results in inaccurate state assessment during the placement of deformable objects or precision components. The present invention introduces a curvature-based adaptive grid partitioning technique, establishing a local coordinate system along the principal curvature direction to accurately describe the surface deformation of objects. It designs intra-modal attention and cross-modal attention mechanisms to effectively fuse visual and force features, dynamically adjusting fusion weights based on information entropy and signal-to-noise ratio to address the reliability differences between different perception modalities in different scenarios. It introduces a temporal attention memory network to correlate and analyze the current state with historical information, capturing the temporal dependencies of object state changes. This achieves high-precision multimodal perception and prediction of object state changes, providing a reliable basis for adjusting placement strategies.

[0131] In an optional embodiment, based on the prediction result of the object state change, the compensation parameters are calculated in real time to dynamically adjust the placement strategy of the industrial robot, wherein the placement strategy includes the steps of placement speed, placement posture and force point.

[0132] Establishing an item state prediction matrix based on the prediction results of the item state change, wherein the item state prediction matrix includes real-time state vectors of multiple time steps, wherein the real-time state vectors include position and posture components, velocity components, and torque components; calculating a risk index based on the item state prediction matrix, wherein the risk index is obtained by weighted calculation of a displacement evaluation index, a strain evaluation index, and a force distribution evaluation index;

[0133] Generating a compensation parameter vector based on the risk index, the compensation parameter vector including position compensation, posture compensation, and force control compensation; constructing a reinforcement learning state space, the reinforcement learning state space including the current real-time state vector, the risk index, and historical execution errors; and constructing a dual network structure of a policy network and a value network based on the reinforcement learning state space;

[0134] The placement strategy is comprehensively optimized through the dual network structure, a velocity compensation strategy is generated according to the velocity component and the risk index, predictive posture compensation is performed on the robot end effector based on the position posture component and the compensation parameter vector, and predictive adjustment is performed on the position of the force point and the contact force using the torque component and the force control compensation amount in the compensation parameter vector;

[0135] The real-time state vector, action vector, reward value and next state vector during the execution process are stored in a priority experience pool; based on the sample data in the priority experience pool, a priority sampling method is used to update the parameters of the policy network and the weight coefficient of the risk index to adaptively optimize the placement strategy.

[0136] For example, the real-time status information of the object is obtained, including position, posture, speed and force data. A multi-sensor fusion method is adopted, and a high-precision visual camera and a force tactile sensor array are configured. The vision uses an industrial camera with a frame rate of 120 frames per second, while the force tactile sensor collects force feedback data at a frequency of 1000Hz. Based on these sensor data, an object state prediction matrix is ​​constructed. The matrix contains state prediction vectors for the next 15 time steps. Each state vector includes a position and posture component (x, y, z coordinates and rotational posture represented by quaternions), a velocity component (linear velocity and angular velocity) and a torque component (three-axis force and three-axis torque). The object state prediction adopts an improved long short-term memory network model. The model input includes the state sequence of the past 10 time steps. Through a three-layer LSTM structure, each layer contains 128 neurons to predict the future state change trend.

[0137] After the prediction matrix is ​​constructed, the risk index is calculated. The risk index is calculated by weighting three types of evaluation indicators: the displacement evaluation indicator accounts for 40%, the strain evaluation indicator accounts for 35%, and the force distribution evaluation indicator accounts for 25%. The displacement evaluation indicator is calculated by predicting the possible position deviation of the object in the future. When the predicted deviation exceeds 5mm, the index value increases linearly. The strain evaluation indicator is calculated by monitoring the degree of deformation of the object and setting thresholds based on different material properties. For example, the allowable deformation threshold for flexible materials is 2%. The force distribution evaluation indicator is calculated by analyzing the force uniformity at the force application point. When the ratio of the standard deviation of the force to the average value exceeds 0.3, it is considered unevenly distributed. In actual applications, the risk index calculated for a fragile glass product is 0.72 (range 0-1), indicating a high risk.

[0138] The compensation parameter vector includes position compensation (x, y, and z axis adjustment values, range ±10mm), attitude compensation (expressed in Euler angles, range ±15 degrees), and force compensation (contact force adjustment range 0-5N). For example, for a detected risk index of 0.72, the generated compensation parameters are: position compensation (3.2mm, -2.8mm, 4.1mm), attitude compensation (5.2°, -3.1°, 8.7°), and force compensation -1.5N.

[0139] To achieve adaptive optimization, a reinforcement learning state space is constructed. The state space has 32 dimensions and includes the current real-time state vector (15 dimensions), the risk index (1 dimension), and the historical execution error vector (16 dimensions). A dual network structure is constructed based on the state space: the policy network uses a four-layer fully connected network with 64-128-64-32 nodes per layer, using the ReLU activation function; the value network uses a three-layer structure with 64-32-1 nodes per layer. The dual networks are trained interactively, with the policy network responsible for action decisions and the value network evaluating action value.

[0140] A speed compensation strategy is generated based on the velocity component and risk index. When the risk index exceeds 0.6, the placement speed is reduced to 65% of the original plan; between 0.3 and 0.6, it is reduced to 80% of the original plan; and when it is below 0.3, the original planned speed is maintained. Simultaneously, predictive posture compensation is performed on the robot's end effector based on the position and posture components and the compensation parameter vector. In the example scenario, the end rotation angle is adjusted from the original plan (30°, 45°, 90°) to (35.2°, 41.9°, 98.7°) to adapt to the predicted change in the object's state. The torque component and force control compensation are also used to adjust the position of the point of contact, changing the original four-point contact plan to six-point uniform contact, and adjusting the contact force from the original setting of 3.0N to 1.5N to reduce the risk of object damage.

[0141] During the execution process, the real-time state vector, action vector, reward value and next state vector are stored in the priority experience pool. The experience pool capacity is set to 10,000 records, each record contains state (32 dimensions), action (12 dimensions), reward value (scalar), next state (32 dimensions) and priority weight (scalar). The time difference error is used to determine the sample priority. The sample with the larger error is given a higher weight and the probability of selection is also greater. In model training, 256 records are sampled from the experience pool to form a batch, and samples with high weights are given priority. The learning rate is set to 0.001 and the discount factor is 0.95. Through half a year of actual industrial scenario application, the product damage rate was reduced from the original 3.2% to 0.8%, and work efficiency was improved by 16.5%.

[0142] The strategy network parameters and risk index weight coefficients are continuously updated based on actual execution results. When new material types or environmental changes are detected, the weighting of each evaluation indicator in the risk index calculation is automatically adjusted. For example, when handling fragile items, the weight of the force distribution evaluation indicator is increased to 45%, and when handling flexible materials, the weight of the strain evaluation indicator is increased to 50%, achieving adaptive optimization of the placement strategy.

[0143] The present invention constructs an item state prediction matrix to evaluate placement risks, generates a compensation parameter vector including position, posture and force control compensation, adopts a dual network structure to optimize the placement strategy, performs predictive adjustments to speed, posture and fulcrum, and continuously optimizes network parameters through priority experience pool and priority sampling, thus realizing adaptive optimization of the placement strategy during the palletizing process and improving palletizing accuracy and stability.

[0144] In an optional embodiment, the step of overall optimizing the placement strategy through the dual network structure includes:

[0145] Construct a policy network based on the attention mechanism and a value network based on distributed evaluation. The policy network includes a multi-head self-attention module and a cross-attention module, and the value network includes a quantile regression module and a risk perception module.

[0146] The multi-head self-attention module performs weight mapping on the state features to generate a state representation vector; the cross-attention module performs feature interaction on the state representation vector and the historical action sequence to generate an action attention matrix;

[0147] constructing an action distribution based on the action attention matrix and the risk index, wherein the action distribution includes a speed strategy distribution, a posture strategy distribution, and a force control strategy distribution; and dynamically adjusting the exploration-exploitation balance according to the risk level by using a soft strategy optimization method with adaptive temperature parameter adjustment.

[0148] The state representation vector is input into the quantile regression module to generate a value distribution; the risk index is input into the risk perception module to calculate a risk weight coefficient; the action values ​​are evaluated and ranked based on the value distribution and the risk weight coefficient, and the optimal placement strategy is selected.

[0149] For example, the multi-head self-attention module in the policy network consists of 8 attention heads, each with a dimension of 64, and maps the original state features to three matrices: query, key, and value through a linear projection matrix. The cross-attention module uses a scaled dot product attention mechanism, and the attention score is output after a nonlinear transformation through a 256-dimensional feedforward neural network. The quantile regression module in the value network sets 32 quantile points, evenly distributed in the interval [0, 1], and estimates the state value distribution through 32 parallel fully connected layers. The risk perception module is composed of a three-layer fully connected network with hidden layer dimensions of 128 and 64 respectively, and the activation function is ReLU.

[0150] To process state features, the multi-head self-attention module receives the robot's currently observed state features, including object position, posture, shape characteristics, and environmental constraints. These features are encoded as a 512-dimensional vector and then processed by eight parallel attention heads. Each attention head independently calculates attention weights, adaptively associating features at different locations. For example, when placing a cup, the module automatically focuses on the relative relationship between the bottom of the cup and the placement surface, assigning higher attention weights to these key areas. The outputs of the attention heads are concatenated and linearly projected to produce the final state representation vector, with a dimension of 512.

[0151] During the feature interaction phase, the cross-attention module uses the state representation vector as the query (Q) and the historical action sequence as the key (K) and value (V) to perform cross-attention calculations. The historical action sequence contains the velocity, posture, and force control instructions for the past 10 steps, with each action encoded as a 64-dimensional vector. This cross-attention mechanism captures correlation patterns between the current state and historical actions. For example, when the historical action shows an object approaching a placement surface, attention is increased to the state features of the contact area. The resulting action attention matrix has a dimension of 512 × 64 and contains the attention distribution for each possible action dimension. Specifically, each row in the matrix corresponds to a dimension of the state representation vector, and each column corresponds to an action feature in the historical action sequence. The matrix element value represents the degree of attention paid to a specific state feature relative to a specific action feature, with values ​​ranging from [0, 1]. For example, an element with a value of 0.85 in the matrix indicates a high correlation between the state feature and the corresponding action feature and should be given a higher weight during policy generation.

[0152] During the risk assessment process, a risk index is calculated based on environmental constraints and task requirements. In practice, the risk index is determined by a combination of three factors: object stability risk (40%), collision risk (40%), and task completion time risk (20%). When an object is placed at an edge, the stability risk value can reach 0.8; when the distance between the object and surrounding obstacles is less than 5 cm, the collision risk value increases to 0.6; and when the task execution time exceeds the preset threshold of 15 seconds, the time risk value increases linearly to 0.5. A weighted average of these three risk factors yields a comprehensive risk index, which ranges from 0 to 1.

[0153] Based on the action-attention matrix and the risk index, three strategy distributions are constructed. First, the action-attention matrix is ​​subjected to dimensionality reduction. Through weighted averaging, the 512×64 matrix is ​​converted into three eigenvectors, each with 128 dimensions, corresponding to the speed strategy, posture strategy, and force control strategy. The speed strategy distribution adopts a multivariate Gaussian distribution, with the mean determined by the first 128 dimensions of the attention matrix and the covariance adjusted by the risk index, reducing the covariance to 30% of the normal value in high-risk situations. The posture strategy distribution adopts a quaternion-parameterized von Mises-Fisher distribution, with a concentration parameter that is negatively correlated with the risk index. The force control strategy distribution adopts a Beta distribution, with the shape parameter adjusted as the risk index increases, so that the distribution is more concentrated within the safe force range.

[0154] During the policy optimization phase, a soft policy optimization method with adaptive temperature parameter adjustment is employed. The initial temperature parameter is set to 1.0 and gradually decays as training progresses. The temperature parameter is dynamically adjusted based on the current risk index: when the risk index is greater than 0.7, the temperature parameter is reduced to 50% of the current value, encouraging the policy to choose known safe actions; when the risk index is less than 0.3, the temperature parameter is increased to 150% of the current value, encouraging more exploration. In an experiment placing high-precision electronic components, a high risk (0.82) was detected near placement. Immediately adjusting the temperature parameter from 0.5 to 0.25 significantly improved the peak value of the policy distribution, allowing for successful precision placement.

[0155] During value assessment, the 512-dimensional state representation vector is input into the quantile regression module, generating a value distribution consisting of 32 quantile points. For example, the evaluation results for a specific placement state show a 25th percentile value of -5.2, a 50th percentile value (median) of 3.7, and a 75th percentile value of 8.9, indicating that the expected return distribution for this state is positive but with some volatility. The risk perception module receives a risk index as input and outputs a set of weighting coefficients corresponding to the 32 quantile points. In high-risk situations (risk index 0.9), weights are tilted toward lower quantiles, with the 25th percentile weight increased to 2.3 times the standard weight. In low-risk situations (risk index 0.2), the weight distribution is more evenly distributed. The risk-adjusted action value is calculated by combining the value distribution and risk weight coefficients: the 32 quantile values ​​are multiplied by the corresponding risk weight coefficients, and then the sum is calculated to obtain the weighted average. For example, when the risk index is 0.8, the weight coefficient for low quantiles (worse outcomes) may be 2.5, and the weight coefficient for high quantiles (better outcomes) may be 0.5. This will more conservatively evaluate the value of actions and prioritize strategies that are still safe in the worst case scenario. Finally, all candidate actions are ranked according to the calculated risk-adjusted value, and the action with the highest value is selected as the optimal placement strategy.

[0156] Existing robot placement strategy optimization technologies primarily employ rule-based deterministic methods or simple reinforcement learning algorithms, typically using fixed-structure neural networks to directly map states to actions, or using a single Q-network to evaluate action value. This paper designs a dual network structure based on an attention mechanism, capturing internal state correlations through multi-head self-attention and mapping states to historical actions through cross-attention. It also introduces a distributed action generation method and dynamically adjusts distribution parameters using a risk index. It combines quantile regression with risk perception, achieving fine-grained control of placement risk through weighted assessment of different quantiles, and employs an adaptive temperature parameter adjustment mechanism to balance exploration and exploitation. This achieves precise control of placement strategies, significantly improving the success rate and robustness of palletizing tasks in complex environments.

[0157] Figure 2 Figure 2 shows a comparative analysis of the performance of dual-network placement strategy optimization. The data shows that the dual-network approach of this invention significantly outperforms existing technologies across all evaluation metrics. In terms of placement success rate, this approach achieved 96.5%, 14.2 percentage points higher than the rule-based approach (82.3%) and 17.8 percentage points higher than the traditional reinforcement learning approach (78.7%). In terms of stability risk control, this approach achieved 92.8%, far exceeding the 75.4% and 69.2% of the other two approaches, respectively. This is attributed to the combined application of the risk perception module and quantile regression.

[0158] In terms of collision avoidance rate, our method achieved 94.3%, significantly higher than the 81.5% and 77.8% of competing methods, demonstrating the advantage of the multi-head self-attention mechanism in capturing the relationship between objects and the environment. In terms of task completion time (represented inversely, higher values ​​indicate faster time), our method's 87.2% performance also surpassed the 72.6% and 65.9% of competing methods.

[0159] The most significant difference was in adaptability to complex environments, where our approach achieved 93.7%, compared to only 64.8% for rule-based approaches and 62.1% for traditional reinforcement learning approaches, a gap of nearly 30 percentage points. This fully demonstrates the significant advantages of the dual network structure combined with risk assessment in handling complex and changing environments.

[0160] According to a second aspect of an embodiment of the present invention, an electronic device is provided, including:

[0161] processor;

[0162] a memory for storing processor-executable instructions;

[0163] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0164] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0165] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-station intelligent sorting and palletizing method for industrial robots based on vision guidance, characterized in that: include: Acquire real-time status information for multiple workstations, including workstation backlogs, waiting times, and the urgency of subsequent process requirements. Establish a multi-dimensional workstation status assessment method based on this real-time status information, predict the material arrival frequency and processing time for each workstation based on historical data, and calculate a workstation priority score. Determine the target workstation based on the workstation priority score. Acquire multi-perspective images of stacked items at the target workstation within a preset time window, analyze the position and posture changes of the items in the multi-perspective images, establish a graph neural network state predictor, and calculate the optimal grasping time window based on the graph neural network state predictor; within the optimal grasping time window, predict the position and posture information of the items at the time of grasping execution based on the movement characteristics of the items, and generate a grasping path based on the prediction; Controlling the industrial robot to perform an object grabbing operation according to the grabbing path; During the item palletizing process, continuous visual image sequences and force sensing data are collected. The displacement and deformation of the stack are detected based on the visual image sequences. The magnitude and distribution of the contact force during the placement process are detected based on the force sensing data, and changes in the item state are predicted. Based on the predicted results of the item state changes, compensation parameters are calculated in real time to dynamically adjust the industrial robot's placement strategy, which includes placement speed, placement posture, and point of force.

2. The method according to claim 1, characterized in that Acquire real-time status information for multiple workstations, including workstation backlog, waiting time, and urgency of subsequent process requirements. Establish a multi-dimensional workstation status assessment method based on this real-time status information. Calculate workstation priority scores based on historical data to predict material arrival frequency and processing time for each workstation. The steps include: Construct a workstation relationship graph and extract inter-workstation correlation features through a graph convolutional network. The inter-workstation correlation features and the real-time status information are input into a multi-head attention mechanism fusion network to obtain global fusion features. Based on historical time series data, the Transformer model is used to predict material arrival frequency and material processing time. Establish a multi-objective constraint evaluation system, including efficiency index, balance index and urgency index. The efficiency index is calculated based on the material processing time and the workstation backlog. The balance index is calculated by the workstation backlog deviation and the waiting time deviation. The urgency index is calculated by the urgency of the subsequent process demand and the material arrival frequency. Establish a weight adaptive mechanism, construct a loss function through the decision effect evaluation value, and dynamically update the weight coefficient based on the gradient information of the loss function; The efficiency index, balance index and urgency index are combined with the global fusion feature, and weighted calculation is performed with the dynamically updated weight coefficient to obtain an initial priority score. The initial priority score is multiplied by the correction term of the workstation's historical performance to obtain the workstation priority score; a score feedback correction mechanism is established based on the performance improvement index and the efficiency improvement index, and the workstation priority score is corrected by the adaptive adjustment coefficient to obtain the final workstation priority score.

3. The method according to claim 1, characterized in that The steps of obtaining multi-view images of stacked objects at the target workstation within a preset time window, analyzing the position and posture change patterns of the objects in the multi-view images, establishing a graph neural network state predictor, and calculating the optimal grasping time window based on the graph neural network state predictor include: Collecting multi-view images of the target area and preprocessing them to obtain a depth confidence image; extracting a feature response map of the depth confidence image; Calculating the spatial correlation between items based on the feature response graph, the spatial correlation is obtained by a weighted combination of feature similarity scores and depth continuity scores of overlapping areas of items, and constructing a spatial relationship graph including an item node set and an associated edge set based on the spatial correlation; Inputting the spatial relationship graph into a graph attention mechanism network, updating the node features through node feature transformation and attention vector, performing object segmentation based on the updated node features, and obtaining segmented object regions; Based on the segmented object area, an object state vector including position, attitude angle, velocity, angular velocity, and spatial relationship state is established, and the object state vector and the spatial relationship graph are input into a graph neural network predictor to obtain a state prediction value and prediction uncertainty at the next moment; Calculating stability metrics of the object's posture, motion, and spatial relationship based on the state prediction value, performing confidence weighting on each stability metric in combination with the prediction uncertainty, combining the weighted stability metrics to obtain a stability score, and combining the stability score with the accessibility score and the visibility score to obtain a comprehensive decision target; The optimal grasping time is determined based on the comprehensive decision-making target, and the time window adjustment amount and range are calculated according to the state prediction covariance matrix to obtain the optimal grasping time window centered on the optimal grasping time.

4. The method according to claim 3, characterized in that The steps of predicting the position and posture information of the object at the time of grasping execution according to the motion characteristics of the object within the optimal grasping time window and generating a grasping path based on the prediction include: Based on the state prediction value and prediction uncertainty output by the graph neural network state predictor, perform a Kalman filter update to obtain a filtered item state vector and a filtered state prediction covariance matrix; Calculating a predicted motion trajectory of the object based on the filtered object state vector and the filtered state prediction covariance matrix, and generating a target pose matrix based on the predicted motion trajectory; Verifying the target pose matrix based on velocity, acceleration, angular acceleration, workspace, and joint angle constraints to determine a reachable target pose that satisfies the constraints; A fifth-order polynomial trajectory planning method is used to generate a position path point sequence, wherein the polynomial coefficients of the fifth-order polynomial trajectory planning method are obtained by solving the start and end positions, velocity and acceleration boundary conditions, and a spherical linear interpolation method is used to generate a posture path point sequence; Calculating the motion execution time from the current position to the achievable target pose based on the position path point sequence and the posture path sequence, and determining the grasping execution time in combination with the clock synchronization error and the execution delay; At the grasping execution moment, the position and posture errors during the path execution process are calculated, the correction amount is calculated based on the position and posture errors, and the grasping path is compensated and adjusted in real time, and the robotic arm is driven to execute the compensated grasping path.

5. The method according to claim 1, characterized in that The steps of detecting the displacement and deformation of the stack according to the visual image sequence, detecting the magnitude and distribution of the contact force during the placement process according to the force sensing data, and predicting the change in the state of the item include: Adaptively meshing the surface of the object according to the object's surface curvature, and calculating the mesh density using the curvature gradient; establishing a local coordinate system based on the principal curvature direction based on the adaptive mesh, analyzing changes in mesh node positions based on the visual image sequence, projecting the object's surface deformation onto the principal curvature direction and decomposing the deformation pattern, calculating a deformation gradient tensor and a strain tensor including the displacement field, and obtaining deformation data representing the stacking displacement and deformation; Extracting visual features from the visual image sequence and force features from the force sensing data, the visual features including object contours, surface textures, and correspondences between feature points between adjacent frames, and the force features including contact point positions and corresponding normal and tangential force components, performing intra-modal attention calculations on the visual features and force features to obtain intra-modal features, and performing cross-modal attention calculations on the visual features and force features to obtain inter-modal features; calculating information entropy and signal-to-noise ratio based on the visual image sequence and force sensing data, and determining dynamic fusion weights for the visual and force features based on the information entropy and signal-to-noise ratio; The intra-modal features, inter-modal features and dynamic fusion weights are input into a temporal attention memory network, and the temporal attention weights are obtained based on the historical state information; the visual features, force features and historical state features are weightedly fused according to the dynamic fusion weights and the temporal attention weights to obtain multimodal fusion features; and the state change of the object is predicted based on the multimodal fusion features and the deformation data.

6. The method according to claim 1, characterized in that Based on the prediction results of the object state change, the compensation parameters are calculated in real time to dynamically adjust the placement strategy of the industrial robot. The placement strategy includes the steps of placement speed, placement posture and force point. Establishing an item state prediction matrix based on the prediction results of the item state change, wherein the item state prediction matrix includes real-time state vectors of multiple time steps, wherein the real-time state vectors include position and posture components, velocity components, and torque components; calculating a risk index based on the item state prediction matrix, wherein the risk index is obtained by weighted calculation of a displacement evaluation index, a strain evaluation index, and a force distribution evaluation index; Generating a compensation parameter vector based on the risk index; constructing a reinforcement learning state space, wherein the reinforcement learning state space includes a current real-time state vector, the risk index, and historical execution errors; and constructing a dual network structure of a policy network and a value network based on the reinforcement learning state space; The placement strategy is comprehensively optimized through the dual network structure, a velocity compensation strategy is generated according to the velocity component and the risk index, predictive posture compensation is performed on the robot end effector based on the position posture component and the compensation parameter vector, and predictive adjustment is performed on the position of the force point and the contact force using the torque component and the force control compensation amount in the compensation parameter vector; The real-time state vector, action vector, reward value and next state vector during the execution process are stored in a priority experience pool; based on the sample data in the priority experience pool, a priority sampling method is used to update the parameters of the policy network and the weight coefficient of the risk index to adaptively optimize the placement strategy.

7. The method according to claim 6, characterized in that The steps of overall optimizing the placement strategy through the dual network structure include: Construct a policy network based on the attention mechanism and a value network based on distributed evaluation. The policy network includes a multi-head self-attention module and a cross-attention module, and the value network includes a quantile regression module and a risk perception module. The multi-head self-attention module performs weight mapping on the state features to generate a state representation vector; the cross-attention module performs feature interaction on the state representation vector and the historical action sequence to generate an action attention matrix; constructing an action distribution based on the action attention matrix and the risk index, wherein the action distribution includes a speed strategy distribution, a posture strategy distribution, and a force control strategy distribution; and dynamically adjusting the exploration-exploitation balance according to the risk level by using a soft strategy optimization method with adaptive temperature parameter adjustment. The state representation vector is input into the quantile regression module to generate a value distribution; the risk index is input into the risk perception module to calculate a risk weight coefficient; the action values ​​are evaluated and ranked based on the value distribution and the risk weight coefficient, and the optimal placement strategy is selected.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images