A robot target grasping method based on deep reinforcement learning

Abstract

The application discloses a kind of robot target grasping methods based on deep reinforcement learning, comprising: collecting multimodal perception data and synchronous fusion;Extract spatial and temporal characteristics, generate state characteristics by fusion network;Based on multimodal state characteristics, positive and negative sample pairs are screened and dataset is formed;Construct and train contrast dynamics model;Collect expert and exploration data, train discriminant network to generate counter-reward signal;Fusion multi-item counter-reward signal, update network using proximal policy optimization algorithm;Constraint is applied to strategy output and combined with real-time correction action tactile feedback;Monitoring grasping performance and dynamically adjusting counter-reward signal weight and learning rate parameter.The application collects and fuses multimodal perception data, uses the deep reinforcement learning method based on generative adversarial imitation learning network and contrast dynamics modeling, realizes the autonomous grasping and adaptive control of robot in complex environment.

Description

Technical Field

[0001] This invention relates to the field of robot control and intelligent manufacturing technology, and in particular to a robot target grasping method based on deep reinforcement learning. Background Technology

[0002] Currently, with the rapid development of intelligent manufacturing and human-machine collaboration technologies, the demand for robots to perform autonomous grasping tasks in complex environments is constantly increasing. Existing robot grasping methods mostly rely on single-modal perception information, such as visual or force data, to achieve target recognition and motion planning through traditional control algorithms or deep learning models. They lack the fusion of multi-source perception data and dynamic interaction modeling, making it difficult to comprehensively reflect the temporal changes and physical constraints during the grasping process. While grasping strategies based on deep reinforcement learning possess self-learning capabilities, they still suffer from problems in practical applications such as low sample utilization, sparse reward signals, low training efficiency, and unstable policy convergence. Furthermore, they lack sufficient representation of the physical constraints and uncertainties during the contact process.

[0003] Existing generative adversarial imitation learning methods mostly employ single-layer discriminative structures, only performing global evaluations of the overall action sequence. This makes it difficult to finely discriminate and optimize behavioral differences at different stages of grasping tasks, such as approach, contact, lifting, carrying, and placement. Furthermore, existing methods do not adequately utilize multimodal perceptual information, failing to effectively integrate dynamic data such as visual, force, and tactile senses. This results in unstable generated adversarial reward signals, oscillations during policy optimization, and generalization ability and grasping security that are insufficient to meet the practical application requirements in complex environments.

[0004] Therefore, how to provide a robot target grasping method based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a robot target grasping method based on deep reinforcement learning. This invention integrates multi-source sensory information processing, contrastive dynamics modeling, and generative adversarial learning networks. By constructing an environmental state vector containing visual, force, tactile, and motion data, and combining contact graph modeling and cross-modal attention mechanisms, it achieves accurate representation of dynamic changes and contact relationships during the grasping process. A generative adversarial learning network with a hierarchical discriminative structure and a staged routing mechanism is employed to finely discriminate and optimize behavioral differences in stages such as approach, contact, lifting, transporting, and placement during the grasping task. Furthermore, a pruned proximal policy optimization algorithm is used to improve the stability and convergence efficiency of policy learning. This invention possesses advantages such as strong multimodal fusion capability, high grasping accuracy, good policy generalization, and high execution safety.

[0006] A robot target grasping method based on deep reinforcement learning according to an embodiment of the present invention includes:

[0007] Collect multi-source sensor data during the robot's target grasping process, synchronize and fuse the collected multi-source sensor data in time, and construct an environmental state vector;

[0008] Based on the constructed environmental state vector, spatial and temporal features of various perceptual components in the environmental state vector are extracted. The spatial and temporal features are then combined through a multimodal fusion network to generate multimodal state features.

[0009] Based on multimodal state features, samples with the same object type and similar contact conditions are selected from historical data as positive sample pairs, and samples with large differences are selected as negative sample pairs to construct a comparison sample dataset.

[0010] Construct a contrastive dynamics model and train the contrastive dynamics model based on the constructed contrastive sample dataset;

[0011] A generative adversarial imitation learning network consisting of a policy network and a discriminant network is constructed. Expert demonstration data and robot autonomous exploration data are collected, and the discriminant network is trained. The output of the discriminant network is then converted into adversarial reward signals.

[0012] The adversarial reward signal, the prediction consistency score of the contrastive dynamics model, the sample feature discrimination and the uncertainty penalty value of the contrastive dynamics model are weighted and fused to construct a composite reward function. The policy network is updated by using a pruning proximal policy optimization algorithm.

[0013] Based on the action output by the strategy network, joint torque, velocity and clamping force constraints are applied to correct out-of-limit actions, and the action parameters are adjusted in real time by combining the uncertainty of the contrast dynamics model and tactile feedback.

[0014] During the grasping process, the success rate, slip rate, hold time, and energy consumption are monitored, and the composite reward weight, policy network learning rate, and uncertainty threshold are dynamically adjusted based on the evaluation results.

[0015] Optionally, the multi-source sensing data includes:

[0016] Visual data acquired by a vision sensor, force data acquired by a force sensor, tactile data acquired by a tactile sensor, and motion data acquired by a joint encoder.

[0017] Optionally, the construction of the environment state vector includes:

[0018] The system performs time synchronization and calibration of multi-source sensor data, completes the preprocessing of visual, force, tactile and motion data; resampling and alignment are performed within a unified time window, and the features of various sensor components are standardized; weighted fusion and anomaly removal of multi-source sensor data are performed, and finally an environmental state vector is generated.

[0019] Optionally, the generation of multimodal state features includes:

[0020] The environmental state vector is decomposed into visual, force, tactile and motion components.

[0021] Visual components are input into a convolutional neural network to extract spatial features, while force, touch, and motion components are input into a one-dimensional convolutional neural network to extract temporal features. The extracted visual-spatial features are used to form a spatial feature vector, and the force, touch, and motion features are combined to form a temporal feature vector. The spatial and temporal feature vectors are input into a multimodal fusion network, and a unified multimodal state feature is generated through weighted fusion. The generated multimodal state feature is then input into a policy network.

[0022] Optionally, constructing the comparison sample dataset includes:

[0023] Extract sample features containing multimodal state features, action information, and object categories and contact conditions from historical crawled data; perform time alignment and normalization on the sample features to ensure consistency of feature dimensions between different samples; calculate the similarity between samples based on the similarity of object categories, contact conditions, and state features; define a positive sample pair when the similarity is higher than the first threshold and the category and contact conditions are consistent, and define a negative sample pair when the similarity is lower than the second threshold or the categories and contact conditions are not synchronized, with the first threshold being greater than the second threshold; integrate the positive and negative sample pairs to construct a comparison sample dataset.

[0024] Optionally, the construction and training of the contrastive dynamics model includes:

[0025] The contrastive dynamics model consists of a state encoder, a contact diagram modeling module, a prediction module, an uncertainty sensing unit, a momentum target encoder and sample memory, and a staged expert routing module.

[0026] The state encoder consists of a modal encoder, a feature alignment and normalization unit, a shared bottleneck layer, and a channel selection unit; the contact graph modeling module consists of a contact point detection unit, a graph construction unit, a message passing and aggregation unit, and a contact feature pooling unit; and the prediction module consists of a physical prior branch, a data-driven branch, a residual fusion unit, and an uncertainty branch.

[0027] The uncertainty perception unit outputs sample and modal channel uncertainties and performs dynamic weighting; the momentum target encoder and sample memory store and update historical sample features; the staged expert routing module performs expert network selection, parameter optimization, and route updating according to different stages of the crawling process.

[0028] The modal encoder in the state encoder is used to extract features from visual, force, tactile and motion data respectively. After the time step and scale are unified by the feature alignment and normalization unit, the data are input into the shared bottleneck layer for feature mapping and fusion to form a unified intermediate feature representation.

[0029] The contact point detection unit and graph construction unit of the contact graph modeling module are invoked to identify contact points and establish graph structures based on force and tactile data; contact relationship features are generated by the message passing and aggregation unit, and then integrated by the contact feature pooling unit and fused with the intermediate features output by the shared bottleneck layer to obtain a fused high-dimensional representation.

[0030] By utilizing the channel selection unit and cross-modal attention mechanism in the state encoder, the modal channels of the fused features are weighted and denoised to generate a unified fused representation;

[0031] The prediction module's physical prior branch and data-driven branch are invoked to generate prediction results, where:

[0032] The physical prior branch is based on analytical calculations of the dynamic equations, while the data-driven branch learns feature maps through neural networks.

[0033] The prediction results of the two branches are used for feature synthesis by the residual fusion unit to obtain the state feature prediction results at the next time step, and physical consistency constraints are introduced.

[0034] The uncertainty of the generated results is estimated by using the uncertainty branch of the prediction module, and the importance of samples and modal channels is dynamically adjusted according to the uncertainty weight; at the same time, the momentum target encoder and sample memory are called to align and update historical samples.

[0035] Based on the stage of the capture task, the corresponding expert sub-network is activated by calling the staged expert routing module to optimize the parameters and update the routes in stages. After the model training is completed, a comparative dynamics model with multimodal fusion, contact topology modeling, physical prior constraints and uncertainty perception capabilities is output.

[0036] Optionally, converting the discriminant network output into an adversarial reward signal includes:

[0037] Construct a generative adversarial imitation learning network consisting of a policy network and a discriminant network, wherein:

[0038] The policy network receives multimodal environmental state features and outputs action distribution; the discriminant network includes three types of hierarchical discriminators: state-level, action-level, and trajectory-level; the hierarchical discriminators are linked with the task phase signal through a staged routing mechanism;

[0039] The robot collects expert demonstration data by performing grasping tasks through human-robot teaching, and records the state and action sequence of visual, force, tactile and motion information generated during the collection of expert demonstration data; at the same time, it controls the robot to autonomously explore in the simulation or real environment based on the current policy network, and collects and processes autonomous exploration data in real time.

[0040] The expert demonstration data and the autonomous exploration data are respectively input into three types of hierarchical discriminators. The corresponding discrimination results are calculated in each stage of the grasping task, including approach, contact, lifting, transporting and placing. The parameters of the three types of hierarchical discriminators are jointly updated by hierarchical weighting, and gradient constraints and spectral normalization are introduced.

[0041] The output of the discriminant network to policy samples is processed by logarithmic mapping and numerical compression to generate adversarial reward signals, which together with the policy entropy regularization term constitute the policy optimization objective.

[0042] By introducing the feature representation and prediction results of the contrastive dynamics model, feature consistency constraints are constructed, and dynamic weighting and gating of training samples and modal channels are applied based on the uncertainty of the model output.

[0043] A target discriminator with momentum updates is set up in the side path of the discriminant network, and a sample memory bank covering a long time span is maintained;

[0044] The discriminant network training and policy network update are performed alternately within a fixed number of steps. By comprehensively applying hierarchical weighting, stage routing, feature consistency, uncertainty weighting and memory resampling strategies, the network training is completed and the optimized policy network and discriminant network parameters are output.

[0045] Optionally, updating the policy network using a pruning-based near-end policy optimization algorithm includes:

[0046] The discriminant network output is used as the adversarial reward signal, the contrastive dynamics model output is used as the prediction consistency score, the sample feature difference is defined as the feature discrimination index, and a penalty term is set according to the model uncertainty. The components are weighted and fused according to preset weights to generate a composite reward signal. The cumulative reward of the capture sequence is calculated based on the composite reward signal, and the advantage quantity obtained by the difference between the cumulative reward and the baseline value is determined. The policy network parameters are updated using a pruning proximal policy optimization algorithm based on the advantage quantity, and the weights are gradually adjusted according to the learning rate while limiting the change range. Change constraints are applied in continuous updates, and the composite reward weight, uncertainty penalty coefficient, learning rate and entropy regularization parameters are dynamically adjusted in combination with online evaluation results to output the optimized policy network parameters.

[0047] Optionally, the real-time adjustment of motion parameters includes:

[0048] Based on the robot's inherent characteristics, upper and lower limits for joint torque, joint velocity, and gripping force are set to establish a safe range for actions, limiting output actions to not exceeding the safe boundary. Action commands output by the policy network are acquired, and it is determined whether they fall within the safe range. Actions exceeding the limits are corrected and restricted to the allowable range. Based on the uncertainty information of the comparative dynamics model and the slippage changes detected by the tactile sensors, the joint velocity gain, force control gain, and gripping force settings are adjusted in real time. When executing corrective actions, a hybrid control method combining force control and position control is adopted, dynamically updating control parameters based on real-time tactile feedback, and increasing normal support force and performing fine-tuning compensation when slippage trends occur. During the grasping phase, the number of constraint violations, control intervention intensity, and safety margin are continuously monitored. When any indicator exceeds the limit, backtracking control or path replanning is triggered, and the corrected and optimized final action sequence is output.

[0049] Optionally, the dynamic adjustment of the composite reward weights, the policy network learning rate, and the uncertainty threshold includes:

[0050] The system sets an online evaluation time window and statistical frequency, continuously collecting seven key indicators: capture success rate, slip rate, hold time, energy consumption, task duration, peak force, and joint boundary crossing count. Within the evaluation window, it calculates the mean, variance, and extreme values ​​of each indicator and compares them with the target value and safety threshold to generate evaluation results. Based on the evaluation results, it dynamically adjusts the composite reward signal weight, policy network learning rate, entropy regularization coefficient, uncertainty threshold, and network update steps. Combining the uncertainty of the comparative dynamics model, the confidence of the discriminant network, and safety constraint information, it revises the upper limit of the composite reward signal weight and learning rate, and updates the gating coefficient of the multimodal fusion network simultaneously. It records the evaluation results and parameter adjustments as a traceable version, automatically rolling back to the previous stable version when consecutive evaluations fall below the threshold, and outputting the currently effective parameters.

[0051] The beneficial effects of this invention are:

[0052] This invention achieves high-precision intelligent optimization of robot grasping tasks by constructing a contrastive dynamics model and a generative adversarial learning network. By collecting visual, force, tactile, and motion data and performing time synchronization and fusion, a unified environmental state vector is constructed. Spatial and temporal features are extracted in the multimodal fusion network, enabling the collaborative expression of multidimensional information and significantly improving the robot's perception and understanding of complex grasping scenarios. The introduction of a contact graph modeling module and a cross-modal attention mechanism accurately characterizes the contact relationship between the end effector and the target object, and the combination of dynamic consistency constraints improves the prediction accuracy and stability of the nonlinear grasping process. Through an uncertainty-aware branch and a momentum target encoding mechanism, high-uncertainty samples can be identified and weights dynamically adjusted, effectively mitigating oscillations and convergence instability in policy training, and improving sample utilization and learning robustness.

[0053] This invention employs a generative adversarial learning network with a hierarchical discriminative structure and a staged routing mechanism to perform fine-grained discrimination and optimization of the approach, contact, lifting, transport, and placement stages of a grasping task. By setting state-level, action-level, and trajectory-level discriminators, different levels of discriminators can be dynamically activated based on the task phase signal, generating continuous, smooth, and more targeted adversarial reward signals. A composite reward function is constructed by combining the consistency score and uncertainty penalty output from the contrastive dynamics model, and updated using a pruned proximal policy optimization algorithm, enabling efficient and stable optimization of the policy network. During the execution phase, tactile feedback and safety constraints are combined to correct action parameters and adjust the learning rate and reward weights in real time, ensuring the safety and accuracy of action execution. This invention effectively improves the policy stability, generalization ability, and execution safety of robot grasping tasks, enabling safe, accurate, and efficient autonomous grasping control in complex dynamic environments. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a flowchart of a robot target grasping method based on deep reinforcement learning proposed in this invention;

[0056] Figure 2 This is a schematic diagram of the generative adversarial imitation learning network structure of a robot target grasping method based on deep reinforcement learning according to the present invention. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0058] refer to Figure 1 and Figure 2 A robot target grasping method based on deep reinforcement learning includes:

[0059] Collect multi-source sensor data during the robot's target grasping process, synchronize and fuse the collected multi-source sensor data in time, and construct an environmental state vector;

[0060] Based on the constructed environmental state vector, spatial and temporal features of various perceptual components in the environmental state vector are extracted. The spatial and temporal features are then combined through a multimodal fusion network to generate multimodal state features.

[0061] Based on multimodal state features, samples with the same object type and similar contact conditions are selected from historical data as positive sample pairs, and samples with large differences are selected as negative sample pairs to construct a comparison sample dataset.

[0062] Construct a contrastive dynamics model and train the contrastive dynamics model based on the constructed contrastive sample dataset;

[0063] A generative adversarial imitation learning network consisting of a policy network and a discriminant network is constructed. Expert demonstration data and robot autonomous exploration data are collected, and the discriminant network is trained. The output of the discriminant network is then converted into adversarial reward signals.

[0064] The adversarial reward signal, the prediction consistency score of the contrastive dynamics model, the sample feature discrimination and the uncertainty penalty value of the contrastive dynamics model are weighted and fused to construct a composite reward function. The policy network is updated by using a pruning proximal policy optimization algorithm.

[0065] Based on the action output by the strategy network, joint torque, velocity and clamping force constraints are applied to correct out-of-limit actions, and the action parameters are adjusted in real time by combining the uncertainty of the contrast dynamics model and tactile feedback.

[0066] During the grasping process, the success rate, slip rate, hold time, and energy consumption are monitored, and the composite reward weight, policy network learning rate, and uncertainty threshold are dynamically adjusted based on the evaluation results.

[0067] In this embodiment, the multi-source sensing data includes:

[0068] Visual data acquired by a vision sensor, force data acquired by a force sensor, tactile data acquired by a tactile sensor, and motion data acquired by a joint encoder.

[0069] In this embodiment, constructing the environment state vector includes:

[0070] The system performs time synchronization and calibration of multi-source sensor data, completes the preprocessing of visual, force, tactile and motion data; resampling and alignment are performed within a unified time window, and the features of various sensor components are standardized; weighted fusion and anomaly removal of multi-source sensor data are performed, and finally an environmental state vector is generated.

[0071] In this embodiment, generating multimodal state features includes:

[0072] The environmental state vector is decomposed into visual, force, tactile and motion components.

[0073] Visual components are input into a convolutional neural network to extract spatial features, while force, touch, and motion components are input into a one-dimensional convolutional neural network to extract temporal features. The extracted visual-spatial features are used to form a spatial feature vector, and the force, touch, and motion features are combined to form a temporal feature vector. The spatial and temporal feature vectors are input into a multimodal fusion network, and a unified multimodal state feature is generated through weighted fusion. The generated multimodal state feature is then input into a policy network.

[0074] In this embodiment, constructing the comparison sample dataset includes:

[0075] Extract sample features containing multimodal state features, action information, and object categories and contact conditions from historical crawled data; perform time alignment and normalization on the sample features to ensure consistency of feature dimensions between different samples; calculate the similarity between samples based on the similarity of object categories, contact conditions, and state features; define a positive sample pair when the similarity is higher than the first threshold and the category and contact conditions are consistent, and define a negative sample pair when the similarity is lower than the second threshold or the categories and contact conditions are not synchronized, with the first threshold being greater than the second threshold; integrate the positive and negative sample pairs to construct a comparison sample dataset.

[0076] In this embodiment, the construction and training of the contrastive dynamics model includes:

[0077] The contrastive dynamics model consists of a state encoder, a contact diagram modeling module, a prediction module, an uncertainty sensing unit, a momentum target encoder and sample memory, and a staged expert routing module.

[0078] The state encoder consists of a modal encoder, a feature alignment and normalization unit, a shared bottleneck layer, and a channel selection unit; the contact graph modeling module consists of a contact point detection unit, a graph construction unit, a message passing and aggregation unit, and a contact feature pooling unit; and the prediction module consists of a physical prior branch, a data-driven branch, a residual fusion unit, and an uncertainty branch.

[0079] The uncertainty perception unit outputs sample and modal channel uncertainties and performs dynamic weighting; the momentum target encoder and sample memory store and update historical sample features; the staged expert routing module performs expert network selection, parameter optimization, and route updating according to different stages of the crawling process.

[0080] The modal encoder in the state encoder is used to extract features from visual, force, tactile and motion data respectively. After the time step and scale are unified by the feature alignment and normalization unit, the data are input into the shared bottleneck layer for feature mapping and fusion to form a unified intermediate feature representation.

[0081] The contact point detection unit and graph construction unit of the contact graph modeling module are invoked to identify contact points and establish graph structures based on force and tactile data; contact relationship features are generated by the message passing and aggregation unit, and then integrated by the contact feature pooling unit and fused with the intermediate features output by the shared bottleneck layer to obtain a fused high-dimensional representation.

[0082] By utilizing the channel selection unit and cross-modal attention mechanism in the state encoder, the modal channels of the fused features are weighted and denoised to generate a unified fused representation;

[0083] The prediction module's physical prior branch and data-driven branch are invoked to generate prediction results, where:

[0084] The physical prior branch is based on analytical calculations of the dynamic equations, while the data-driven branch learns feature maps through neural networks.

[0085] The prediction results of the two branches are used for feature synthesis by the residual fusion unit to obtain the state feature prediction results at the next time step, and physical consistency constraints are introduced.

[0086] The uncertainty of the generated results is estimated by using the uncertainty branch of the prediction module, and the importance of samples and modal channels is dynamically adjusted according to the uncertainty weight; at the same time, the momentum target encoder and sample memory are called to align and update historical samples.

[0087] Based on the stage of the capture task, the corresponding expert sub-network is activated by calling the staged expert routing module to optimize the parameters and update the routes in stages. After the model training is completed, a comparative dynamics model with multimodal fusion, contact topology modeling, physical prior constraints and uncertainty perception capabilities is output.

[0088] In this embodiment, converting the discriminant network output into an adversarial reward signal includes:

[0089] Construct a generative adversarial imitation learning network consisting of a policy network and a discriminant network, wherein:

[0090] The policy network receives multimodal environmental state features and outputs action distribution; the discriminant network includes three types of hierarchical discriminators: state-level, action-level, and trajectory-level; the hierarchical discriminators are linked with the task phase signal through a staged routing mechanism;

[0091] The robot collects expert demonstration data by performing grasping tasks through human-robot teaching, and records the state and action sequence of visual, force, tactile and motion information generated during the collection of expert demonstration data; at the same time, it controls the robot to autonomously explore in the simulation or real environment based on the current policy network, and collects and processes autonomous exploration data in real time.

[0092] The expert demonstration data and the autonomous exploration data are respectively input into three types of hierarchical discriminators. The corresponding discrimination results are calculated in each stage of the grasping task, including approach, contact, lifting, transporting and placing. The parameters of the three types of hierarchical discriminators are jointly updated by hierarchical weighting, and gradient constraints and spectral normalization are introduced.

[0093] The output of the discriminant network to policy samples is processed by logarithmic mapping and numerical compression to generate adversarial reward signals, which together with the policy entropy regularization term constitute the policy optimization objective.

[0094] By introducing the feature representation and prediction results of the contrastive dynamics model, feature consistency constraints are constructed, and dynamic weighting and gating of training samples and modal channels are applied based on the uncertainty of the model output.

[0095] A target discriminator with momentum updates is set up in the side path of the discriminant network, and a sample memory bank covering a long time span is maintained;

[0096] The discriminant network training and policy network update are performed alternately within a fixed number of steps. By comprehensively applying hierarchical weighting, stage routing, feature consistency, uncertainty weighting and memory resampling strategies, the network training is completed and the optimized policy network and discriminant network parameters are output.

[0097] In this embodiment, updating the policy network using a pruning-based near-end policy optimization algorithm includes:

[0098] The discriminant network output is used as the adversarial reward signal, the contrastive dynamics model output is used as the prediction consistency score, the sample feature difference is defined as the feature discrimination index, and a penalty term is set according to the model uncertainty. The components are weighted and fused according to preset weights to generate a composite reward signal. The cumulative reward of the capture sequence is calculated based on the composite reward signal, and the advantage quantity obtained by the difference between the cumulative reward and the baseline value is determined. The policy network parameters are updated using a pruning proximal policy optimization algorithm based on the advantage quantity, and the weights are gradually adjusted according to the learning rate while limiting the change range. Change constraints are applied in continuous updates, and the composite reward weight, uncertainty penalty coefficient, learning rate and entropy regularization parameters are dynamically adjusted in combination with online evaluation results to output the optimized policy network parameters.

[0099] In this embodiment, the real-time adjustment of motion parameters includes:

[0100] Based on the robot's inherent characteristics, upper and lower limits for joint torque, joint velocity, and gripping force are set to establish a safe range for actions, limiting output actions to not exceeding the safe boundary. Action commands output by the policy network are acquired, and it is determined whether they fall within the safe range. Actions exceeding the limits are corrected and restricted to the allowable range. Based on the uncertainty information of the comparative dynamics model and the slippage changes detected by the tactile sensors, the joint velocity gain, force control gain, and gripping force settings are adjusted in real time. When executing corrective actions, a hybrid control method combining force control and position control is adopted, dynamically updating control parameters based on real-time tactile feedback, and increasing normal support force and performing fine-tuning compensation when slippage trends occur. During the grasping phase, the number of constraint violations, control intervention intensity, and safety margin are continuously monitored. When any indicator exceeds the limit, backtracking control or path replanning is triggered, and the corrected and optimized final action sequence is output.

[0101] In this embodiment, the dynamic adjustment of the composite reward weight, the policy network learning rate, and the uncertainty threshold includes:

[0102] The system sets an online evaluation time window and statistical frequency, continuously collecting seven key indicators: capture success rate, slip rate, hold time, energy consumption, task duration, peak force, and joint boundary crossing count. Within the evaluation window, it calculates the mean, variance, and extreme values ​​of each indicator and compares them with the target value and safety threshold to generate evaluation results. Based on the evaluation results, it dynamically adjusts the composite reward signal weight, policy network learning rate, entropy regularization coefficient, uncertainty threshold, and network update steps. Combining the uncertainty of the comparative dynamics model, the confidence of the discriminant network, and safety constraint information, it revises the upper limit of the composite reward signal weight and learning rate, and updates the gating coefficient of the multimodal fusion network simultaneously. It records the evaluation results and parameter adjustments as a traceable version, automatically rolling back to the previous stable version when consecutive evaluations fall below the threshold, and outputting the currently effective parameters.

[0103] Example 1:

[0104] To verify the feasibility of this invention in practice, it was experimentally verified by applying it to the automated sorting of multiple types of parts on an industrial production line. The experiment was conducted on a flexible assembly line at a smart manufacturing experimental center in a certain city. The experimental platform used a six-DOF UR5 robotic arm, equipped with a high-definition vision camera (1280×720 resolution), a six-dimensional force sensor, a flexible tactile array, and a high-precision joint encoder to achieve multimodal information acquisition and fusion of the working environment. The sorting objects included 15 different shapes, sizes, and materials of parts, such as nuts, gears, bearings, and electrical housings. The objects were randomly stacked, with varied postures, and some overlapped and occluded. Traditional visual grasping algorithms achieved a grasping success rate of only 83.2% in this environment, failing to effectively distinguish similar structural objects and exhibiting problems such as unstable grasping actions and poor strategy generalization.

[0105] During the experiment, the system first calibrated and synchronously acquired multimodal sensing devices. By fusing visual, force, tactile, and motion information, an environmental state vector was constructed, enabling the robot to simultaneously perceive the spatial position, contact state, and force changes of objects, achieving stable perception of complex stacked environments. The experiment used a human-robot teaching method to collect 1200 sets of expert demonstration data, and then used a policy network for autonomous exploration in a simulation environment, generating approximately 8000 sets of exploration samples. Both types of data were processed uniformly after time step and feature alignment, and used to train an adversarial imitation learning network. Furthermore, the system dynamically adjusted the grasping strategy through multimodal feature fusion and task stage recognition mechanisms, enabling the robot to adaptively switch between discrimination and control logic according to different stages such as approach, contact, lifting, carrying, and placement.

[0106] Experimental results show that the method of this invention achieves a 97.6% success rate in grasping under conditions of random stacking, occlusion, and dynamic interference, with an average grasping time of 2.31 seconds per attempt and energy consumption reduced by approximately 12.5% ​​compared to traditional visual grasping algorithms. During a 72-hour continuous stability test, the system did not exhibit any false grasps, slippage, or boundary violations. Compared to traditional methods, this invention achieves significant improvements in grasping accuracy, execution speed, and energy efficiency, validating the effectiveness of multimodal information fusion, staged strategy learning, and contrastive dynamics modeling. It possesses high precision, high robustness, and adaptive safety control capabilities, meeting the autonomous sorting needs of multiple types of parts in complex industrial production environments.

[0107] Table 1. Experimental results comparing the grasping performance of robots based on generative adversarial imitation learning networks.

[0108] Traditional visual grasping algorithms Single vision none Rule control — 83.2 2.83 34.5 6.8 Single-modal reinforcement learning methods Visual input Single-layer discriminator Q-learning 12 89.9 2.65 32.8 4.3 Multimodal reinforcement learning methods Sight + Force + Touch Single-stage discriminator DDPG 9 94.3 2.47 31.6 2.5 Multimodal reinforcement learning + contrastive dynamics modeling method Vision + Force + Touch + Movement Single-stage discriminator PPO 8 96.2 2.38 30.8 1.8 Method of the present invention Vision + Force + Touch + Movement Hierarchical discrimination structure + staged routing mechanism Pruning-based near-end strategy optimization 7 97.6 2.31 30.2 1.2

[0109] As shown in Table 1, the method of this invention exhibits significant advantages in grasping performance, learning efficiency, and energy consumption control. Traditional visual grasping algorithms rely on a single visual input, with a grasping success rate of only 83.2% and a false grasping rate as high as 6.8%. Single-modal reinforcement learning methods achieve a certain degree of self-learning capability through the Q-learning algorithm, increasing the success rate to 89.9%, but still suffer from slow policy convergence and long training time. Multimodal reinforcement learning methods further incorporate visual, force, and tactile information, achieving a success rate of 94.3%, but due to the use of a single-stage discriminative structure, they cannot perform refined optimization for different features of the grasping task, resulting in limited performance improvement.

[0110] This invention introduces a contrastive dynamics modeling and hierarchical generative adversarial learning mechanism into its algorithm structure, utilizing a hierarchical structure of state-level, action-level, and trajectory-level discriminators to achieve multi-level discrimination and optimization. Simultaneously, a pruning proximal strategy optimization algorithm is employed, effectively improving learning stability and convergence efficiency, reducing training time to 7 hours. The multimodal fusion structure enables the collaborative processing of visual, force, tactile, and motion data, improving the accuracy of state representation and the reliability of decision-making.

[0111] Overall, the method of this invention achieves a 97.6% success rate in grasping data, reduces the average grasping time to 2.31 seconds, lowers energy consumption to 30.2 J, and has a false grasping rate of only 1.2%. Compared with other comparative algorithms, the performance improvement is significant, especially in grasping accuracy, execution stability, and energy efficiency optimization, fully verifying the effectiveness and engineering application value of the method adopted in this invention.

[0112] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robot target grasping method based on deep reinforcement learning, characterized in that, include: Collect multi-source sensor data during the robot's target grasping process, synchronize and fuse the collected multi-source sensor data in time, and construct an environmental state vector; Based on the constructed environmental state vector, spatial and temporal features of various perceptual components in the environmental state vector are extracted. The spatial and temporal features are then combined through a multimodal fusion network to generate multimodal state features. Based on multimodal state features, samples with the same object type and similar contact conditions are selected from historical data as positive sample pairs, and samples with large differences are selected as negative sample pairs to construct a comparison sample dataset. Construct a contrastive dynamics model and train the contrastive dynamics model based on the constructed contrastive sample dataset; A generative adversarial imitation learning network consisting of a policy network and a discriminant network is constructed. Expert demonstration data and robot autonomous exploration data are collected, and the discriminant network is trained. The output of the discriminant network is then converted into adversarial reward signals. The adversarial reward signal, the prediction consistency score of the contrastive dynamics model, the sample feature discrimination and the uncertainty penalty value of the contrastive dynamics model are weighted and fused to construct a composite reward function. The policy network is updated by using a pruning proximal policy optimization algorithm. Based on the action output by the strategy network, joint torque, velocity and clamping force constraints are applied to correct out-of-limit actions, and the action parameters are adjusted in real time by combining the uncertainty of the contrast dynamics model and tactile feedback. During the grasping process, the success rate, slip rate, hold time, and energy consumption are monitored, and the composite reward weight, policy network learning rate, and uncertainty threshold are dynamically adjusted based on the evaluation results.

2. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The multi-source sensing data includes: Visual data acquired by a vision sensor, force data acquired by a force sensor, tactile data acquired by a tactile sensor, and motion data acquired by a joint encoder.

3. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The constructed environment state vector includes: The system performs time synchronization and calibration of multi-source sensor data, completes the preprocessing of visual, force, tactile and motion data; resampling and alignment are performed within a unified time window, and the features of various sensor components are standardized; weighted fusion and anomaly removal of multi-source sensor data are performed, and finally an environmental state vector is generated.

4. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The generation of multimodal state features includes: The environmental state vector is decomposed into visual, force, tactile and motion components. Visual components are input into a convolutional neural network to extract spatial features, while force, touch, and motion components are input into a one-dimensional convolutional neural network to extract temporal features. The extracted visual-spatial features are used to form a spatial feature vector, and the force, touch, and motion features are combined to form a temporal feature vector. The spatial and temporal feature vectors are input into a multimodal fusion network, and a unified multimodal state feature is generated through weighted fusion. The generated multimodal state feature is then input into a policy network.

5. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The construction of the comparison sample dataset includes: Extract sample features containing multimodal state features, action information, and object categories and contact conditions from historical crawled data; perform time alignment and normalization on the sample features to ensure consistency of feature dimensions between different samples; calculate the similarity between samples based on the similarity of object categories, contact conditions, and state features; define a positive sample pair when the similarity is higher than the first threshold and the category and contact conditions are consistent, and define a negative sample pair when the similarity is lower than the second threshold or the categories and contact conditions are not synchronized, with the first threshold being greater than the second threshold; integrate the positive and negative sample pairs to construct a comparison sample dataset.

6. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The construction and training of the contrastive dynamics model includes: The contrastive dynamics model consists of a state encoder, a contact diagram modeling module, a prediction module, an uncertainty sensing unit, a momentum target encoder and sample memory, and a staged expert routing module. The state encoder consists of a modal encoder, a feature alignment and normalization unit, a shared bottleneck layer, and a channel selection unit; the contact graph modeling module consists of a contact point detection unit, a graph construction unit, a message passing and aggregation unit, and a contact feature pooling unit; and the prediction module consists of a physical prior branch, a data-driven branch, a residual fusion unit, and an uncertainty branch. The uncertainty perception unit outputs sample and modal channel uncertainties and performs dynamic weighting; the momentum target encoder and sample memory store and update historical sample features; the staged expert routing module performs expert network selection, parameter optimization, and route updating according to different stages of the crawling process. The modal encoder in the state encoder is used to extract features from visual, force, tactile and motion data respectively. After the time step and scale are unified by the feature alignment and normalization unit, the data are input into the shared bottleneck layer for feature mapping and fusion to form a unified intermediate feature representation. The contact point detection unit and graph construction unit of the contact graph modeling module are invoked to identify contact points and establish graph structures based on force and tactile data; contact relationship features are generated by the message passing and aggregation unit, and then integrated by the contact feature pooling unit and fused with the intermediate features output by the shared bottleneck layer to obtain a fused high-dimensional representation. By utilizing the channel selection unit and cross-modal attention mechanism in the state encoder, the modal channels of the fused features are weighted and denoised to generate a unified fused representation; The prediction module's physical prior branch and data-driven branch are invoked to generate prediction results, where: The physical prior branch is based on analytical calculations of the dynamic equations, while the data-driven branch learns feature maps through neural networks. The prediction results of the two branches are used for feature synthesis by the residual fusion unit to obtain the state feature prediction results at the next time step, and physical consistency constraints are introduced. The uncertainty of the generated results is estimated by using the uncertainty branch of the prediction module, and the importance of samples and modal channels is dynamically adjusted according to the uncertainty weight; at the same time, the momentum target encoder and sample memory are called to align and update historical samples. Based on the stage of the capture task, the corresponding expert sub-network is activated by calling the staged expert routing module to optimize the parameters and update the routes in stages. After the model training is completed, a comparative dynamics model with multimodal fusion, contact topology modeling, physical prior constraints and uncertainty perception capabilities is output.

7. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The process of converting the discriminant network output into an adversarial reward signal includes: Construct a generative adversarial imitation learning network consisting of a policy network and a discriminant network, wherein: The policy network receives multimodal environmental state features and outputs action distribution; the discriminant network includes three types of hierarchical discriminators: state-level, action-level, and trajectory-level; the hierarchical discriminators are linked with the task phase signal through a staged routing mechanism; The robot collects expert demonstration data by performing grasping tasks through human-robot teaching, and records the state and action sequence of visual, force, tactile and motion information generated during the collection of expert demonstration data; at the same time, it controls the robot to autonomously explore in the simulation or real environment based on the current policy network, and collects and processes autonomous exploration data in real time. The expert demonstration data and the autonomous exploration data are respectively input into three types of hierarchical discriminators. The corresponding discrimination results are calculated in each stage of the grasping task, including approach, contact, lifting, transporting and placing. The parameters of the three types of hierarchical discriminators are jointly updated by hierarchical weighting, and gradient constraints and spectral normalization are introduced. The output of the discriminant network to policy samples is processed by logarithmic mapping and numerical compression to generate adversarial reward signals, which together with the policy entropy regularization term constitute the policy optimization objective. By introducing the feature representation and prediction results of the contrastive dynamics model, feature consistency constraints are constructed, and dynamic weighting and gating of training samples and modal channels are applied based on the uncertainty of the model output. A target discriminator with momentum updates is set up in the side path of the discriminant network, and a sample memory bank covering a long time span is maintained; The discriminant network training and policy network update are performed alternately within a fixed number of steps. By comprehensively applying hierarchical weighting, stage routing, feature consistency, uncertainty weighting and memory resampling strategies, the network training is completed and the optimized policy network and discriminant network parameters are output.

8. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The method of updating the policy network using a pruning-based near-end policy optimization algorithm includes: The discriminant network output is used as the adversarial reward signal, the contrastive dynamics model output is used as the prediction consistency score, the sample feature difference is defined as the feature discrimination index, and a penalty term is set according to the model uncertainty. The components are weighted and fused according to preset weights to generate a composite reward signal. The cumulative reward of the capture sequence is calculated based on the composite reward signal, and the advantage quantity obtained by the difference between the cumulative reward and the baseline value is determined. The policy network parameters are updated using a pruning proximal policy optimization algorithm based on the advantage quantity, and the weights are gradually adjusted according to the learning rate while limiting the change range. Change constraints are applied in continuous updates, and the composite reward weight, uncertainty penalty coefficient, learning rate and entropy regularization parameters are dynamically adjusted in combination with online evaluation results to output the optimized policy network parameters.

9. The robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The real-time adjustment of action parameters includes: Based on the robot's inherent characteristics, upper and lower limits for joint torque, joint velocity, and gripping force are set to establish a safe range for actions, limiting output actions to not exceeding the safe boundary. Action commands output by the policy network are acquired, and it is determined whether they fall within the safe range. Actions exceeding the limits are corrected and restricted to the allowable range. Based on the uncertainty information of the comparative dynamics model and the slippage changes detected by the tactile sensors, the joint velocity gain, force control gain, and gripping force settings are adjusted in real time. When executing corrective actions, a hybrid control method combining force control and position control is adopted, dynamically updating control parameters based on real-time tactile feedback, and increasing normal support force and performing fine-tuning compensation when slippage trends occur. During the grasping phase, the number of constraint violations, control intervention intensity, and safety margin are continuously monitored. When any indicator exceeds the limit, backtracking control or path replanning is triggered, and the corrected and optimized final action sequence is output.

10. A robot target grasping method based on deep reinforcement learning according to claim 1, characterized in that, The dynamic adjustment of the composite reward weights, the policy network learning rate, and the uncertainty threshold includes: The system sets an online evaluation time window and statistical frequency, continuously collecting seven key indicators: capture success rate, slip rate, hold time, energy consumption, task duration, peak force, and joint boundary crossing count. Within the evaluation window, it calculates the mean, variance, and extreme values ​​of each indicator and compares them with the target value and safety threshold to generate evaluation results. Based on the evaluation results, it dynamically adjusts the composite reward signal weight, policy network learning rate, entropy regularization coefficient, uncertainty threshold, and network update steps. Combining the uncertainty of the comparative dynamics model, the confidence of the discriminant network, and safety constraint information, it revises the upper limit of the composite reward signal weight and learning rate, and updates the gating coefficient of the multimodal fusion network simultaneously. It records the evaluation results and parameter adjustments as a traceable version, automatically rolling back to the previous stable version when consecutive evaluations fall below the threshold, and outputting the currently effective parameters.

Images