A robotic arm grasping decision-making method and system based on grasping neural networks
By using a grasping neural network-based method, the contact state information between the robotic arm and the object is collected and analyzed in real time, and the grasping force is dynamically adjusted. This solves the problem that a fixed force threshold is difficult to adapt to different objects, and improves the grasping stability and safety of the robotic arm.
Patent Information
- Application Number
- CN202611588584.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, when humanoid robots grasp different objects, the preset fixed force threshold is difficult to cover all object types, resulting in insufficient or excessive grasping force, which affects grasping stability and safety.
By employing a grasping neural network-based method, the contact state information between the robotic arm and the object is collected in real time. The trained grasping neural network is used for forward inference calculation to dynamically adjust the grasping force and identify the grasping state, thereby achieving adaptive grasping decision-making.
It improves the stability and safety of the robotic arm in grasping different objects in complex environments, and meets the adaptive grasping needs of humanoid robots in diverse application scenarios.
Smart Images

Figure CN123103697A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of humanoid robot technology, and in particular to a robotic arm grasping decision-making method and system based on a grasping neural network. Background Technology
[0002] Currently, the gripping force control of humanoid robots' robotic hands when grasping objects typically employs a threshold judgment scheme based on tactile feedback. This scheme uses tactile sensors placed at the tips of the robotic fingers to detect the contact force between the fingertips and the object in real time during the grasping process. When the contact force reaches a preset threshold, the system determines that the grasping is complete and stops the finger closing; throughout the grasping process, the gripping force gradually increases at a preset fixed rate until it reaches the preset threshold and then remains constant.
[0003] However, due to the vast differences in the materials, surface friction characteristics, and shapes of objects in real-world applications, the gripping force required for stable grasping varies significantly from object to object. Preset fixed force thresholds are insufficient to cover all possible object types. When encountering undefined objects, the system still configures the gripping force according to the preset threshold, which may result in the object slipping due to insufficient force or damage due to excessive force. This severely restricts the stability of the robotic arm in grasping different objects in complex, unstructured environments. Summary of the Invention
[0004] This application provides a robotic arm grasping decision-making method and system based on a grasping neural network. It enables the robotic arm to dynamically adjust the grasping force and identify the current grasping state based on real-time state feedback information when grasping different objects. This achieves adaptive adjustment of grasping force and decision-making operation, and solves the technical problem in existing grasping control schemes based on fixed thresholds that cause objects to slip or be damaged because the preset threshold cannot cover the different grasping forces required by different objects.
[0005] In a first aspect, embodiments of this application provide a robotic arm grasping decision-making method based on a grasping neural network, comprising: During the process of the robotic arm performing the grasping operation of the target object, the status feedback information corresponding to the current grasping operation is collected in real time. The status feedback information includes the contact status information between the robotic arm and the target object. The state feedback information is input into a pre-trained grasping neural network. After forward inference calculation based on the grasping neural network, the corresponding grasping state recognition result and grasping force adjustment instruction are output. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force and grasping state labels. Based on the grasping status recognition result, the robot arm makes corresponding decision-making operations and adjusts the grasping force of the robot arm according to the grasping force adjustment command.
[0006] Furthermore, the training process for the capture neural network includes: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes sample state feedback data and corresponding gripping force annotations and gripping state annotations. The sample state feedback data comes from the data collected during the gripping process of objects with various attributes, including material, size, shape and surface texture. The training data in the training dataset is input into the grasping neural network, which outputs the corresponding prediction results. The loss function is calculated based on the prediction results and the corresponding gripping force label and grasping state label. The network parameters of the grasping neural network are iteratively adjusted based on the loss function to minimize the loss function until the preset training termination condition is met, and the training ends, resulting in the trained grasping neural network.
[0007] Furthermore, the training dataset is constructed, including: The original sensor signal corresponding to the sample state feedback data is retained, and during the training data preprocessing, randomized noise perturbation is applied to the copy of the original sensor signal. The randomized noise perturbation includes superimposing Gaussian white noise with a standard deviation within a preset range onto the copy of the original sensor signal.
[0008] Furthermore, after adjusting the gripping force of the robotic arm according to the gripping force adjustment command, it also includes: Obtain the execution results of the crawling operation; If the current crawling operation fails based on the execution result, or if the current crawling operation succeeds but the prediction confidence of the crawling state recognition result output by the crawling neural network is lower than the preset confidence threshold, the status feedback information and execution result corresponding to the current crawling operation are recorded as online data. The crawling neural network is incrementally trained periodically based on online data to update its network parameters.
[0009] Furthermore, after outputting the corresponding grasping state recognition result and grasping force adjustment instruction after performing forward inference calculation based on the grasping neural network, it also includes: Obtain the predicted confidence level of the current grasping state recognition result, and compare the predicted confidence level with the preset confidence level threshold; If the prediction confidence level is lower than the confidence level threshold, switch to setting a conservative strategy to control the robotic arm to perform grasping operations. The conservative strategy includes at least one of reducing the finger closing speed, increasing the bending angle of the finger joints, and controlling the fingers to perform reciprocating motions of a set amplitude.
[0010] Furthermore, the grasping state recognition results include stable grasping state, sliding state, and loss of contact state; Based on the grasping state recognition result, the robotic arm performs corresponding decision-making operations, including: If the grasping state recognition result is a sliding state, the gripping force of the robotic arm is increased according to the gripping force adjustment command; If the capture status recognition result indicates a loss of contact, the pre-defined recapture process is triggered. If the grasping state recognition result is a stable grasping state, maintain the current gripping force of the robotic arm, or fine-tune the current gripping force according to the gripping force adjustment command within the preset fine-tuning range, and perform the lifting or transporting operation of the target object.
[0011] Furthermore, before adjusting the gripping force of the robotic arm according to the gripping force adjustment command, it also includes: Obtain preset safety prior configuration information, which includes the maximum permissible output force and maximum finger joint stroke of the robot arm; The candidate target control quantity is calculated based on the current control quantity and gripping force adjustment command of the robot arm, and the candidate target control quantity is compared with the safety range defined by the safety prior configuration information. If any parameter value of the candidate target control quantity exceeds the safe range, the corresponding parameter value of the candidate target control quantity is limited to the safe range, and the limited candidate target control quantity is used to control the robot arm.
[0012] Furthermore, the state feedback information includes at least two of the following: tactile modal data, force modal data, and visual modal data; Before inputting the state feedback information into the grasping neural network, the data of different modalities are time-aligned according to the timestamps corresponding to each modal data, and the data of each modality is normalized. The data of each modality is input into the corresponding feature encoding subnetwork in the grasping neural network to obtain the feature vectors corresponding to each modality. The feature vectors corresponding to each modality are mapped to a feature space of the same dimension, and the mapped feature vectors are weighted and fused according to the confidence weights of each modality data to obtain a comprehensive feature vector for forward inference computation.
[0013] Furthermore, when the robotic arm adopts torque control mode, the gripping force adjustment command includes the target torque increment relative to the current target torque; when the robotic arm adopts position control mode or impedance control mode, the gripping force adjustment command includes the target angle increment relative to the current target angle.
[0014] Furthermore, before extracting the grasping state recognition result from the grasping neural network output, the following steps are also included: The status feedback information of multiple consecutive sampling moments within a preset time window is obtained, and the temporal features are extracted based on the status feedback information. The temporal features include the contact area change rate, the moving speed and direction of the pressure center position, the temporal fluctuation variance of the total normal contact force, and the changing trend of the ratio of tangential force to normal force. The temporal features are input into a pre-trained sliding prediction classifier to obtain a sliding prediction result indicating whether the target object has signs of sliding. If the sliding prediction result indicates the presence of sliding precursors, and the grasping neural network has not yet output the sliding state, increase the gripping force of the robotic arm.
[0015] In a second aspect, embodiments of this application provide a robotic arm grasping decision system based on a grasping neural network, comprising: The information acquisition module is used to collect the status feedback information corresponding to the current grasping operation in real time during the process of the robot arm performing the grasping operation of the target object. The status feedback information includes the contact status information between the robot arm and the target object. The network computing module is used to input state feedback information into a pre-trained grasping neural network, and output the corresponding grasping state recognition result and grasping force adjustment instruction after performing forward inference calculation based on the grasping neural network. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels. The grasping decision module is used to execute the corresponding decision operation of the robotic arm based on the grasping status recognition result, and to adjust the gripping force of the robotic arm according to the gripping force adjustment instruction.
[0016] Furthermore, the information acquisition module includes a tactile information acquisition unit for acquiring contact state information, and at least one of a force information acquisition unit, a visual information acquisition unit, and a joint state acquisition unit; The network computing module includes a feature encoding unit, a feature fusion unit, a state classification unit, and a force adjustment output unit; The capture decision module includes a state decision unit and a safety limiting unit.
[0017] In a third aspect, embodiments of this application provide an electronic device, including: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the robotic arm grasping decision method based on the grasping neural network as described in the first aspect.
[0018] In a fourth aspect, embodiments of this application provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the robotic arm grasping decision method based on a grasping neural network as described in the first aspect.
[0019] This embodiment of the application collects real-time state feedback information corresponding to the current grasping operation during the grasping operation of the robotic arm. The state feedback information includes the contact state information between the robotic arm and the target object. The state feedback information is input into a pre-trained grasping neural network. After forward inference calculation based on the grasping neural network, the corresponding grasping state recognition result and grasping force adjustment command are output. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels. The corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. By employing the aforementioned technical means, the state feedback information of the robotic arm during the grasping operation is collected in real time and input into a pre-trained grasping neural network. After forward inference calculation, the grasping state recognition result and the grasping force adjustment command are output simultaneously. Then, the corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. This allows the robotic arm to dynamically adjust the grasping force and identify the current grasping state based on real-time state feedback information when grasping different objects. This achieves adaptive adjustment of grasping force and decision operation, improves the stability and safety of the robotic arm in grasping different objects in complex unstructured environments, and meets the needs of humanoid robots for adaptive grasping decision in diverse application scenarios. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart of a robotic arm grasping decision-making method based on a grasping neural network, provided in an embodiment of this application; Figure 2 This is a flowchart of the training process of the capture neural network in the embodiments of this application; Figure 3 This is a flowchart of the crawling operation based on a conservative strategy in the embodiments of this application; Figure 4This is a flowchart of the crawling decision-making process in the embodiments of this application; Figure 5 This is a flowchart of the incremental training process of the capture neural network in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a robotic arm grasping decision system based on a grasping neural network provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0023] Example:
[0024] Figure 1 A flowchart of a robotic arm grasping decision-making method based on a grasping neural network, according to an embodiment of this application, is provided. This method can be executed by a robotic arm grasping decision-making device based on a grasping neural network. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Generally, this device can be a control device such as a robotic arm controller.
[0025] The following description uses a robotic arm controller as the main entity executing a robotic arm grasping decision-making method based on a grasping neural network as an example. (Refer to...) Figure 1 The robotic arm grasping decision-making method based on grasping neural networks specifically includes the following steps: Step S110: During the process of the robotic arm performing the grasping operation of the target object, the status feedback information corresponding to the current grasping operation is collected in real time. The status feedback information includes the contact status information between the robotic arm and the target object.
[0026] In this embodiment, the status feedback information includes contact status information between the robotic arm and the target object, which is collected in real time during the robotic arm's grasping operation of the target object. The contact status information is collected by an array of tactile sensors located at the fingertips of each finger of the robotic arm. This array of tactile sensors can be capacitive or piezoresistive. Piezoresistive tactile sensor arrays can use conductive rubber or piezoresistive film as the sensitive material, and the resistance of the sensitive material changes when force is applied. Capacitive tactile sensor arrays can obtain corresponding electrical signals by detecting changes in capacitance caused by changes in electrode spacing, dielectric constant, or effective area due to force. The controller converts the electrical signals output by each sensing unit into corresponding pressure values according to a pre-calibrated correspondence between electrical signals and pressure. Each fingertip tactile sensor contains multiple sensing units arranged in rows and columns. The detection signals from each sensing unit are converted from analog to digital to form a pressure distribution heatmap. The grayscale value of each pixel in the heatmap represents the magnitude of the pressure at that location; a larger grayscale value indicates a greater contact force at that location. The controller continuously acquires the detection signals from each fingertip tactile sensor at a preset sampling frequency, forming a real-time status feedback data stream.
[0027] Then, based on the aforementioned pressure distribution heatmap, contact state information is extracted. The pressure values of all sensing units in the heatmap are integrated and summed to obtain the total normal contact force. For tactile sensors with tangential force detection capability, the tangential force components detected by each effective contact unit are coordinate transformed and vector-summed to obtain the tangential contact force. For tactile sensors without direct tangential force detection capability, features characterizing the tangential sliding trend can be extracted based on changes in pressure center position, contact area, and pressure distribution. Sensing units with pressure values greater than a preset contact threshold in the heatmap are identified as effective contact units. The number of effective contact units is counted, and this number is multiplied by the area of a single sensing unit to obtain the contact area. Using the geometric center of the heatmap as the origin, the coordinates of each effective contact unit are weighted and averaged according to their pressure values. The resulting weighted centroid coordinates are the pressure center position. The controller also uses the original pressure readings, contact forces, contact areas, and pressure center position data of each sensing unit as components of the state feedback information.
[0028] Optionally, the state feedback information may also include force interaction information between the robotic arm and its surrounding environment. This force interaction information is collected by a six-dimensional force sensor installed on the robotic wrist to detect the forces and torques exerted on the robotic wrist and collisions with non-target objects in the environment. By collecting the above state feedback information in real time, the controller can comprehensively perceive the force and contact states of the robotic arm during the grasping process, providing a data foundation for subsequent neural network inference.
[0029] Optionally, the state feedback information includes at least two of tactile modal data, force modal data, and visual modal data. Before inputting the state feedback information into the grasping neural network, the data of different modalities are time-aligned according to the timestamps corresponding to each modal data, and the data of each modality is normalized. The data of each modality are input into the corresponding feature encoding subnetwork in the grasping neural network to obtain the feature vectors corresponding to each modality. The feature vectors corresponding to each modality are mapped to a feature space of the same dimension, and the mapped feature vectors are weighted and fused according to the confidence weights corresponding to each modal data to obtain a comprehensive feature vector for forward inference calculation.
[0030] In addition, the state feedback information can also include visual perception information. Visual perception information is acquired by using an RGB-D camera to capture RGB images and point cloud data of the target object. A pixel mask of the target object is extracted using an instance segmentation model, and this mask is projected onto the point cloud to obtain a subset of the target object's point cloud. Then, visual features such as the bounding box size, spatial position, and orientation of the target object are calculated. Before inputting the multimodal state feedback information into the grasping neural network, the controller acquires the confidence weights corresponding to the tactile, force, and visual modal data respectively. Each confidence weight is pre-calibrated based on the signal quality or historical reliability of the corresponding sensor. The controller can time-align modal data at different sampling frequencies based on the timestamps corresponding to each modal data and normalize the data. When data for any modality is missing, times out, or the signal quality is below a preset standard, the controller reduces the confidence weight of the corresponding modality or performs subsequent inference only based on the remaining valid modalities. The controller inputs each modal data into the corresponding feature encoding sub-network in the grasping neural network, and each feature encoding sub-network outputs the feature vector of its corresponding modality. Based on the confidence weights corresponding to each modality, the feature vectors are weighted and fused to obtain the fused comprehensive feature vector.
[0031] The controller maps the feature vectors of each modality to a feature space of the same dimension through a projection layer and normalizes the confidence weights. Each mapped feature vector is then multiplied by its corresponding confidence weight coefficient, and the weighted feature vectors are summed element-wise to obtain a comprehensive feature vector. Alternatively, the feature vectors of each modality can be combined using feature concatenation, attention fusion, or gating fusion to form the comprehensive feature vector. This comprehensive feature vector serves as state feedback information input to the subsequent fully connected layer of the grasping neural network for forward inference calculation, outputting grasping force adjustment commands and grasping state recognition results. Through this confidence-weighted fusion method, the contribution of different modal information to the final decision is dynamically adjusted according to its reliability. When the signal quality of a certain modality deteriorates due to sensor noise or environmental interference, its corresponding weight coefficient can be reduced accordingly, thereby avoiding the negative impact of low-quality modal information on the fusion result and effectively improving the robustness of multimodal information fusion.
[0032] Step S120: Input the state feedback information into the pre-trained grasping neural network, and output the corresponding grasping state recognition result and grasping force adjustment instruction after performing forward inference calculation based on the grasping neural network; the grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels.
[0033] Furthermore, after acquiring the state feedback information in real time, the controller inputs this information into a pre-trained grasping neural network. The grasping neural network includes a feature encoding part and a multilayer perceptron structure connected to the feature encoding part. When the state feedback information is single-modal data or a pre-extracted unified feature vector, the grasping neural network can also employ a multilayer perceptron structure, using the state feedback information as the input feature vector and performing forward inference calculations through multilayer nonlinear transformations. Each dimension of the input feature vector corresponds to the corresponding dimension data of the state feedback information and the current angle values of each finger joint. The current angle values of each finger joint are obtained in real time through angle sensors or joint encoders installed at each finger joint of the robotic arm. The current angle values of each finger joint can be used as a component of the state feedback information, synchronously input into the grasping neural network along with tactile modal data, force modal data, and visual modal data. The input feature vector undergoes feature extraction and transformation through multiple hidden layers of the grasping neural network. Each hidden layer performs a linear transformation on the input data and then maps it through a nonlinear activation function, abstracting deep-level features related to grasping stability and the physical properties of the object layer by layer. In the output layer of the grasping neural network, the network's calculation results are mapped to the grasping state category space and the grasping force adjustment command value range through a preset output activation function, respectively, outputting the corresponding grasping state recognition result and grasping force adjustment command. The grasping state recognition result is used to characterize the grasping state category between the robot and the target object, and the grasping force adjustment command is used to indicate the amount of torque or angle that should be increased or decreased for each finger joint. Specifically, the state classification node can output the probability corresponding to each grasping state category through the Softmax activation function, and the category corresponding to the highest probability is determined as the grasping state recognition result; the force output node outputs the target torque increment or target angle increment relative to the current joint control quantity.
[0034] The grasping neural network is pre-trained offline based on multiple sets of training data. The training data includes sample state feedback data of objects with different attributes, along with corresponding grasping force and grasping state labels. Objects with different attributes refer to objects with different materials, sizes, shapes, and surface textures to ensure the training data covers the widest possible range of object types. The grasping force label includes the target torque increment or target angle increment required to adjust relative to the current torque or angle value of each finger joint in the corresponding sample state. This label can be determined based on the difference between the target control value required to achieve a stable grasping state and the current control value. The grasping state labels include category labels such as stable grasping state, sliding state, and loss of contact state. Through sufficient data acquisition and labeling, the grasping neural network can learn the mapping relationship between state feedback information and grasping state recognition and grasping force adjustment commands, enabling it to output accurate decision results when facing objects within the training data distribution range.
[0035] Optionally, refer to Figure 2 The training process of a capture neural network includes: Step S1201: Construct a training dataset. The training dataset includes multiple sets of training data. Each set of training data includes sample state feedback data and corresponding gripping force annotation and gripping state annotation. The sample state feedback data comes from the data collected during the gripping process of objects with various attributes, including material, size, shape and surface texture. Step S1202: Input the training data in the training dataset into the grasping neural network, output the corresponding prediction results, calculate the loss function based on the prediction results and the corresponding gripping force label and grasping state label, iteratively adjust the network parameters of the grasping neural network based on the loss function to minimize the loss function, and end the training when the preset training termination condition is met, and obtain the trained grasping neural network.
[0036] In the offline training phase of the grasping neural network, the first step is to construct a training dataset. During this construction, multiple grasping operations are performed on objects with different properties on a robotic arm or data acquisition platform. Sample state feedback data is recorded synchronously during the grasping process, and the corresponding grasping force and grasping state are labeled for each grasping operation. The sample state feedback data can include contact force distribution characteristics collected by tactile sensors and force and torque characteristics collected by a six-dimensional force sensor. When the grasping neural network enables visual modality or joint state input, the sample state feedback data also includes visual perception information and the angle or torque values of each finger joint, ensuring consistency in the data type and dimension of the input during the training and actual inference phases. For grasping force labeling, the current torque or angle value of each finger joint in the current sample state is read, and the target torque or angle value required to achieve the corresponding target grasping state is determined. The difference between the target torque value and the current torque value, or the difference between the target angle value and the current angle value, is used as the grasping force label for that sample. For grasping status labeling, the grasping status category corresponding to that moment is labeled by manual or automated system based on the actual result of the grasping operation and the tactile signal change characteristics during the process.
[0037] Data acquisition covers a wide range of objects with diverse shapes, sizes, weights, and surface materials, including spherical, cylindrical, cubic, flat, and irregularly shaped objects. The training data encompasses a broad range of object types, from fragile and soft objects to hard and smooth ones, and from small, delicate parts to large everyday items, ensuring sufficient diversity to enable the model to generalize to unseen object types. Each training sample corresponds to a specific moment in a grasping action, including sample state feedback data at that moment, along with corresponding gripping force and grasping state annotations. Alternatively, each training sample can correspond to a sequence of state feedback data within a preset time window, allowing the grasping neural network to identify sliding trends based on continuous state changes.
[0038] After constructing the training dataset, a supervised learning approach was used to train the initial grasping neural network. During training, the training data from each group in the training dataset were input into the initial grasping neural network in batches. The initial grasping neural network consists of an input layer, multiple fully connected hidden layers, and an output layer. When the input data includes multiple modalities, the initial grasping neural network also includes feature encoding subnetworks corresponding to each modality. The outputs of each feature encoding subnetwork are fused and then input into the multiple fully connected hidden layers. The input layer receives the feature vectors corresponding to the sample state feedback data, and the dimension of the feature vectors corresponds to the number of features in the input data. After the input layer are multiple fully connected hidden layers, which are stacked sequentially, with the output of the previous hidden layer serving as the input of the next hidden layer. Each hidden layer contains a predetermined number of neurons, and neurons in adjacent layers are fully connected through weight matrices and bias vectors. After each hidden layer, a non-linear activation function is set, using a modified linear unit or its variant as the activation function to introduce non-linear transformation capabilities. The output of the last hidden layer is connected to the output layer, which includes a force output node and a state classification node. The force output node is used to output the grip force adjustment command, and the state classification node is used to output the grip state recognition result.
[0039] After forward propagation, the training data outputs corresponding prediction results, including predicted grip strength adjustment and predicted grasping state. The controller calculates a loss function based on these predictions and the corresponding grip strength and grasping state labels. The loss function includes a grip strength loss term and a grasping state loss term. The grip strength loss term is measured using the mean squared error loss function to assess the deviation between the predicted grip strength adjustment and the labeled grip strength. The grasping state loss term is measured using the cross-entropy loss function to assess the deviation between the predicted grasping state probability distribution and the labeled grasping state. The loss function is a weighted sum of the grip strength and grasping state loss terms, with weights pre-set based on the relative importance of grip strength control and grasping state recognition in the decision-making task. A regularization loss term is then introduced, which is the product of the squares of all model weight parameters and a preset regularization coefficient. This regularization term is used to suppress overfitting and improve the model's generalization ability to unseen objects. The total loss function is the sum of the weighted losses and the regularization losses.
[0040] Based on the total loss function, the backpropagation algorithm is used to calculate the gradient of the total loss function with respect to the weight matrix and bias vector of each layer. An adaptive moment estimation optimizer is then used to iteratively update the weight matrix and bias vector of each layer based on the calculated gradient, causing the loss function value to gradually decrease until convergence. During training, the optimal model parameters are determined by monitoring the loss changes on the validation set. Training is terminated early when the validation set loss no longer decreases over multiple consecutive training epochs to obtain the model parameters with the best generalization performance. After training, the network structure parameters and the trained weight parameters are stored in the controller's memory for online inference during the grasping process. The grasping neural network obtained through this training process can simultaneously output grasping state recognition results and grasping force adjustment instructions based on the input state feedback information, achieving joint prediction from state feedback features to grasping state and grasping force adjustment amount.
[0041] Optionally, a training dataset is constructed, including: During the process of collecting sample state feedback data, the original sensor signal is retained, and randomized noise perturbation is applied to a copy of the original sensor signal during the training data preprocessing process. The randomized noise perturbation includes superimposing Gaussian white noise with a standard deviation within a preset range onto the sensor signal.
[0042] To enhance the generalization ability of the grasping neural network under different sensor noise conditions and environmental interference, a domain randomization strategy is introduced during the training data preprocessing. After collecting sample state feedback data from tactile and force sensors, Gaussian white noise with a standard deviation within a preset range is superimposed on copies of the retained original sensor data. The amplitude of the Gaussian white noise is generated independently at each sampling time according to a preset randomization range, resulting in differentiated distributions of sensor readings under different noise conditions for the same grasping operation. Through domain randomization, the training dataset covers data distributions under various sensor noise conditions, enabling the grasping neural network to learn robustness to sensor noise during training. Furthermore, domain randomization can also include randomization of environmental parameters such as lighting conditions and background texture to expand the distribution range of the training data. Through this domain randomization strategy, the model maintains stable decision-making performance even when facing unavoidable sensor noise and environmental changes in real-world scenarios after deployment, effectively reducing the domain difference between the simulation and real-world environments.
[0043] Optionally, refer to Figure 3 After performing forward inference calculations based on the grasping neural network and outputting the corresponding grasping state recognition result and grasping force adjustment command, it also includes: Step S1203: Obtain the predicted confidence of the current grasping state recognition result, and compare the predicted confidence with the preset confidence threshold. Step S1204: If the prediction confidence level is lower than the confidence level threshold, switch to setting a conservative strategy to control the robotic arm to perform the grasping operation. The conservative strategy includes at least one of reducing the finger closing speed, increasing the bending angle of the finger joint, and controlling the finger to perform a reciprocating motion of a set amplitude.
[0044] The grasping neural network outputs both the grasping state recognition result and a predicted confidence level for that result. This predicted confidence level is determined by the maximum probability value among the probability distributions of each grasping state category output by the state classification node. After acquiring the grasping state recognition result from the grasping neural network, the controller simultaneously acquires this predicted confidence level and compares it with a preset confidence threshold. This threshold is pre-set based on the security requirements of the application scenario; higher security requirements correspond to higher confidence thresholds, typically ranging from 0.7 to 0.9. A higher predicted confidence level indicates greater certainty in the grasping neural network's prediction of the current state classification, resulting in higher decision reliability; conversely, a lower predicted confidence level indicates greater uncertainty regarding the current input, leading to lower decision reliability.
[0045] Entropy can also be used as a quantitative assessment method for prediction uncertainty. Specifically, when the grasping neural network outputs the grasping state recognition result, the state classification node outputs the probability value corresponding to each grasping state category, and the sum of the probability values is normalized to 1. The controller calculates the entropy value based on the probability value corresponding to each grasping state category. The calculation formula is to take the logarithm of each probability value to the base of the natural constant, multiply each probability value by its logarithm, take the negative of the product, and then sum the products corresponding to each grasping state category. The smaller the entropy value, the more concentrated the probability distribution and the higher the prediction certainty; the larger the entropy value, the more dispersed the probability distribution among multiple state categories and the higher the prediction uncertainty. The controller compares the calculated entropy value with a preset entropy threshold. When the entropy value is lower than the preset entropy threshold, the prediction confidence is deemed to meet the requirements; when the entropy value is higher than or equal to the preset entropy threshold, the prediction confidence is deemed to not meet the requirements.
[0046] When the controller determines that the prediction confidence level is lower than the preset confidence level threshold, it indicates that the reliability of the current decision of the grasping neural network is insufficient. At this time, the controller switches to a conservative strategy to control the robot to perform the grasping operation. The conservative strategies include: 1. Reducing the finger closing speed by multiplying the target speed of the finger joint by a preset deceleration factor less than 1, typically 0.5, to make finger movements smoother and avoid damage or slippage caused by rapid movements; 2. Increasing the bending angle of the finger joint within the joint travel and maximum allowable output force range defined by the safety prior configuration information to attempt to increase the contact area between the finger and the target object. This is achieved by adding a preset angle increment to the finger joint angle at the current position, typically 3 to 5 degrees, to further bend the finger and increase the contact area with the target object, thereby improving grip stability by increasing the contact area in the absence of precise force control information; stopping the increase of bending angle when the contact force or joint torque reaches the corresponding safety limit; 3. Controlling the finger to perform small reciprocating movements by performing reciprocating movements within a preset range near the current position to obtain additional tactile information, and using the dynamic changes in tactile signals during the reciprocating movements to reduce the uncertainty of the grasping neural network prediction. The amplitude, speed, and number of reciprocating movements are all limited within a preset safety range, and the reciprocating movement stops when the target object is detected to be sliding, colliding, or the contact force exceeds the safety limit. Through this conservative strategy, the system can continue to perform the grasping operation in a safe mode when the neural network decision is unreliable, effectively improving the system's safety redundancy.
[0047] Step S130: Execute the corresponding decision operation of the robot arm according to the grasping state recognition result, and adjust the grasping force of the robot arm according to the grasping force adjustment instruction.
[0048] After the grasping neural network outputs the grasping state recognition result and the grasping force adjustment command, the controller executes the corresponding decision operation of the robot hand based on the grasping state recognition result, and adjusts the gripping force of the robot hand according to the gripping force adjustment command. The grasping state recognition result is used to characterize the current grasping state category of the robot hand, and different state categories correspond to different decision operations. The gripping force adjustment command is the target torque increment or target angle increment of each finger joint, used to instruct the controller to increase, decrease, or maintain the gripping force of each finger based on the current gripping force. The controller converts the gripping force adjustment command into specific joint control commands and sends them to the servo drivers of each finger joint for execution. Specifically, when the robot hand adopts torque control mode, the gripping force adjustment command includes the target torque increment; when the robot hand adopts position control mode or impedance control mode, the gripping force adjustment command includes the target angle increment.
[0049] Optionally, when the robot arm uses torque control mode, the gripping force adjustment command includes the target torque increment relative to the current target torque; when the robot arm uses position control mode or impedance control mode, the gripping force adjustment command includes the target angle increment relative to the current target angle. In this way, the physical quantity carried by the gripping force adjustment command matches the actual control mode used by the robot arm, avoiding the mixing of torque increments and angle increments belonging to different physical quantities in the same command. The controller can directly superimpose the adjustment command onto the corresponding current control quantity and send it to the joint actuator for execution, enabling the robot arm to correctly implement gripping force adjustment in different control modes. As an optional implementation, the controller can also pre-store the correspondence between each control mode and the type of adjustment quantity, and determine the type of adjustment quantity after reading the robot arm's current control mode identifier; if the robot arm supports control mode switching, the controller synchronously updates the parsing method of the adjustment quantity at the time of mode switching.
[0050] Optionally, refer to Figure 4 The grasping state recognition results include stable grasping state, sliding state, and loss of contact state; correspondingly, the robot arm performs corresponding decision operations based on the grasping state recognition results, including: Step S1301: If the grasping state recognition result is a sliding state, increase the grasping force of the robotic arm according to the grasping force adjustment command; Step S1302: If the grab state recognition result is a lost contact state, trigger the set re-grab process; Step S1303: If the grasping state recognition result is a stable grasping state, maintain the current gripping force of the robotic arm, or fine-tune the current gripping force according to the gripping force adjustment instruction limited to the preset fine-tuning range, and perform the lifting or transporting operation of the target object.
[0051] Among them, steps S1301, S1302 and S1303 are mutually exclusive branches to be executed according to different grasping states, rather than sequential steps to be executed in turn.
[0052] When the grasping neural network identifies a sliding state, it indicates that the target object has slipped relative to the robotic arm's fingers, and the current gripping force is insufficient to overcome the tangential force generated by the object's gravity or external disturbances. In this case, the controller increases the robotic arm's gripping force based on the gripping force adjustment command output by the grasping neural network. In the sliding state, the controller restricts the gripping force adjustment command to directions that increase or maintain the gripping force. Specifically, based on the target angle increment of each finger joint in the gripping force adjustment command, this increment is superimposed on the current angle of each finger joint, causing the fingers to bend further, increasing the contact pressure between the fingers and the target object's surface, thereby increasing friction to suppress slippage. When the robotic arm uses torque control mode, the controller superimposes the target torque increment on the current target torque and sends the superimposed target torque to the joint actuators; when the robotic arm uses position control mode or impedance control mode, the controller superimposes the target angle increment on the current target angle. Alternatively, given the joint stiffness parameters or impedance control parameters, the controller can convert between the target torque increment and the target angle increment based on a preset torque-angle mapping relationship. Through this closed-loop adjustment mechanism, the robotic arm can increase its gripping force in a timely manner when it detects a slipping tendency, effectively preventing objects from slipping.
[0053] When the grasping neural network identifies a loss of contact as the current grasping state, it indicates that the robotic arm's fingers have lost contact with the target object, and the grasping operation has failed. Specifically, all fingertip tactile sensors detect contact forces below a preset contact detection threshold, and this state persists for more than a preset confirmation time. Upon determining a loss of contact, the controller triggers a pre-defined re-grasping procedure. This involves the controller controlling the robotic arm's fingers to appropriately open to their initial open posture, controlling the end effector to retract to a preset initial position or grasping preparation position, re-acquiring the updated six-DOF pose of the target object through the visual perception module, replanning the motion trajectory based on the updated pose, and re-executing the grasping operation. After initiating the re-grasping procedure, the controller stops executing the current grasping force adjustment command, and the re-grasping procedure generates motion control commands for controlling the robotic arm to retract, reposition, and re-grasp. Through this re-grasping procedure, the robotic arm can automatically recover and retry after a grasping failure without manual intervention.
[0054] When the grasping neural network identifies the current grasping state as stable, it indicates that the robotic arm has firmly grasped the target object, and the current gripping force is sufficient to maintain the object's stability in the grasping posture. Specifically, if the tactile sensors of at least two fingertips detect continuous contact, and the variance of the contact area and contact force within a preset time period is lower than a preset stability threshold, then the stable grasping state output by the grasping neural network is determined to meet the stability verification conditions. In the case of a stable grasping state, the controller maintains the current gripping force of the robotic arm, that is, maintains the current angle and torque output of each finger joint, while controlling the end effector of the robotic arm to perform lifting or transporting operations to move the target object to the target position. In the stable grasping state, the controller can set the gripping force adjustment command to zero or limit it to a preset fine-tuning range. During lifting or transporting, the controller continuously monitors the status feedback information. If slippage is detected again during transport, the gripping force adjustment is re-executed.
[0055] Optionally, before adjusting the gripping force of the robotic arm according to the gripping force adjustment command, the method further includes: Obtain preset safety prior configuration information, which includes the maximum permissible output force and maximum finger joint stroke of the robot arm; The candidate target control quantity is calculated based on the current control quantity and gripping force adjustment command of the robot arm, and the candidate target control quantity is compared with the safety range defined by the safety prior configuration information. If any parameter value of the candidate target control quantity exceeds the safe range, the corresponding parameter value of the candidate target control quantity is limited to the safe range, and the limited candidate target control quantity is used to control the robot arm.
[0056] To ensure operational safety during the grasping process and prevent damage to objects or equipment due to abnormal output of the grasping neural network or excessive gripping force when facing unknown objects, the controller executes a safety priori limiting procedure before actually using the gripping force adjustment command to control the robotic arm. Safety priori configuration information is pre-stored in the controller's memory as a configuration file. This configuration file defines safety priori data entries corresponding to different object categories in a structured data format. Each data entry includes at least an object category identifier, maximum permissible output force, and maximum finger joint travel. The object category of the target object can be determined based on visual recognition results, user input, or pre-configured task information; if the target object category cannot be determined, the controller adopts a preset conservative safety limit corresponding to the unknown object category.
[0057] After receiving the grip strength adjustment command, the controller calculates candidate target control quantities based on the current control quantity of each finger joint and the grip strength adjustment command. In torque control mode, the candidate target control quantity is the sum of the current target torque and the target torque increment; in position control mode or impedance control mode, the candidate target control quantity is the sum of the current target angle and the target angle increment. If none of the candidate target control quantities exceed the corresponding safety range, the candidate target control quantities are sent to the finger joint actuator for execution. If any parameter value in the adjustment command exceeds the corresponding safety threshold, the parameter value is limited to the corresponding upper or lower limit, and an over-limit alarm flag is generated. The limitation range of the safety prior configuration information can be configured according to actual needs and is not fixed here.
[0058] Optionally, refer to Figure 5 After adjusting the gripping force of the robotic arm according to the gripping force adjustment command, it also includes: Step S1304: Obtain the execution result of the crawling operation; Step S1305: If the current crawling operation fails or succeeds and the prediction confidence of the crawling state recognition result output by the crawling neural network is lower than the preset confidence threshold, record the status feedback information and execution result corresponding to the current crawling operation as online data. Step S1306: Periodically perform incremental training on the crawling neural network based on online data to update the network parameters of the crawling neural network.
[0059] After each grasping operation is completed, the controller obtains the execution result information for that grasping operation. The execution result information includes the determination of whether the grasp was successful, as well as key data during the grasping process, such as the maximum contact force, grasping duration, and records of events such as slippage or loss of contact. Among them, the success of the grasp is determined by a comprehensive method of visual confirmation, tactile confirmation, and force confirmation. If the robot arm can stably move the object to the target position after lifting it and does not slip during the movement, the grasp is considered successful; if the object slips during the lifting process or cannot be moved from its original position, the grasp is considered a failure.
[0060] The controller determines whether the data from the current grasping operation is valuable for recording based on the execution result and the prediction confidence level output by the grasping neural network. Data recording is triggered in two scenarios: if the grasping operation fails, this failure case is valuable for incremental training of the model, helping it learn failure patterns; if the grasping operation succeeds but the prediction confidence level of the grasping state recognition result output by the grasping neural network is lower than a preset confidence threshold, this low-confidence success case reflects insufficient decision-making certainty of the model under the current input, and including this data in the training set can help the model improve its certainty on this type of data distribution. On the other hand, when the grasping neural network outputs the grasping state recognition result, the state classification node outputs the probability value corresponding to each grasping state category. The sum of these probability values is normalized to one, and the highest probability value is the prediction confidence level. Under the above recording conditions, the controller records the timestamped state feedback information corresponding to the current grasping operation, the current control quantity of each finger joint, the grasping force adjustment command, the prediction probability of each grasping state category, and the execution result as online data, stored in the controller's online data cache.
[0061] Once the online data accumulates to a set amount, the controller periodically initiates an incremental training process. This process incrementally trains the crawling neural network based on the accumulated online data to update the network parameters. The frequency of incremental training is preset according to the rate of online data accumulation, such as once every 24 hours or once after accumulating a preset number of online data points. During each incremental training iteration, a batch of data is sampled from the accumulated online data. This batch of online data can be mixed with historical training data extracted from the original training dataset. The current network parameters of the crawling neural network after offline training are used as initial values, and the network parameters are updated using the same loss function and optimizer as in offline training. The learning rate for incremental training is set lower than the initial learning rate in offline training to avoid drastic perturbations to the learned stable parameters. After incremental training is complete, the updated crawling neural network is tested using independent validation data. Only if the updated crawling neural network meets the preset state recognition accuracy, crawling success rate, and safety indicators are the updated network parameters deployed for subsequent online inference, while the network parameters before the update are retained for fallback in case of anomalies. Through incremental training, the grasping neural network can gradually adapt to new object types and scene changes in the deployment environment, enabling continuous model evolution and constantly improving grasping decision-making performance in specific application scenarios. After incremental training is completed, the network parameters that have been verified to meet the preset update conditions are stored in the controller memory for subsequent online inference.
[0062] Optionally, before the grasping neural network outputs the grasping state recognition result, the method further includes: acquiring state feedback information from multiple consecutive sampling moments within a preset time window; extracting temporal features based on the state feedback information, including the contact area change rate, the moving speed and direction of the pressure center position, the temporal fluctuation variance of the total normal contact force, and the changing trend of the ratio of tangential force to normal force; inputting the temporal features into a pre-trained sliding prediction classifier to obtain a sliding prediction result indicating whether the target object exhibits signs of sliding; and increasing the gripping force of the robotic arm when the sliding prediction result indicates the presence of signs of sliding and the grasping neural network has not yet output the sliding state. In this way, the robotic arm can respond in advance before the grasping neural network gives a sliding state determination, shortening the response delay from the appearance of a sliding trend to the increase in gripping force, and reducing the risk of the target object slipping due to response lag. As an optional implementation, the sliding prediction classifier can also be implemented using a Long Short-Term Memory (LSTM) network or a temporal convolutional network, and supervised training can be performed based on historically collected temporal feature samples and corresponding sliding precursor annotations. When the type of object being grasped is relatively simple, the controller can also directly obtain the sliding prediction result based on the threshold comparison between the contact area change rate and the pressure center movement speed, without enabling the sliding prediction classifier. Specifically, the controller collects a sequence of state feedback information at continuous time intervals with a fixed time window length of 200 to 500 milliseconds. The window contains tactile sensor readings, contact area, pressure center position, and six-dimensional force sensor data from multiple sampling times. The controller extracts temporal features from each modal data within the time window, including the rate of change of the contact area per unit time, the movement speed and direction of the pressure center position per unit time, the temporal variance of the total normal contact force, and the trend of the ratio of tangential force to normal force. The controller concatenates the extracted multimodal temporal features to form a sliding prediction feature vector, which is then input into the pre-trained sliding prediction classifier. The slip prediction classifier is a binary classifier built on a temporal neural network. It takes a slip prediction feature vector as input and outputs the presence or absence of slip precursors. Slip precursors are identified when the contact area continuously decreases at a rate exceeding a preset area change rate threshold, the pressure center moves towards the fingertip edge at a rate exceeding a preset speed threshold, and the ratio of tangential force to normal force continuously increases and approaches the static friction coefficient limit. When the slip prediction classifier identifies slip precursors but the grasping neural network has not yet output a slip state, the controller increases the gripping force in advance to suppress the impending slip, thereby further improving grasping stability.
[0063] Furthermore, before the gripping neural network outputs the gripping force adjustment command, the physical properties of the target object can be identified online, and the target reference value for gripping force adjustment can be adaptively set, making the force adjustment more targeted. Specifically, in the initial contact phase after the robotic arm finger makes contact with the target object, the controller controls the finger to apply a gradually increasing contact force at a preset slow closing speed, while simultaneously collecting feedback data from the tactile sensor and the six-dimensional force sensor. As the contact force gradually increases from zero, the controller records the relationship curve between the contact force and the contact area, and calculates the slope of this curve as a hardness-related parameter used to characterize the surface deformation properties of the target object. Simultaneously, the controller applies a small tangential perturbation when the contact force is stable, and uses the six-dimensional force sensor to detect the ratio of the tangential force to the normal force, estimating the friction coefficient of the target object's surface. When the tangential perturbation is gradually increased until initial slippage is detected, the ratio of the tangential force to the normal force at the initial slippage can be used as an estimate of the static friction coefficient. The controller uses the estimated target object surface hardness-related parameter level and friction-related parameter level as a query index to retrieve the corresponding target gripping force reference value from a preset gripping force parameter table. A gripping force parameter table was pre-established through offline experiments. Each combination of hardness-related parameter levels and friction-related parameter levels in the table corresponds to an optimal gripping force benchmark value. The controller adjusts the increment based on the retrieved target gripping force benchmark value and the gripping force output by the gripping neural network to obtain the final target control quantity. This ensures that the adjustment of gripping force is not only based on the generalization decision of the neural network but also incorporates adaptive optimization based on the online identification results of the physical properties of the current target object, thereby further improving gripping stability.
[0064] As described above, during the process of the robotic arm performing the grasping operation of the target object, the state feedback information corresponding to the current grasping operation is collected in real time. The state feedback information includes the contact state information between the robotic arm and the target object. The state feedback information is input into a pre-trained grasping neural network. After forward inference calculation based on the grasping neural network, the corresponding grasping state recognition result and grasping force adjustment command are output. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels. The corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. By employing the aforementioned technical means, the state feedback information of the robotic arm during the grasping operation is collected in real time and input into a pre-trained grasping neural network. After forward inference calculation, the grasping state recognition result and the grasping force adjustment command are output simultaneously. Then, the corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. This allows the robotic arm to dynamically adjust the grasping force and identify the current grasping state based on real-time state feedback information when grasping different objects. This achieves adaptive adjustment of grasping force and decision operation, improves the stability and safety of the robotic arm in grasping different objects in complex unstructured environments, and meets the needs of humanoid robots for adaptive grasping decision in diverse application scenarios.
[0065] Based on the above embodiments, Figure 6 This is a schematic diagram of a robotic arm grasping decision system based on a grasping neural network, provided as an embodiment of this application. (Reference) Figure 6 The robotic arm grasping decision system based on a grasping neural network provided in this embodiment specifically includes: The information acquisition module 21 is used to collect the status feedback information corresponding to the current grasping operation in real time during the process of the robot arm performing the grasping operation of the target object. The status feedback information includes the contact status information between the robot arm and the target object. Network computing module 22 is used to input state feedback information into a pre-trained grasping neural network, and output the corresponding grasping state recognition result and grasping force adjustment instruction after performing forward inference calculation based on the grasping neural network. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels. The grasping decision module 23 is used to execute the corresponding decision operation of the robot arm according to the grasping state recognition result, and to adjust the grasping force of the robot arm according to the grasping force adjustment instruction.
[0066] Optionally, the information acquisition module 21 includes at least one of a tactile information acquisition unit, a force information acquisition unit, a visual information acquisition unit, and a joint state acquisition unit; the network computing module 22 includes a feature encoding unit, a feature fusion unit, a state classification unit, and a force adjustment output unit; and the grasping decision module 23 includes a state decision unit and a safety limiting unit.
[0067] Specifically, during the grasping operation, the tactile information acquisition unit in the information acquisition module 21 acquires the fingertip contact force distribution, while the force information acquisition unit, visual information acquisition unit, and joint state acquisition unit acquire wrist force interaction information, visual features, and the angles of each finger joint, respectively. These collectively constitute state feedback information and are output to the network computing module 22. The feature encoding unit in the network computing module 22 encodes each modal data to obtain feature vectors, and the feature fusion unit maps each feature vector to the same dimension and weights them according to confidence weights to obtain a comprehensive feature vector. The state classification unit and the force adjustment output unit share this comprehensive feature vector and output the grasping state recognition result and the grasping force adjustment command respectively through forward inference. The state decision unit in the grasping decision module 23 executes the corresponding decision operation based on the grasping state recognition result, and the safety limiting unit calculates the candidate target control quantity based on the current control quantity and the grasping force adjustment command, limits the amplitude, and then controls the robot to adjust the grasping force.
[0068] Specifically, the training process of the capture neural network includes: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes sample state feedback data and corresponding gripping force annotations and gripping state annotations. The sample state feedback data comes from the data collected during the gripping process of objects with various attributes, including material, size, shape and surface texture. The training data in the training dataset is input into the grasping neural network, which outputs the corresponding prediction results. The loss function is calculated based on the prediction results and the corresponding gripping force label and grasping state label. The network parameters of the grasping neural network are iteratively adjusted based on the loss function to minimize the loss function until the preset training termination condition is met, and the training ends, resulting in the trained grasping neural network.
[0069] Specifically, the training dataset is constructed, including: During the process of collecting sample state feedback data, the original sensor signal is retained, and randomized noise perturbation is applied to a copy of the original sensor signal during the training data preprocessing process. The randomized noise perturbation includes superimposing Gaussian white noise with a standard deviation within a preset range onto the sensor signal.
[0070] Specifically, after adjusting the gripping force of the robotic arm according to the gripping force adjustment command, it also includes: Obtain the execution results of the crawling operation; If the current crawling operation fails based on the execution result, or if the current crawling operation succeeds but the prediction confidence of the crawling state recognition result output by the crawling neural network is lower than the preset confidence threshold, the status feedback information and execution result corresponding to the current crawling operation are recorded as online data. The crawling neural network is incrementally trained periodically based on online data to update its network parameters.
[0071] Specifically, after outputting the corresponding grasping state recognition result and grasping force adjustment instruction after performing forward inference calculation based on the grasping neural network, the process also includes: Obtain the predicted confidence level of the current grasping state recognition result, and compare the predicted confidence level with the preset confidence level threshold; If the prediction confidence level is lower than the confidence threshold, a conservative strategy is switched to control the robotic arm to perform the grasping operation. The conservative strategy includes at least one of reducing the finger closing speed, increasing the bending angle of the finger joints, and controlling the fingers to perform reciprocating motions of a set amplitude. Increasing the bending angle of the finger joints and controlling the fingers to perform reciprocating motions are both limited by safety prior configuration information and stop when the target object is detected to be sliding, colliding, or the contact force exceeds the safety limit.
[0072] Specifically, the grasping state recognition results include stable grasping state, sliding state, and loss of contact state; Based on the grasping state recognition result, the robotic arm performs corresponding decision-making operations, including: If the grasping state recognition result is a sliding state, the gripping force of the robotic arm is increased according to the gripping force adjustment command; If the capture status recognition result indicates a loss of contact, the pre-defined recapture process is triggered. If the grasping state recognition result is a stable grasping state, maintain the current gripping force of the robotic arm, or fine-tune the current gripping force according to the gripping force adjustment command within the preset fine-tuning range, and perform the lifting or transporting operation of the target object.
[0073] Specifically, before adjusting the gripping force of the robotic arm according to the gripping force adjustment command, the process also includes: Obtain preset safety prior configuration information, which includes the maximum permissible output force and maximum finger joint stroke of the robot arm; The candidate target control quantity is calculated based on the current control quantity and gripping force adjustment command of the robot arm, and the candidate target control quantity is compared with the safety range defined by the safety prior configuration information. If any parameter value of the candidate target control quantity exceeds the limit range of the safety prior configuration information, the corresponding parameter value of the candidate target control quantity is limited to the limit range corresponding to the safety prior configuration information, and the limited candidate target control quantity is used to control the robot arm.
[0074] Specifically, the state feedback information includes at least two of the following: tactile modal data, force modal data, and visual modal data; Before inputting the state feedback information into the grasping neural network, the data of different modalities are time-aligned according to the timestamps corresponding to each modal data, and the data of each modality is normalized. The data of each modality is input into the corresponding feature encoding subnetwork in the grasping neural network to obtain the feature vectors corresponding to each modality. The feature vectors corresponding to each modality are mapped to a feature space of the same dimension, and the mapped feature vectors are weighted and fused according to the confidence weights of each modality data to obtain a comprehensive feature vector for forward inference computation.
[0075] Specifically, when the robotic arm adopts torque control mode, the gripping force adjustment command includes the target torque increment relative to the current target torque; when the robotic arm adopts position control mode or impedance control mode, the gripping force adjustment command includes the target angle increment relative to the current target angle.
[0076] Specifically, before the grasping neural network outputs the grasping state recognition result, it also includes: The status feedback information of multiple consecutive sampling moments within a preset time window is obtained, and the temporal features are extracted based on the status feedback information. The temporal features include the contact area change rate, the moving speed and direction of the pressure center position, the temporal fluctuation variance of the total normal contact force, and the changing trend of the ratio of tangential force to normal force. The temporal features are input into a pre-trained sliding prediction classifier to obtain a sliding prediction result indicating whether the target object has signs of sliding. If the sliding prediction result indicates the presence of sliding precursors, and the grasping neural network has not yet output the sliding state, increase the gripping force of the robotic arm.
[0077] As described above, during the process of the robotic arm performing the grasping operation of the target object, the state feedback information corresponding to the current grasping operation is collected in real time. The state feedback information includes the contact state information between the robotic arm and the target object. The state feedback information is input into a pre-trained grasping neural network. After forward inference calculation based on the grasping neural network, the corresponding grasping state recognition result and grasping force adjustment command are output. The grasping neural network is pre-trained based on multiple sets of training data, including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels. The corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. By employing the aforementioned technical means, the state feedback information of the robotic arm during the grasping operation is collected in real time and input into a pre-trained grasping neural network. After forward inference calculation, the grasping state recognition result and the grasping force adjustment command are output simultaneously. Then, the corresponding decision operation of the robotic arm is executed according to the grasping state recognition result, and the grasping force of the robotic arm is adjusted according to the grasping force adjustment command. This allows the robotic arm to dynamically adjust the grasping force and identify the current grasping state based on real-time state feedback information when grasping different objects. This achieves adaptive adjustment of grasping force and decision operation, improves the stability and safety of the robotic arm in grasping different objects in complex unstructured environments, and meets the needs of humanoid robots for adaptive grasping decision in diverse application scenarios.
[0078] The robotic arm grasping decision system based on grasping neural network provided in this application embodiment can be used to execute the robotic arm grasping decision method based on grasping neural network provided in the above embodiment, and has corresponding functions and beneficial effects.
[0079] This application also provides an electronic device, see embodiments thereof. Figure 7 The electronic device includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The electronic device may have one or more processors and one or more memories. The processor, memory, communication module, input device, and output device of the electronic device can be connected via a bus or other means.
[0080] Memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the grasping decision method for a robotic arm based on a grasping neural network described in any embodiment of this application (e.g., the information acquisition module, network computing module, and grasping decision module in a robotic arm grasping decision system based on a grasping neural network). Memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0081] The communication module is used for data transmission.
[0082] The processor executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in memory, thereby realizing the above-mentioned robotic arm grasping decision-making method based on grasping neural networks.
[0083] Input devices can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the device. Output devices may include display devices such as displays.
[0084] The electronic device provided above can be used to execute the robotic arm grasping decision method based on grasping neural network provided in the above embodiments, and has corresponding functions and beneficial effects.
[0085] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute a robotic arm grasping decision method based on a grasping neural network. The robotic arm grasping decision method based on a grasping neural network includes: during the process of the robotic arm performing a grasping operation on a target object, real-time acquisition of state feedback information corresponding to the current grasping operation, the state feedback information including contact state information between the robotic arm and the target object; inputting the state feedback information into a pre-trained grasping neural network, performing forward inference calculation based on the grasping neural network, and outputting a corresponding grasping state recognition result and a grasping force adjustment instruction; the grasping neural network is pre-trained based on multiple sets of training data, the training data including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels; executing the corresponding decision operation of the robotic arm according to the grasping state recognition result, and adjusting the grasping force of the robotic arm according to the grasping force adjustment instruction.
[0086] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.
[0087] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the gripping decision method of the robotic arm based on the gripping neural network as described above, but can also execute related operations in the gripping decision method of the robotic arm based on the gripping neural network provided in any embodiment of this application.
[0088] The robotic arm grasping decision system, storage medium, and electronic device based on grasping neural network provided in the above embodiments can execute the robotic arm grasping decision method based on grasping neural network provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the robotic arm grasping decision method based on grasping neural network provided in any embodiment of this application.
[0089] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.
Claims
1. A method for robot grasping decision based on a grasping neural network, characterized in that, The method comprises the following steps: During the execution of the target object grasping operation by the robot arm, real-time state feedback information corresponding to the current grasping operation is collected, and the state feedback information comprises contact state information between the robot arm and the target object; The state feedback information is input into a pre-trained grasping neural network, and a corresponding grasping state recognition result and grasping force adjustment instruction are output after forward inference calculation based on the grasping neural network; the grasping neural network is pre-trained based on a plurality of sets of training data, and the training data comprises sample state feedback data of objects with different attributes and corresponding grasping force annotations and grasping state annotations; According to the grasping state recognition result, a corresponding decision operation of the robot arm is executed, and the grasping force of the robot arm is adjusted according to the grasping force adjustment instruction.
2. The method of claim 1, wherein, The training process of the grasping neural network comprises the following steps: A training data set is constructed, the training data set comprises a plurality of sets of training data, each set of training data comprises the sample state feedback data and corresponding grasping force annotations and grasping state annotations, the sample state feedback data is collected during the grasping process of a plurality of objects with different attributes, and the plurality of objects with different attributes comprise material, size, shape and surface texture; The training data in the training data set is input into the grasping neural network, and a corresponding prediction result is output, a loss function is calculated based on the prediction result and corresponding grasping force annotations and grasping state annotations, network parameters of the grasping neural network are iteratively adjusted based on the loss function, the loss function is minimized, and the training is ended when a preset training termination condition is met, and a trained grasping neural network is obtained.
3. The method of claim 2, wherein, The construction of the training data set comprises the following steps: The original sensor signal corresponding to the sample state feedback data is retained, and a randomized noise disturbance is applied to a copy of the original sensor signal during the training data preprocessing process, and the randomized noise disturbance comprises superimposing Gaussian white noise with a standard deviation in a preset range on the copy of the original sensor signal.
4. The method of claim 2, wherein, After the grasping force of the robot arm is adjusted according to the grasping force adjustment instruction, the following steps are further included: An execution result of the grasping operation is obtained; In the case that the current grasping operation fails or the current grasping operation succeeds and the prediction confidence of the grasping state recognition result output by the grasping neural network is lower than a preset confidence threshold, the state feedback information corresponding to the current grasping operation and the execution result are recorded as online data; The grasping neural network is incrementally trained based on the online data to update the network parameters of the grasping neural network.
5. The grasp neural network based robotic hand grasp decision method of claim 1, wherein, After the corresponding grasping state recognition result and grasping force adjustment instruction are output after forward inference calculation based on the grasping neural network, the following steps are further included: The prediction confidence of the current grasping state recognition result is obtained, and the prediction confidence is compared with a preset confidence threshold; In a case where the prediction confidence is lower than the confidence threshold, a conservative strategy is switched to control the robot hand to perform a grasping operation, the conservative strategy including at least one of reducing a finger closing speed, increasing a bending angle of a finger joint, and controlling the finger to perform a set-amplitude reciprocating motion.
6. The method of claim 1-5, wherein, The grasping state recognition result includes a stable grasping state, a sliding state, and a lost contact state. The corresponding decision operation of the robot hand according to the grasping state recognition result includes: In a case where the grasping state recognition result is the sliding state, the grasping force of the robot hand is increased according to the grasping force adjustment instruction; In a case where the grasping state recognition result is the lost contact state, a set re-grasping process is triggered; In a case where the grasping state recognition result is the stable grasping state, the grasping force of the robot hand is maintained, or the grasping force is fine-tuned according to the grasping force adjustment instruction limited in a preset fine-tuning range, and a lifting or carrying operation of the target object is performed.
7. The grasp neural network-based robotic hand grasp decision method of claim 6, wherein, Before the grasping force of the robot hand is adjusted according to the grasping force adjustment instruction, the method further includes: obtaining preset safety prior configuration information, the safety prior configuration information including a maximum allowed output force of the robot hand and a maximum finger joint stroke; calculating a candidate target control quantity according to a current control quantity of the robot hand and the grasping force adjustment instruction, and comparing the candidate target control quantity with a safety range defined by the safety prior configuration information; in a case where any parameter value of the candidate target control quantity exceeds the safety range, limiting the corresponding parameter value of the candidate target control quantity to the safety range, and controlling the robot hand by using the limited candidate target control quantity.
8. The grasp neural network based robotic hand grasp decision method of claim 1, wherein, The state feedback information includes at least two of tactile modality data, force tactile modality data, and visual modality data; Before the state feedback information is input into the grasping neural network, the data of different modalities are time-aligned according to time stamps corresponding to the modal data, and the data of each modality are normalized; the data of each modality are respectively input into corresponding feature encoding sub-networks of the grasping neural network to obtain feature vectors corresponding to each modality respectively; the feature vectors corresponding to each modality are mapped to a feature space of the same dimension, and the mapped feature vectors are weighted and fused according to confidence weights corresponding to the modal data to obtain a comprehensive feature vector for the forward inference calculation.
9. The grasp neural network based robotic hand grasp decision method of claim 1, wherein, In a case where the robot hand adopts a torque control mode, the grasping force adjustment instruction includes a target torque increment relative to a current target torque; in a case where the robot hand adopts a position control mode or an impedance control mode, the grasping force adjustment instruction includes a target angle increment relative to a current target angle.
10. The grasp neural network based robotic hand grasp decision method of claim 6, wherein, Before the grasping neural network outputs the grasping state recognition result, the method further includes: Obtaining state feedback information at a plurality of sampling time points in a preset time window, extracting time sequence features from the state feedback information, the time sequence features including a contact area change rate, a moving speed and a moving direction of a pressure center position, a time domain fluctuation variance of a total normal contact force, and a change trend of a ratio of a tangential force to a normal force; Inputting the time sequence features into a pre-trained slip prediction classifier to obtain a slip prediction result indicating whether a target object has a slip precursor; In a case where the slip prediction result indicates that there is a slip precursor and the slip state has not been output by the grasping neural network, increasing a grasping force of the robot arm.
11. A robotic gripper decision system based on a grasping neural network, the system comprising: Comprise: An information acquisition module configured to acquire, in real time, state feedback information corresponding to a grasping operation of a target object performed by a robot arm, the state feedback information including contact state information between the robot arm and the target object; A network calculation module configured to input the state feedback information into a pre-trained grasping neural network, and output a grasping state recognition result and a grasping force adjustment instruction based on a forward inference calculation of the grasping neural network; the grasping neural network is pre-trained based on a plurality of sets of training data, the training data including sample state feedback data of objects with different attributes and corresponding grasping force labels and grasping state labels; A grasping decision module configured to perform a corresponding decision operation of the robot arm according to the grasping state recognition result, and adjust the grasping force of the robot arm according to the grasping force adjustment instruction.
12. The robot arm grasping decision system based on a grasping neural network according to claim 11, wherein: The information acquisition module comprises a tactile information acquisition unit for acquiring the contact state information, and at least one of a force information acquisition unit, a visual information acquisition unit, and a joint state acquisition unit; The network calculation module comprises a feature encoding unit, a feature fusion unit, a state classification unit, and a force adjustment amount output unit; The grasping decision module comprises a state decision unit and a safety limiting unit.
13. An electronic device, comprising: Comprise: A memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the robot arm grasping decision method based on a grasping neural network according to any one of claims 1-10.
14. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are configured to perform the robot arm grasping decision method based on a grasping neural network according to any one of claims 1-10.
