Double-arm handover and grabbing attitude optimization method and system based on autoregression strategy

Abstract

The invention provides a double-arm handover and grabbing attitude optimization method and system based on an autoregressive strategy, and relates to the technical field of mechanical arm motion attitude optimizing.The method comprises the steps that an initial handover generation network is constructed, and is used for generating handover units according to input object point cloud and task description; the handover unit comprises a handover posture, a grabbing posture and a termination probability, historical object point cloud, task description and a corresponding handover posture and grabbing posture combined autoregression strategy are obtained to train the network, a target handover generation network is obtained, and during actual execution, the current point cloud and task description are collected in each time step and input into the network, and the target handover generation network is obtained; and a current step handover unit is generated, double-arm motion trail execution is planned, the process is repeated until the termination probability of the current step exceeds a threshold value, generation is stopped, optimization of the object grabbing posture is completed, and the problems that an existing double-arm handover method is low in planning efficiency and poor in environmental adaptability, and autonomous operation and assembly capacity improvement of a robot are limited are solved.

Description

Technical Field

[0001] This invention relates to the field of robotic arm motion posture optimization technology, and in particular to a method and system for optimizing the handover and grasping posture of two arms based on an autoregressive strategy. Background Technology

[0002] In recent years, with the rapid development of industrial automation and service robot technologies, the ability of robots to autonomously complete tasks in complex environments has attracted widespread attention. In industrial scenarios such as flexible manufacturing, warehousing and logistics, and furniture assembly, as well as service applications such as home services and medical assistance, robots typically need to grasp objects in a posture that conforms to task constraints in order to perform assembly, insertion, or usage tasks. This is of great significance for improving production efficiency, reducing labor costs, and expanding the application scope of robots. In industrial scenarios, it enables more precise and efficient assembly and logistics operations; in service applications, it better meets people's needs in home services and medical assistance. Using robots to complete these tasks also reduces errors and risks caused by human factors, improving overall work quality and safety. Simultaneously, it promotes the development of related industries towards intelligence and automation, bringing more possibilities and development opportunities to the future of industry and service sectors.

[0003] In robot manipulation tasks, to address the issue of adjusting the object grasping posture, existing dual-arm handover methods typically plan the handover process based on a regrasp graph. The regrasp graph uses the possible poses of the object and their corresponding grasping postures as nodes, and feasible handover relationships as edges between nodes. The dual-arm handover plan is generated by searching paths in the graph. Each searched path corresponds to a series of grasping and handover operations, which can be used to guide the two arms to collaboratively adjust the object's pose and grasping posture. This method can achieve the adjustment of the object's pose and grasping posture to a certain extent. In addition, this type of method is combined with offline planning, that is, the complete grasping and handover operation process is determined in advance so that the robot can execute the task according to the preset steps.

[0004] However, the above-mentioned technical solutions have significant drawbacks: as the number of possible grasping postures and poses of the object increases, the node size and search space of the regrasping graph expand rapidly, leading to a significant increase in the computational resources and time consumed during the planning process. Moreover, such methods typically employ a one-time offline planning approach, meaning that the complete grasping and handover operations are determined before execution. When the object's pose deviates or the environment changes during execution, it is often necessary to reconstruct or search for a new grasping graph, making it difficult to adjust and adapt the handover process in real time. This results in low efficiency for the robot in actual operation, making it unable to cope well with complex and changing environments and task requirements, and limiting the improvement of the robot's autonomous operation and assembly capabilities. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, the present invention aims to provide a method for optimizing the handover and grasping posture of two robotic arms based on an autoregressive strategy. This method fully leverages the collaborative operation capabilities of the two robotic arms without relying on large-scale re-grasping maps, achieving efficient and real-time adjustable handover and grasping posture optimization from the initial state until the object is grasped in a posture that meets the task constraints. This effectively enhances the robot's autonomous operation and assembly capabilities.

[0006] The above-mentioned objective of this invention is achieved through the following technical solution: A method for optimizing the handover and grasping posture based on an autoregressive strategy includes: An initial handover generation network is constructed, which is used to generate handover units based on the input object point cloud and task description. The handover unit includes handover pose, grasping pose and termination probability. Acquire historical object point clouds, task descriptions, and corresponding handover poses and grasping postures; construct a training set; and train the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network. At the current time step, collect the object point cloud and task description, input them into the target handover generation network, and obtain the handover unit at the current time step; Based on the handover pose and grasping posture in the handover unit of the current time step, the motion trajectory of the two arms is planned and executed until the termination probability in the handover unit of the current time step is greater than the preset termination probability threshold, at which point generation stops, and the optimization of the object grasping posture is completed.

[0007] By adopting the above technical solution, a handover generation network based on an autoregressive strategy is constructed and trained. This network can generate and execute handover unit sequences with simplified steps and low motion costs online based on real-time perceived object point clouds and task descriptions. This improves the planning efficiency of dual-arm handover tasks. During execution, the handover strategy can be dynamically adjusted according to environmental changes, enhancing the robot's adaptability to object pose deviations and environmental disturbances. Ultimately, this achieves efficient and robust dual-arm grasping posture optimization from the initial state to a posture that meets task constraints, effectively improving the robot's autonomous operation and assembly performance.

[0008] Preferably, the initial handover generation network includes a feature encoding network and an autoregressive decoding network, and the construction of the initial handover generation network includes: The object point cloud and task description are encoded and mapped using the feature encoding network to obtain global conditional features; Obtain historical handover units and perform embedding mapping on the historical handover units to obtain historical handover embedding features; The autoregressive decoding network is used to fuse and decode the historical handover features and the global condition features to obtain the handover unit.

[0009] By adopting the above technical solution, the feature encoding network integrates the geometric features of objects and the semantic features of tasks into global conditional features, which comprehensively represent the environmental state and task constraints. Combined with the embedded features of historical handover units for autoregressive decoding, the network can effectively utilize temporal information to generate handover units that meet task requirements and are temporally coherent, which significantly improves the accuracy and continuity of handover planning.

[0010] Preferably, the feature encoding network includes a point cloud encoding unit, a task encoding unit, and a feature fusion mapping unit. The point cloud encoding unit includes a local feature extraction subunit and a global feature generation subunit. The task encoding unit includes a semantic transformation subunit and a semantic understanding subunit. The step of encoding and mapping the object point cloud and task description through the feature encoding network to obtain global conditional features includes: The local feature extraction subunit extracts local geometric features from the object point cloud to obtain local features. The global feature generation subunit performs global feature aggregation on the local features to obtain geometric features; The semantic transformation subunit performs semantic transformation on the task description to obtain a word segmentation sequence; The semantic understanding subunit performs semantic understanding on the segmented word sequence to obtain task features; The geometric features and the task features are fused by the feature fusion mapping unit to obtain global conditional features.

[0011] By adopting the above technical solutions, the point cloud coding unit effectively extracts the local details and overall shape information of the object's geometric structure through local feature extraction and global feature aggregation. The task coding unit accurately parses the semantic connotation of the task description through semantic conversion and semantic understanding. The feature fusion mapping unit organically integrates geometric features and task features to generate global conditional features that simultaneously contain environmental perception information and task constraint information.

[0012] Preferably, the autoregressive decoding network includes a position encoding subunit, a temporal modeling subunit, a feature fusion subunit, and an output mapping subunit. The step of fusing and decoding the historical handover features and the global conditional features through the autoregressive decoding network to obtain the handover unit includes: The historical handover features are obtained by adding location encoding information to the historical handover embedded features through the location encoding subunit; The target handover features are obtained by performing time-series relationship modeling on the historical handover features through the time-series modeling subunit. The feature fusion subunit performs feature fusion on the target intersection feature and the global condition feature to obtain the fused feature; The output mapping subunit performs output mapping on the fused features to obtain the handover unit.

[0013] By adopting the above technical solutions, the location encoding and temporal modeling subunit adds temporal information to historical handover features and captures long-range dependencies, which enhances the network's global understanding of the handover process. The feature fusion and output mapping subunit deeply fuses temporal information with global conditions and decodes them into handover units, realizing end-to-end generation from historical information and current conditions to complete handover decisions.

[0014] Preferably, the training set further includes real termination labels, and the step of training the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network includes: From the training set, obtain object point clouds, task descriptions, and corresponding handover poses and grasping postures at multiple consecutive time steps to construct a sample sequence; For the first time step, the object point cloud and task description of the current time step are input into the initial handover generation network to obtain the prediction handover unit of the first time step. For each subsequent time step, the object point cloud and task description of the current time step, as well as the predicted handover units generated in the previous time steps, are used as historical handover units and input into the initial handover generation network to obtain the predicted handover units of the current time step. A regression loss is constructed based on the difference between the predicted handover unit and the corresponding handover pose and grasping pose at each time step; A termination loss is constructed based on the difference between the predicted termination probability in the handover unit at each time step and the actual termination label. Construct sequence length weights based on the length of the sample sequences; Based on the length of the robotic arm's motion trajectory corresponding to each time step in the sample sequence, a trajectory length weight is constructed. Based on the regression loss, the termination loss, the sequence length weight, and the trajectory length weight, a total loss function is constructed. The initial handover generation network is optimized based on the total loss function to obtain the target handover generation network.

[0015] By adopting the above technical solution, an autoregressive strategy and multi-objective joint optimization are introduced into the training process, enabling the network to generate single-step decisions based on the current state and to master the global handover planning capability by utilizing historical information. At the same time, through the dual constraints of regression loss and termination loss, and by introducing sequence length and trajectory length weights, the network is guided to generate handover unit sequences with fewer steps and lower motion costs, effectively improving the handover efficiency and rationality.

[0016] Preferably, the regression loss is constructed based on the difference between the predicted handover unit at each time step and the corresponding handover pose and grasping pose, including: For each time step, calculate the first difference between the predicted handover pose in the predicted handover unit and the corresponding real handover pose, and calculate the second difference between the predicted grasping pose in the predicted handover unit and the corresponding real grasping pose. Based on the first difference and the second difference, the regression loss for each time step is constructed.

[0017] By adopting the above technical solution, the predicted values ​​of handover pose and grasping pose are independently supervised, enabling the network to accurately learn the two core components of the handover unit. Based on the pose prediction loss, a regression loss is constructed, which effectively improves the accuracy and executability of generating handover units.

[0018] Preferably, the step of planning and executing the dual-arm motion trajectory based on the handover pose and grasping posture in the handover unit of the current time step, until the termination probability in the handover unit of the current time step is greater than a preset termination probability threshold, stops generation, and completes the optimization of the object grasping posture, including: Based on the grasping posture in the handover unit of the first time step, the trajectory lengths of the two robotic arms are planned respectively, and the trajectory length comparison results are obtained. Based on the trajectory length comparison results, the gripping arm is determined, and the motion trajectory of the gripping arm is planned to obtain the initial gripping trajectory; Based on the initial grasping trajectory, the grasping arm is controlled to perform grasping, thereby obtaining the grasped object; For each subsequent time step, the gripping arm of the current time step is determined based on the robotic arm that has grasped the object at the end of the previous time step, and the receiving arm of the current time step is determined based on another robotic arm. Based on the handover pose in the handover unit of the current time step, the handover trajectory of the gripping arm and the gripping trajectory of the receiving arm are planned respectively. Based on the handover trajectory of the grasping arm and the grasping trajectory of the receiving arm, the grasping arm and the receiving arm are controlled to perform movements to obtain the handed-over object; Based on the handed-over object, the original gripping arm releases its gripper and returns to a preset safe position after the handover is completed, thus completing the execution of the current time step. The termination probability in the handover unit of the current time step is compared with a preset termination probability threshold. If the termination probability is greater than the preset termination probability threshold, generation stops, thus completing the optimization of the object gripping posture.

[0019] By adopting the above technical solution, the gripping arm is adaptively selected in the initial grasping phase to shorten the execution time, and the roles of the two arms are dynamically assigned and the trajectory is precisely planned in the subsequent handover, achieving efficient handover through dual-arm collaboration. Simultaneously, the handover process is stopped promptly by judging the termination probability in real time, ensuring that the grasping is completed in a posture that meets the task requirements, effectively avoiding redundant operations.

[0020] The second objective of this invention is to provide a dual-arm handover and grasping posture optimization system based on an autoregressive strategy.

[0021] The second objective of this invention is achieved through the following technical solution: A dual-arm handover and grasping posture optimization system based on an autoregressive strategy includes: An initial network construction module is used to construct an initial handover generation network. The initial handover generation network is used to generate handover units based on the input object point cloud and task description. The handover unit includes handover pose, grasping pose, and termination probability. The target network training module is used to acquire historical object point clouds, task descriptions, and corresponding handover poses and grasping postures to construct a training set, and to train the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network. The handover unit generation module is used to collect object point cloud and task description at the current time step, input them into the target handover generation network, and obtain the handover unit at the current time step. The trajectory planning and execution module is used to plan and execute the movement trajectory of both arms based on the handover pose and grasping posture in the handover unit of the current time step, until the termination probability in the handover unit of the current time step is greater than a preset termination probability threshold, at which point generation stops, thus completing the optimization of the object grasping posture.

[0022] By adopting the above technical solutions, the modules work together to achieve a complete closed loop from environmental perception and decision generation to motion execution, enabling the robot to efficiently and robustly complete the tasks of handing over the two arms and optimizing the grasping posture, effectively improving the robot's autonomous operation and assembly capabilities in complex environments.

[0023] The third objective of this invention is to provide an electronic device.

[0024] The above-mentioned objective three of this invention is achieved through the following technical solution: An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed the above-described autoregressive strategy-based dual-arm handover and grasping posture optimization method.

[0025] The fourth objective of this invention is to provide a computer-readable storage medium capable of storing corresponding programs.

[0026] The fourth objective of this invention is achieved through the following technical solution: A computer-readable storage medium storing a computer program that can be loaded by a processor and executed by the autoregressive strategy-based method for optimizing the hand-arm handover and grasping posture described above.

[0027] In summary, the present invention has at least one of the following beneficial technical effects: This invention first constructs a handover generation network. This network takes object point clouds and task descriptions as input and generates handover units containing handover poses, grasping postures, and termination probabilities. Then, using expert-annotated dual-arm handover examples, an autoregressive strategy is employed to train the network. Combining sequence length weights and trajectory length weights, the network is guided to generate handover unit sequences with fewer steps and lower overall motion costs. Next, based on the trained handover generation network, handover units are generated online according to real-time perceived object point clouds and task descriptions. The motion trajectories of the dual robotic arms are planned and executed accordingly, thereby achieving gradual adjustment and optimization of the object grasping posture. The handover process ends when the termination probability reaches a threshold, ensuring the object is grasped in a posture that conforms to task constraints. This invention eliminates the need to construct large-scale re-grasping maps, and during execution, the handover generation network can dynamically generate handover units based on real-time object point clouds, achieving adaptive adjustments to environmental changes. This improves the robustness of dual-arm handover execution and ultimately enhances the robot's autonomous operation and assembly capabilities. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the steps of a method for optimizing the handover and grasping posture based on an autoregressive strategy, as provided in Embodiment 1 of the present invention.

[0029] Figure 2 This is a schematic diagram of the initial handover generation network of a dual-arm handover and grasping posture optimization method based on an autoregressive strategy provided in Embodiment 1 of the present invention.

[0030] Figure 3 This is a specific implementation process of a method for optimizing the handover and grasping posture based on an autoregressive strategy, provided in Embodiment 1 of the present invention.

[0031] Figure 4This is a structural block diagram of a dual-arm handover and grasping posture optimization system based on an autoregressive strategy provided in Embodiment 2 of the present invention. Detailed Implementation

[0032] This invention provides a method and system for optimizing the handover and grasping posture of dual-arm robots based on an autoregressive strategy. This addresses the technical problem of low efficiency in existing robots during actual operation, their inability to cope well with complex and changing environments and task requirements, and the resulting limitations on the improvement of robot autonomous operation and assembly capabilities. Without relying on large-scale re-grasping maps, this invention fully leverages the collaborative operation capabilities of the two robotic arms to achieve efficient and real-time adjustable handover and grasping posture optimization from the initial state until the robot grasps objects in a posture that conforms to task constraints, effectively improving the robot's autonomous operation and assembly capabilities.

[0033] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] It should be noted that, in the embodiments of this invention, when the relevant object information and other related data are used in specific products or technologies, permission or consent from the object is required, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this invention involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the individual's consent; if sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0035] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure.

[0036] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship. Example

[0037] Please see Figures 1 to 3 The present invention provides a method for optimizing the handover and grasping posture of two arms based on an autoregressive strategy, comprising: Step 101: Construct an initial handover generation network. The initial handover generation network is used to generate handover units based on the input object point cloud and task description. The handover units include handover pose, grasping pose, and termination probability.

[0038] The initial handover generation network is an end-to-end prediction model that takes object point clouds and task descriptions as input and outputs the handover units required at the current time step to guide the handover decisions for each arm.

[0039] An object point cloud is a set of three-dimensional point coordinates on the surface of an object obtained by collecting data from a depth camera and preprocessing it. It is used to represent the current geometry, spatial position, and orientation of the object.

[0040] The task description is an operational objective given in natural language, such as "grab a hammer and strike it", which is used to constrain the subsequent operational requirements that the final grasping posture must meet.

[0041] The handover unit is the core data structure output by the network, which includes three components: handover pose, grasping pose, and termination probability. It is used to guide the handover operation of the two arms at the current time step.

[0042] The handover pose refers to the position and orientation of an object at the moment the two arms meet. It is usually represented by six degrees of freedom and is used to plan the target position to which the gripper arm will move the object.

[0043] The grasping posture is the position and attitude of the end effector of the robotic arm when grasping an object at the junction posture. It is represented by six degrees of freedom. This posture is not pre-attached to the robotic arm, but is assigned to the grasping arm or receiving arm according to actual needs during execution.

[0044] The termination probability is a scalar value between 0 and 1, which is predicted and output by the network. It is used to characterize whether the current time step is the last step of the handover unit sequence. When the value exceeds a preset threshold, it means that the task constraints have been met and subsequent handover generation stops.

[0045] It should be noted that "dual arms" refers to two robotic arms working together (such as the dual arms of a robot, or two independently installed robotic arms). During the handover process, they respectively assume the roles of a grasping arm and a receiving arm. The grasping arm is responsible for holding the object, adjusting its posture, and moving the object to the handover position. The receiving arm is responsible for grasping the object at the handover position to continue subsequent operations. The two work together alternately to gradually optimize the object grasping posture.

[0046] Preferably, step 101 can be implemented using the following sub-steps: S11. Encode and map the object point cloud and task description through a feature coding network to obtain global conditional features.

[0047] The feature encoding network is a neural network module consisting of point cloud encoding units and task encoding units connected in parallel and then connected to a feature fusion mapping unit. It is used to uniformly encode multimodal input information into fixed-dimensional global conditional features. The global conditional features are joint feature vectors that integrate object geometric information and task semantic constraints. They also contain spatial structural information of objects in the current scene and the constraints of the task target on the grasping posture, providing a comprehensive environment and task context for the generation of the handover unit.

[0048] Preferably, S11 may include the following sub-steps: The feature coding network includes point cloud coding units, task coding units, and feature fusion mapping units. The point cloud coding unit includes local feature extraction subunits and global feature generation subunits, and the task coding unit includes semantic transformation subunits and semantic understanding subunits.

[0049] The feature encoding network consists of point cloud encoding units, task encoding units, and feature fusion mapping units, and is used to encode multimodal inputs into global conditional features.

[0050] The point cloud coding unit is a sub-network in the feature coding network responsible for processing the point cloud of an object. It is composed of a local feature extraction sub-unit and a global feature generation sub-unit connected in series, and is used to extract feature vectors representing the geometric structure of an object from the 3D point cloud.

[0051] The task encoding unit is a sub-network in the feature encoding network responsible for processing task descriptions. It consists of a semantic transformation sub-unit and a semantic understanding sub-unit connected in series. It is used to extract semantic feature vectors that represent task constraints from task descriptions in natural language form.

[0052] The feature fusion mapping unit is a neural network layer in the feature encoding network responsible for fusing multimodal features. It is used to concatenate or weightedly fuse the geometric features output by the point cloud encoding unit with the task features output by the task encoding unit, and then map them into a unified global conditional feature through a fully connected layer.

[0053] The local feature extraction subunit is the first component of the point cloud coding unit. It is used to independently perform feature transformation on each point in the input point cloud, extract the local geometric structure information in the neighborhood of each point, and output the local feature vector corresponding to each point.

[0054] The global feature generation subunit is the second component in the point cloud coding unit. It is used to aggregate the local features of all points output by the local feature extraction subunit and extract the global feature vector through methods such as max pooling to represent the geometric structure of the entire point cloud.

[0055] The semantic transformation subunit is the first component of the task encoding unit. It is used to perform preprocessing such as word segmentation and encoding on the task description in natural language form, convert the text into a numerical representation that the model can process, and output word segmentation sequence or word embedding vector.

[0056] The semantic understanding subunit is the second component of the task coding unit. It is used to perform deep semantic understanding on the word segmentation sequence output by the semantic transformation subunit. It extracts the semantic constraint information contained in the task description through stacked Transformer coding layers and outputs the task feature vector.

[0057] S111. Local geometric features are extracted from the object point cloud through the local feature extraction sub-unit to obtain local features.

[0058] Local features are high-dimensional feature vectors extracted for each individual point in a point cloud. They are used to characterize the geometric structure information of that point and its surrounding local neighborhood, such as the curvature of the surface near the point, the change of the normal vector, and the local point cloud distribution density.

[0059] In this embodiment of the invention, the local feature extraction subunit uses the PointNet architecture to process the input point cloud: the input point cloud contains N points (N=1024), each point uses three-dimensional coordinates (x,y,z) as initial features, and is mapped to 64-dimensional, 128-dimensional and 256-dimensional feature spaces layer by layer through three convolutional layers. After each convolution, the ReLU activation function is used, and finally the 256-dimensional local feature vector of each point is output.

[0060] S112. Geometric features are obtained by aggregating local features into global features through global feature generation sub-units.

[0061] Geometric features are global feature vectors obtained by aggregating the local features of all points in a point cloud. They are used to characterize the overall geometric structure of an object, such as its global shape, size distribution, and overall spatial structure.

[0062] In this embodiment of the invention, the sub-unit first performs a cross-channel global max pooling operation to take the maximum value of each feature channel in the dimension of the point, and aggregates the N×256 local feature tensors into a 1×256 global feature vector, thereby extracting the global geometric information of the entire point cloud. Then, the 256-dimensional global feature vector obtained by pooling is input into a fully connected network layer for feature enhancement, and the final 256-dimensional geometric feature vector is output.

[0063] S113. The task description is semantically transformed through the semantic transformation subunit to obtain the word segmentation sequence.

[0064] Word segmentation sequences are sequences of word indices obtained by segmenting natural language task descriptions into semantic units, and are used to convert text information into numerical representations that can be processed by the model.

[0065] In this embodiment of the invention, the semantic transformation subunit uses the BERT word segmenter to process the task description: first, the natural language short sentence (such as "grasp the hammer to strike") is segmented into words, each word is mapped to the corresponding index in the pre-trained vocabulary, and start marker [CLS] and end marker [SEP] are added; finally, a fixed-length word segmentation sequence is output (such as a preset length of 64), and any insufficient part is padded with zeros.

[0066] S114. Semantic understanding of the word segmentation sequence is performed through the semantic understanding subunit to obtain task features.

[0067] Task features are feature vectors obtained by deep semantic encoding of word segmentation sequences. They are used to represent the operational goals and constraints contained in the task description, such as the type of tool to be captured and the semantic connotations of subsequent actions.

[0068] In this embodiment of the invention, the semantic understanding subunit employs a two-layer Transformer-based encoder architecture to perform semantic understanding on the segmented sequence. Each layer includes a multi-head self-attention sublayer and a fully connected sublayer. Semantic features are extracted layer by layer through stacked encoding, ultimately outputting a 64-dimensional task feature vector. The calculation process of the multi-head self-attention sublayer is shown below:

[0070] In the formula, Q, K, and V represent the query, key, and value vectors obtained from the input feature mapping, respectively. Let be the dimension of the key vector. Let T be the normalized exponential function, and T be the transpose. The output of the attention mechanism represents the feature representation obtained by weighting and summing the values ​​based on the similarity between the query and the key.

[0071] S115. Geometric features and task features are fused through the feature fusion mapping unit to obtain global conditional features.

[0072] In this embodiment of the invention, the feature fusion mapping unit concatenates geometric features and task features along the feature dimension to obtain a 320-dimensional concatenated feature vector. Then, the concatenated feature is input into a fully connected layer for feature fusion and dimension transformation, and the final global conditional feature is output, which maintains a dimension of 320.

[0073] S12. Obtain historical handover units and perform embedding mapping on historical handover units to obtain historical handover embedding features.

[0074] Historical handover units refer to handover units that have been generated and executed before the current time step. They contain three parts of information: handover pose, grasping posture, and termination probability, and are used to provide historical context reference for the current handover generation.

[0075] Historical handover embedding features are high-dimensional feature vectors obtained by mapping historical handover units through an embedding layer. They are used to convert discrete handover unit data into a continuous representation that can be processed by a neural network, which facilitates subsequent time series modeling and feature fusion.

[0076] In this embodiment of the invention, the handover units of the most recent 4 time steps are first obtained from the historical cache as historical handover units. If there are less than 4 steps, they are filled with zero vectors. Each historical handover unit contains 13-dimensional original data (6-dimensional handover pose, 6-dimensional grasping pose, and 1-dimensional termination probability). It is mapped to a 256-dimensional feature vector through two layers of fully connected network (13→128→256). Each layer is followed by a ReLU activation function, and finally a K×256-dimensional (K=4) historical handover embedding feature tensor is obtained.

[0077] S13. The historical handover features and global condition features are fused and decoded through an autoregressive decoding network to obtain the handover unit.

[0078] Preferably, S13 may include the following sub-steps: The autoregressive decoding network includes a position encoding subunit, a temporal modeling subunit, a feature fusion subunit, and an output mapping subunit.

[0079] The autoregressive decoding network is a neural network module based on the Transformer architecture. It takes historical handover features and global conditional features as inputs and generates handover units step by step through an autoregressive approach.

[0080] It should be noted that the network includes a position encoding subunit at the input end, a five-layer stacked decoding layer, and an output mapping subunit at the output end. Each decoding layer contains a temporal modeling subunit, a feature fusion subunit, and a fully connected subunit. It uses historical generation information to guide the current step prediction, ensuring the temporal coherence and task consistency of the output sequence.

[0081] The position encoding subunit is a module in the autoregressive decoding network used to add temporal position information to the historical handover features. It generates a position encoding vector through a sine function and adds it to the historical handover embedded features, enabling the network to perceive the relative or absolute positional relationship of different time steps in the sequence.

[0082] The temporal modeling subunit is a module in the autoregressive decoding network used to capture the temporal dependencies between historical handover units. It models the historical handover features after adding positional encoding based on the multi-head self-attention mechanism, enabling the network to focus on long-range dependencies and contextual information between different time steps in the sequence.

[0083] The feature fusion subunit is a module in the autoregressive decoding network used to perform cross-modal fusion of the target handover features after time-series modeling and the global condition features. This unit adopts a cross-attention mechanism, using historical handover features as queries and global condition features as keys and values, to inject environmental and task constraint information into the decoding process, guiding the generated handover units to meet the task requirements.

[0084] The output mapping subunit is a module in the autoregressive decoding network used to decode fused features into specific handover units. It consists of a multi-layer fully connected network and maps high-dimensional fused features to complete handover units that include handover pose, grasping pose, and termination probability.

[0085] S121. Add position coding information to the historical handover embedded features through the position coding sub-unit to obtain the historical handover features.

[0086] Historical handover features are feature representations obtained by superimposing temporal position encoding information on historical handover embedded features. They not only contain the semantic information of historical handover units, but also incorporate the positional relationships of each handover unit in the sequence.

[0087] In this embodiment of the invention, the position encoding subunit uses a sine function to generate a fixed position code, adding temporal position information to each historical handover embedding feature. For the pos-th position in the sequence (pos=0,1,2,3), the 2i-th and 2i+1-th dimensions of its position code are calculated according to the following formulas:

[0088] In the formula, pos is the position index, representing the absolute position number of the current time step in the sequence; i is the dimension index, representing the dimension number in the feature vector; d is the feature dimension, representing the total dimension of the position encoding vector; 2i and 2i+1 are the even and odd dimensions of the feature vector, respectively; 10000 is the fundamental frequency, a constant used to control wavelength changes; and sin and cos are the sine and cosine functions, respectively. This formula generates different encoding vectors for different positions, and there is a linear relationship between the encodings of any two positions, facilitating the network's learning of relative position information. The generated position encoding vector has a dimension of 256, the same as the historical intersection embedding feature. The position encoding vector is element-wise added to the corresponding historical intersection embedding feature to obtain the historical intersection feature that incorporates temporal position information.

[0089] S122. The historical handover features are modeled using a time-series modeling sub-unit to obtain the target handover features.

[0090] The target handover feature is a feature representation obtained by modeling the temporal dependency relationship of the historical handover feature.

[0091] In this embodiment of the invention, the temporal modeling subunit uses stacked multi-head self-attention layers to model the temporal relationship of historical handover features. The specific modeling method is similar to step S114 and will not be described again here.

[0092] S123. The target handover features and global condition features are fused by the feature fusion subunit to obtain the fused features.

[0093] Fusion features are feature representations obtained by cross-modal fusion of target handover features containing historical time-series information and global condition features containing environmental and task constraints.

[0094] In this embodiment of the invention, the feature fusion subunit employs a multi-head cross-attention mechanism to fuse target intersection features and global conditional features: first, the target intersection features are linearly mapped to a query matrix, and the global conditional features are linearly mapped to a key matrix and a value matrix, all three having a dimension of K×256. The cross-attention is calculated as follows:

[0096] In the formula, The query matrix is ​​obtained by linear transformation mapping from the historical handover features H. The key matrix is ​​obtained by linear transformation mapping from the global conditional feature C. The value matrix is ​​obtained by linear transformation mapping from the global conditional feature C. Let be the dimension of the key vector, softmax be the normalized exponential function, and T be the transpose. This is the output of the cross-attention mechanism. This mechanism uses historical information as a query to retrieve the most relevant environmental and task constraints from global conditions and performs weighted aggregation. Notably, this embodiment employs multi-head parallel computation, with each head performing cross-attention operations in different feature subspaces. The outputs of all heads are concatenated and fused through a fully connected layer to obtain intermediate features that fuse historical time series and global constraints. These are then processed through residual connections and layer normalization to finally output a K×256-dimensional fused feature tensor.

[0097] S124. The fused features are output-mapped by the output mapping subunit to obtain the handover unit.

[0098] In this embodiment of the invention, the output mapping subunit first extracts the 256-dimensional feature vector corresponding to the last time step (the current time step) as the basis for decision-making. This vector integrates historical time series information and global conditional constraints. Then, the vector is input into a two-layer fully connected network for mapping: the first layer maps the 256 dimensions to a 128-dimensional hidden layer and follows the ReLU activation function; the second layer maps the 128 dimensions to a 13-dimensional output space to obtain the original data of the handover unit at the current time step. The first 6 dimensions represent the handover pose (position x, y, z and posture rx, ry, rz), the middle 6 dimensions represent the grasping posture (position x, y, z and posture rx, ry, rz), and the last dimension is mapped to a termination probability of 0 to 1 by the Sigmoid function. The final output handover unit is used to guide the two-arm handover operation at the current time step, including the grasping arm moving the object to the handover pose, the receiving arm grasping the object in the grasping posture, and determining whether to terminate the handover process.

[0099] Step 102: Obtain historical object point clouds, task descriptions, and corresponding handover poses and grasping postures to construct a training set. Based on the training set and the autoregressive strategy, train the initial handover generation network to obtain the target handover generation network.

[0100] Autoregressive strategies refer to a specific method for training initial handover generative networks. The core idea is not only to teach the network to make decisions based on the current state, but also to enable it to perform subsequent reasoning and planning based on historical decisions. To achieve this goal, the training process includes the following two complementary modes: 1. Single-step prediction mode: During training, the network uses only the object point cloud and task description of the current time step as input to independently predict the handover unit of that time step. This mode forces the network to learn the ability to directly map from the current environmental state to the optimal single-step action, enabling it to make immediate responses based solely on real-time perception information during the execution phase, thereby effectively dealing with state deviations and environmental disturbances that occur during the execution process.

[0101] 2. Sequence Generation Mode: During training, the actual execution process is simulated, and the network performs autoregressive sequence generation. Specifically, the network first generates the prediction handover unit for the first time step. Subsequently, when generating the prediction handover unit for the second time step, in addition to inputting the object point cloud and task description for the current time step, the prediction handover unit generated in the previous step is also fed back to the network as "historical information" to assist the prediction of the current step. This process is repeated until the entire sample sequence is generated. The purpose is to enable the trained network to understand and utilize the temporal dependencies of the handover process, allowing it to formulate a coherent, globally planned multi-step handover strategy.

[0102] Thus, the network can independently predict during single-step generation, enabling adaptive adjustments to state deviations and environmental changes, and can also utilize historical information during continuous generation to achieve global planning of the handover unit sequence.

[0103] It should be noted that the task description and the corresponding handover pose and grasping posture are all historical data. Like the historical object point cloud data, this data comes from multiple two-arm handover examples. These examples are demonstrated or annotated by experts. Each example contains data from multiple time steps arranged in chronological order. In each time step, there is the object point cloud at the current moment, the corresponding task description, and the optimal handover unit executed by the expert at that moment (i.e., the handover pose, grasping posture, and its corresponding true termination label).

[0104] Preferably, step 102 may include the following sub-steps: S21. Obtain object point clouds, task descriptions, and corresponding handover poses and grasping postures from multiple consecutive time steps in the training set to construct a sample sequence.

[0105] A sample sequence is a series of one or more consecutive time-step data points extracted from a complete double-arm handover example. These consecutive data points constitute a sample sequence, which serves as the input-output pair processed during one forward propagation and backward update of the network.

[0106] In this embodiment of the invention, a complete two-arm handover example is randomly sampled from a pre-built training set. For this example, a starting time step is randomly selected. and an end time step It must satisfy 1 ≤ < ≤ K, where K is the total number of steps in this example. All data within the continuous interval from the start time step to the end time step will be extracted to form a sample sequence. This sequence includes the object point cloud, task description, actual handover pose, actual grasping pose, and actual termination label for each time step within the interval. The actual termination label is only included in the sample sequence. Set the time interval to 1, and set all other time intervals to 0.

[0107] S22. For the first time step, input the object point cloud and task description of the current time step into the initial handover generation network to obtain the prediction handover unit of the first time step.

[0108] The predicted handover unit is the handover unit output by the initial handover generation network after the object point cloud and task description at the current time step are input into the network. This handover unit has not yet been corrected for the true values.

[0109] In this embodiment of the invention, for the first time step (i.e. the starting time step) in the sample sequence, the corresponding object point cloud and task description are used as input and fed into the initial handover generation network. At this time, since this time step is the starting position of the sequence, there is no historical handover unit. Therefore, the historical handover feature input part is set as a zero vector for filling. The network performs forward propagation based on the current perception information. After feature encoding and autoregressive decoding, the predicted handover unit of the first time step is output. The predicted handover unit specifically includes: predicted handover pose, predicted grasping pose, and predicted termination probability.

[0110] S23. For each subsequent time step, the object point cloud and task description of the current time step, as well as the predicted handover units generated in the previous time steps, are used as historical handover units and input into the initial handover generation network to obtain the predicted handover units of the current time step.

[0111] In this embodiment of the invention, for the sample sequence from arrive For each subsequent time step i, the following autoregressive generation process is executed: First, obtain the object point cloud and task description for the current time step i; second, combine all previous time steps (i.e., from...) The predicted handover units generated and saved in step i-1 are embedded and mapped to obtain historical handover embedding features. After adding position encoding, historical handover features are formed. Then, the perceptual information of the current time step and the historical handover features are input into the initial handover generation network. Finally, after feature encoding and autoregressive decoding, the network outputs the predicted handover unit of the current time step i, which specifically includes the predicted handover pose, the predicted grasping pose, and the predicted termination probability. It is worth mentioning that this predicted handover unit is used for loss calculation in the current step and will be saved and used for generation in subsequent time steps.

[0112] S24. Based on the difference between the predicted handover unit and the corresponding handover pose and grasping pose at each time step, construct a regression loss.

[0113] Preferably, S24 may include the following sub-steps: S131. For each time step, calculate the first difference between the predicted handover pose in the predicted handover unit and the corresponding real handover pose, and calculate the second difference between the predicted grasping pose in the predicted handover unit and the corresponding real grasping pose.

[0114] The true handover pose is a standard handover pose in the training set, labeled by experts for that time step. It represents the ideal position and posture that the object should be in at the moment the arms are handed over in that step, and is usually represented by six-degree-of-freedom parameters (x, y, z, rx, ry, rz).

[0115] The true grasping posture is the standard grasping posture corresponding to the time step, labeled by experts in the training set. It represents the ideal position and posture that the end effector of the robotic arm should be in at the moment of handover between the two arms in that step. This posture is not pre-bound to a specific robotic arm, but is mapped according to the role allocation of the two arms during actual execution.

[0116] The first difference is the error metric between the predicted handover pose and the actual handover pose, which is used to quantify the degree of deviation of the network in the handover pose prediction.

[0117] The second difference is the error metric between the predicted grasping pose and the actual grasping pose, which quantifies the degree of deviation of the network from the grasping pose prediction.

[0118] In this embodiment of the invention, for each time step i in the sample sequence, let the handover pose predicted by the network be... The grasping posture is The actual handover poses corresponding to the training set are The actual grasping posture is Then the first difference is calculated as The second difference is calculated as In the formula, This represents the square of the second norm of a vector, which is the sum of the squares of the errors over each dimension.

[0119] S132. Based on the first and second differences, construct the regression loss for each time step.

[0120] The regression loss is a joint loss value constructed for a single time step, which combines the handover pose prediction error and the grasping pose prediction error of that step.

[0121] In this embodiment of the invention, for each time step in the sample sequence, the sum of the first difference and the second difference is calculated to obtain the regression loss, and the specific formula is as follows:

[0123] In the formula, The pose prediction loss is used for the i-th time step.

[0124] S25. Based on the difference between the predicted termination probability and the actual termination label in the handover unit at each time step, construct the termination loss.

[0125] Termination loss is a loss value used for a single time step to measure the difference between the network's predicted termination probability and the actual judgment of whether the process should terminate at that step. It is designed to train the network to accurately determine whether the handover process should end.

[0126] In this embodiment of the invention, for each time step i in the sample sequence, let the termination probability of the network prediction output be... (Values ​​range from 0 to 1), the actual termination label for this step is (Values ​​are 0 or 1, and are only applied in the last step of the sequence) (Where the value is 1, the rest are 0). The difference between the two is calculated using the binary cross-entropy function to obtain the termination loss at this time step. The specific formula is as follows:

[0128] In the formula, This is the termination loss at the i-th time step. The smaller this loss value, the more accurate the network's judgment is in determining whether the current time step is the termination step.

[0129] S26. Construct sequence length weights based on the length of the sample sequence.

[0130] Sequence length weights are scalar weight values ​​calculated based on the actual length of the entire sample sequence. They are used to differentiate the weights of sequences of different lengths during training, guiding the network to prioritize the generation of handover unit sequences with fewer steps.

[0131] In this embodiment of the invention, let the actual length of the current sample sequence be K (i.e., from...). arrive (Total number of time steps included), the preset maximum length of the handover unit sequence is Sequence length weights Calculate using the following formula: ; As can be seen from the formula, the shorter the sequence length K, The larger the value, the longer the sequence length K. The smaller the value, the more likely the maximum length of the handover unit sequence can be determined by statistical analysis of the sequence length of all double-arm handover examples in the training set. For example, the step distribution of all complete handover examples in the training set can be statistically analyzed, and the upper limit of the range of steps with high frequency or the maximum value of the steps of all examples can be set as the maximum length of the handover unit sequence. Alternatively, the maximum length of the handover unit sequence can be directly set based on empirical values. The specific value can be set as needed. Preferably, the maximum length of the handover unit sequence can be set to 10.

[0132] S27. Construct trajectory length weights based on the length of the robotic arm's motion trajectory corresponding to each time step in the sample sequence.

[0133] The trajectory length weight is a scalar weight value calculated for the entire sample sequence based on the total length of the robotic arm's motion trajectory at all time steps in the sequence. It is used to differentiate and weight sequences with different motion costs during training, so as to guide the network to tend to generate handover unit sequences with lower overall motion costs.

[0134] In this embodiment of the invention, for each time step i in the sample sequence, an existing motion planning algorithm (such as the RRT algorithm) is invoked to calculate the length of the robotic arm's motion trajectory corresponding to the handover at that step, given the handover pose of adjacent steps and the grasping posture. Then, the trajectory lengths at all time steps in the sample sequence are summed to obtain the total trajectory length. Trajectory length weight The specific calculation formula is as follows:

[0136] In the formula, K is the length of the current sample sequence. Let be the length of the robotic arm's motion trajectory at the i-th time step. For a very small positive number, such as This is used to avoid a denominator of 0. As the formula shows, the shorter the total trajectory length, the larger the trajectory length weight; the longer the total trajectory length, the smaller the trajectory length weight. By adjusting this weight, the network can be encouraged to learn handover strategies with lower overall motion costs during training. S28. Construct the total loss function based on regression loss, termination loss, sequence length weight, and trajectory length weight.

[0137] The total loss function is a joint loss function that integrates the pose prediction accuracy, termination judgment accuracy, sequence length efficiency, and motion trajectory cost at each time step, and is used to comprehensively evaluate and optimize the performance of the handover generation network.

[0138] In this embodiment of the invention, the specific formula for calculating the total loss function is as follows:

[0140] In the formula, L is the total loss function value, and K is the length of the current sample sequence. Weights based on sequence length. As the trajectory length weight, and These are all preset loss weighting coefficients used to adjust the proportion of regression loss and termination loss. Let i be the regression loss at the i-th time step. Let be the termination loss at the i-th time step, where the preset loss weight coefficient can be set as needed based on empirical values, preferably 1.0 or 0.5.

[0141] S29. Optimize the initial handover generation network based on the total loss function to obtain the target handover generation network.

[0142] In this embodiment of the invention, the total loss function is used as the optimization objective. The parameters of the initial handover generation network are iteratively updated using the mini-batch stochastic gradient descent method. Specifically, a batch of sample sequences is randomly sampled from the training set each time, the total loss of each sample sequence is calculated and the average value is obtained. Then, the gradient of the loss function with respect to the network parameters is calculated through the backpropagation algorithm, and the network parameters are adjusted using the Adam optimizer. The above process is repeated until the total loss function converges or the preset maximum number of training rounds is reached. At this point, the network parameters are saved, and the trained target handover generation network is obtained. This network can be used in actual dual-arm handover tasks to realize online, real-time handover unit generation and grasping posture optimization.

[0143] Step 103: At the current time step, collect the object point cloud and task description, input them into the target handover generation network, and obtain the handover unit at the current time step.

[0144] In this embodiment of the invention, when performing a dual-arm handover task, for the current time step, a scene depth image is first acquired using an RGB-D depth camera positioned above the robot's workspace. This depth image is then back-projected into three-dimensional space using camera intrinsic parameters to obtain the original point cloud of the scene. Background noise is then removed using a pass-through filter, and the number of points in the point cloud is normalized to 1024 points using a farthest-point sampling algorithm, with each point represented by three-dimensional coordinates. Simultaneously, a natural language description of the current task is acquired, which remains unchanged throughout the handover process. The object point cloud and the task description are then fed into a trained target handover generation network. The network performs forward propagation calculations and outputs the handover unit for the current time step (i.e., handover pose, grasping posture, and termination probability).

[0145] Step 104: Based on the handover pose and grasping posture in the handover unit of the current time step, plan and execute the motion trajectory of both arms until the termination probability in the handover unit of the current time step is greater than the preset termination probability threshold, then stop generating and complete the optimization of the object grasping posture.

[0146] Understandably, at the initial time step, based on the grasping posture predicted in the current step, the RRT algorithm is used to plan the trajectories of the two robotic arms' end effectors from their current positions to the grasping posture, and the corresponding trajectory lengths are calculated. The trajectory lengths of the two arms are compared, and the robotic arm with the shorter trajectory length is selected as the grasping arm, and it is controlled to move along the planned trajectory to complete the initial grasping of the object.

[0147] For each subsequent time step, based on the object's grasping state after the handover in the previous time step, the robotic arm currently grasping the object is designated as the grasping arm, and the other robotic arm as the receiving arm. While maintaining stable grasping of the object and unchanged grasping posture, the grasping arm uses the RRT algorithm to plan its trajectory and move the object to the predicted handover pose for the current step. Simultaneously, the receiving arm plans its trajectory based on the predicted grasping posture for the current step and moves to the handover pose to complete the grasping of the object. After the grasping handover is complete, the original grasping arm releases its gripper and returns to a preset safe position.

[0148] After each handover action is completed, it is determined whether the predicted termination probability of the current step reaches a preset threshold (e.g., 0.8). If the threshold is reached, the generation and execution of subsequent handover units are stopped. At this point, the robot has grasped the object in a posture that meets the task constraints, and the entire handover process ends. If the threshold is not reached, the scene depth image is re-acquired using an RGB-D camera, and after preprocessing, an updated object point cloud is obtained. This point cloud and the task description are then input again into the target handover generation network. The network independently generates the next handover unit based on the current perception state (without relying on historical handover units), and then continues to execute the dual-arm handover operation to further optimize the grasping posture or correct state deviations generated during the handover process. The above generation and execution process is repeated until the termination probability meets the threshold condition.

[0149] It should be noted that during the training phase, an autoregressive strategy is adopted, in which the handover units of historical predictions are used as subsequent inputs to train the network to learn global planning capabilities. During the execution process, in order to respond to environmental changes in real time, the network input at each time step is based only on the current perception information and does not rely on the prediction results of the previous time step for autoregression, thereby achieving closed-loop control.

[0150] Preferably, step 104 may include the following sub-steps: S31. Based on the grasping posture in the handover unit of the first time step, plan the trajectory lengths of the two robotic arms respectively, and obtain the trajectory length comparison results.

[0151] The trajectory length comparison result is a quantitative result obtained by calculating and comparing the trajectory lengths of the two robotic arms as they move to the target grasping posture. It is used to determine which robotic arm is closer to the target grasping posture and has a lower motion cost, thus serving as the basis for selecting the initial grasping arm.

[0152] In this embodiment of the invention, for the first time step (i.e., the initial grasping step), the predicted grasping posture in the generated first time step handover unit is first obtained. The grasping posture indicates the target position and posture that the end effector of the robotic arm should be in at the current step. Then, the pose of the left end effector at the current moment is obtained. The pose of the right robotic arm's end effector at the current moment. The fast expanding random tree (RRT) motion planning algorithm is used to plan the pose from the current end effector in the configuration space of the left and right arms respectively. Calculate the collision-free motion trajectory of the target to the grasping posture, and calculate the lengths of the two corresponding trajectories. and The trajectory length is represented by the sum of the Euclidean distances between the path points. By comparing the lengths of two motion trajectories, the trajectory length comparison result is obtained.

[0153] S32. Based on the trajectory length comparison results, determine the gripper arm and plan the motion trajectory of the gripper arm to obtain the initial gripping trajectory.

[0154] The gripper arm is the robotic arm selected at the initial time step to perform the first gripping operation. It is selected based on the shorter trajectory length of the two robotic arms moving to the target gripping posture, in order to minimize the motion cost and time cost of the initial gripping.

[0155] The initial grasp trajectory is a collision-free motion path planned for the selected grasping arm, from its current end-effector pose to the target grasping pose. It includes a series of continuous intermediate pose points to guide the grasping arm to complete the first grasp of the object.

[0156] In an embodiment of the present invention, if < If so, then the left robotic arm is selected as the gripping arm. > If so, the right robotic arm is selected as the gripping arm. = You can choose any one of them (for example, the left robotic arm is selected by default) as the gripper. After the gripper arm is determined, the RRT motion trajectory planned for the gripper arm in step S31 is used as the initial gripping trajectory. This trajectory includes a series of continuous intermediate pose points from the current end pose of the gripper arm to the target gripping pose, and its safety has been ensured by collision detection.

[0157] S33. Based on the initial grasping trajectory, control the grasping arm to perform grasping and obtain the grasped object.

[0158] An object that has been grasped is an object that has been stably held by the end effector of the robotic arm after the grasping arm has performed the grasping action. At this point, the object has been removed from its initial position, is held by the grasping arm, and can be used for subsequent handover operations.

[0159] In this embodiment of the invention, the initial grasping trajectory is sent to the underlying controller of the grasping arm, which controls the grasping arm to move along the trajectory to the target grasping posture. During the movement, each joint of the grasping arm is servo-controlled according to the trajectory interpolation points to ensure that the end effector smoothly and accurately reaches the predetermined position. When the end effector of the grasping arm reaches the target grasping posture, it controls the gripper to close and performs a grasping operation on the object. The force or tactile sensors on the gripper provide feedback signals to determine whether the grasping is successful. If the grasping is successful, the object is now stably held by the grasping arm and is recorded as a grasped object. If the grasping fails, the trajectory is replanned or error handling is performed according to a preset retry strategy. After the grasping is completed, the grasping arm maintains the current grasping posture and prepares to proceed to the subsequent handover steps.

[0160] S34. For each subsequent time step, determine the gripping arm for the current time step based on the robotic arm that has already gripped the object at the end of the previous time step, and determine the receiving arm for the current time step based on another robotic arm. Based on the handover pose in the handover unit of the current time step, plan the handover trajectory of the gripping arm and the gripping trajectory of the receiving arm respectively.

[0161] The handover trajectory is a collision-free motion path planned for the gripper arm from its current holding position to the handover position predicted at the current time step. It requires the gripper arm to maintain stable gripping of the object and keep its gripping posture unchanged during the movement.

[0162] The grasping trajectory is a collision-free motion path planned for the receiving arm from its current idle pose to the grasping pose predicted at the current time step, so that the receiving arm can accurately grasp objects at the handover pose.

[0163] In this embodiment of the invention, for each subsequent time step i (i≥2) after the first time step, the roles of the two arms are first determined according to the state after the execution of the previous time step i-1. The robotic arm that has grasped the object at the end of the previous time step is determined as the grasping arm of the current time step, and the other robotic arm is determined as the receiving arm of the current time step. Then, the handover unit of the current time step i generated in step 103 is obtained, and the handover pose and grasping pose are extracted from it. The handover pose represents the position and pose of the object when the handover occurs in the current step, and the grasping pose represents the position and pose of the end of the receiving arm when the handover occurs in the current step. Next, the RRT motion planning algorithm is used to plan a collision-free motion trajectory for the grasping arm from the current holding pose (i.e., the pose at the end of the previous time step) to the target handover pose, and the handover trajectory of the grasping arm is obtained. During the planning process, it is necessary to ensure that the grasping arm maintains stable gripping of the object during the movement and does not collide with the environment. Simultaneously, the RRT motion planning algorithm is employed to plan a collision-free motion trajectory for the receiving arm, moving it from its current idle pose to the target grasping pose, thus obtaining the receiving arm's grasping trajectory. This trajectory ensures that the receiving arm can accurately reach the predetermined position, ready to grasp the object.

[0164] S35. Based on the handover trajectory of the gripping arm and the gripping trajectory of the receiving arm, control the gripping arm and the receiving arm to perform movements to obtain the handed-over object.

[0165] After the handover, the object is transferred from the original gripping arm to the receiving arm and is stably held by it after the gripping arm and receiving arm work together to perform the handover operation. This marks the completion of the handover task between the two arms at the current time step.

[0166] In this embodiment of the invention, the handover trajectory of the gripper arm and the gripping trajectory of the receiving arm planned in step S34 are sent to the underlying controllers of the gripper arm and the receiving arm, respectively, to control the coordinated movement of the two arms. To ensure the safety and synchronization of the handover process, a master-slave synchronous control strategy can be adopted: the gripper arm moves along the handover trajectory to the target handover pose, while the receiving arm moves along the gripping trajectory to the target gripping pose. During the movement of the two arms, the controller monitors the position, speed, and torque of each joint in real time to ensure smooth movement without collisions. When the gripper arm reaches the target handover pose and remains stable, the receiving arm has also reached the target gripping pose. At this time, the controller controls the gripper arm's jaws to close, taking the object from the gripper arm. The force or tactile sensor on the receiving arm's jaws provides feedback signals to determine whether the handover is successful. If it is determined that the receiving arm has successfully gripped the object, the object is now held by the receiving arm, and this is recorded as the object state after the handover. After confirming the successful handover, the gripper arm maintains the current pose and waits for the next instruction. If the handover fails, the trajectory is replanned according to the preset retry strategy or error handling is performed.

[0167] S36. Based on the handed-over object, control the original gripping arm to release the gripper and return to the preset safe position after the handover is completed, complete the execution of the current time step, and compare the termination probability in the handover unit of the current time step with the preset termination probability threshold. If the termination probability is greater than the preset termination probability threshold, stop the generation and complete the optimization of the object gripping posture.

[0168] A preset safe position is a fixed posture that is set in advance for the robotic arm, away from the work area and without interfering with objects or the environment. It is used to allow the idle robotic arm to retreat after the handover task is completed, leaving space for subsequent operations.

[0169] The termination probability threshold is a preset scalar value between 0 and 1 (e.g., 0.8) used to determine whether the current time step is the last step of the entire handover process. When the termination probability predicted by the network exceeds the threshold, it is considered that the task constraints have been met, and the generation and execution of subsequent handover units are stopped.

[0170] In this embodiment of the invention, after confirming that the receiving arm has successfully grasped the object and obtained the transferred object, the original grasping arm (i.e., the robotic arm holding the object in the previous time step) is controlled to release its gripper and release the object. Subsequently, the grasping arm is moved from its current pose to a preset safe position to avoid interfering with subsequent operations or causing accidental collisions. At this point, the handover operation of both arms in the current time step is completed. Next, the termination probability in the handover unit of the current time step is obtained and compared with a preset termination probability threshold. If the termination probability is greater than the preset termination probability threshold, the current time step is determined to be the end of the entire handover process. The process terminates at step i+1, when the object has been grasped by the receiving arm in a posture that meets the task constraints. At this point, the generation and execution of subsequent handover units stop, and the entire dual-arm handover and grasping posture optimization process ends. Otherwise, it is determined that the handover process is not yet complete and needs to be optimized further. At this point, the process enters the next time step i+1, returns to step 103, re-collects the object point cloud in the current scene, and inputs it into the target handover generation network along with the task description to generate the next handover unit. Steps S31 to S36 are repeated until the termination probability meets the threshold condition, and finally the gradual optimization of the object grasping posture is achieved.

[0171] To clearly demonstrate the above implementation details, a specific application scenario is provided below: An automotive parts distribution center needs robots to pick up different models of automotive universal joints (irregularly shaped metal parts) from turnover boxes and place them precisely onto fixtures on the assembly line so that workers or automated equipment can then tighten them with bolts. The universal joints are randomly placed in the turnover boxes, possibly in various orientations such as vertical, horizontal, or tilted, while the assembly fixtures require them to be placed in a specific orientation with their flange faces upwards and locating pin holes aligned.

[0172] First, the robot scans the turnover box using a top-mounted RGB-D camera to acquire point cloud data of the gimbal. Simultaneously, it receives the task description "grab the gimbal for assembly." Then, the target handover generation network generates the first handover unit based on the current point cloud and the task description. It compares the trajectory lengths of the left and right robotic arms to the predicted grasping posture and selects the right arm, which has the shorter distance, as the grasping arm. The right arm grasps the gimbal from the turnover box in any feasible posture. After the right arm grasps the gimbal, the network generates the second handover unit based on the real-time updated point cloud. At this point, the right arm, acting as the grasping arm, moves the gimbal to the predicted handover pose (e.g., placing the gimbal in the center of space). The left arm, acting as the receiving arm, moves to the predicted grasping posture to prepare for receiving. The two arms work together to complete the first handover. The universal joint is transferred to the left arm. After the left arm holds the object, the network generates the third handover unit. The termination probability increases significantly. The left arm moves the universal joint to a more precise pose that is closer to the assembly requirements (the flange surface is completely horizontal and the locating pin holes are aligned). The right arm receives the object in a precise grasping posture. At this point, the termination probability exceeds the threshold of 0.8, and the handover process ends. Finally, the right arm stably grasps the universal joint in a posture that fully meets the assembly constraints and can directly place it into the positioning slot of the assembly fixture without manual intervention or secondary adjustment. Example

[0173] Please see Figure 4 The present invention provides a dual-arm handover and grasping posture optimization system based on an autoregressive strategy, comprising: The initial network construction module 201 is used to construct the initial handover generation network. The initial handover generation network is used to generate handover units based on the input object point cloud and task description. The handover units include handover pose, grasping pose and termination probability. The target network training module 202 is used to acquire historical object point clouds, task descriptions and corresponding handover poses and grasping postures to construct a training set, and to train the initial handover generation network based on the training set and the autoregressive strategy to obtain the target handover generation network. The handover unit generation module 203 is used to collect object point cloud and task description at the current time step, input them into the target handover generation network, and obtain the handover unit at the current time step. The trajectory planning and execution module 204 is used to plan and execute the motion trajectory of the two arms based on the handover pose and grasping posture in the handover unit of the current time step, until the termination probability in the handover unit of the current time step is greater than the preset termination probability threshold, at which point the generation stops and the optimization of the object grasping posture is completed.

[0174] Since the above is a system corresponding to a method for optimizing the hand-arm hand-over and grasping posture based on an autoregressive strategy, its implementation principle is the same as that of a method for optimizing the hand-arm hand-over and grasping posture based on an autoregressive strategy. For the sake of convenience and brevity, those skilled in the art can clearly understand that the specific working process of the system and modules described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here. Example

[0175] An electronic device according to an embodiment of the present invention includes: a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the dual-arm handover and grasping posture optimization method based on autoregressive strategy as described in any of the above embodiments.

[0176] The memory can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory has storage space for program code used to perform any of the method steps described above. Example

[0177] This invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed, it implements the method for optimizing the double-arm handover and grasping posture based on an autoregressive strategy according to any of the above embodiments.

[0178] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the handover and grasping posture of two arms based on an autoregressive strategy, characterized in that, include: An initial handover generation network is constructed, which is used to generate handover units based on the input object point cloud and task description. The handover unit includes handover pose, grasping pose and termination probability. Acquire historical object point clouds, task descriptions, and corresponding handover poses and grasping postures; construct a training set; and train the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network. At the current time step, collect the object point cloud and task description, input them into the target handover generation network, and obtain the handover unit at the current time step; Based on the handover pose and grasping posture in the handover unit of the current time step, the motion trajectory of the two arms is planned and executed until the termination probability in the handover unit of the current time step is greater than the preset termination probability threshold, at which point generation stops, and the optimization of the object grasping posture is completed.

2. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 1, characterized in that, The initial handover generation network includes a feature encoding network and an autoregressive decoding network. Constructing the initial handover generation network includes: The object point cloud and task description are encoded and mapped using the feature encoding network to obtain global conditional features; Obtain historical handover units and perform embedding mapping on the historical handover units to obtain historical handover embedding features; The autoregressive decoding network is used to fuse and decode the historical handover features and the global condition features to obtain the handover unit.

3. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 2, characterized in that, The feature encoding network includes a point cloud encoding unit, a task encoding unit, and a feature fusion mapping unit. The point cloud encoding unit includes a local feature extraction subunit and a global feature generation subunit. The task encoding unit includes a semantic transformation subunit and a semantic understanding subunit. The process of encoding and mapping the object point cloud and task description through the feature encoding network to obtain global conditional features includes: The local feature extraction subunit extracts local geometric features from the object point cloud to obtain local features. The global feature generation subunit performs global feature aggregation on the local features to obtain geometric features; The semantic transformation subunit performs semantic transformation on the task description to obtain a word segmentation sequence; The semantic understanding subunit performs semantic understanding on the segmented word sequence to obtain task features; The geometric features and the task features are fused by the feature fusion mapping unit to obtain global conditional features.

4. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 2, characterized in that, The autoregressive decoding network includes a position encoding subunit, a temporal modeling subunit, a feature fusion subunit, and an output mapping subunit. The step of fusing and decoding the historical handover features and the global conditional features through the autoregressive decoding network to obtain the handover unit includes: The historical handover features are obtained by adding location encoding information to the historical handover embedded features through the location encoding subunit; The target handover features are obtained by performing time-series relationship modeling on the historical handover features through the time-series modeling subunit. The feature fusion subunit performs feature fusion on the target intersection feature and the global condition feature to obtain the fused feature; The output mapping subunit performs output mapping on the fused features to obtain the handover unit.

5. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 1, characterized in that, The training set also includes real termination labels. The process of training the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network includes: From the training set, obtain object point clouds, task descriptions, and corresponding handover poses and grasping postures at multiple consecutive time steps to construct a sample sequence; For the first time step, the object point cloud and task description of the current time step are input into the initial handover generation network to obtain the prediction handover unit of the first time step. For each subsequent time step, the object point cloud and task description of the current time step, as well as the predicted handover units generated in the previous time steps, are used as historical handover units and input into the initial handover generation network to obtain the predicted handover units of the current time step. A regression loss is constructed based on the difference between the predicted handover unit and the corresponding handover pose and grasping pose at each time step; A termination loss is constructed based on the difference between the predicted termination probability in the handover unit at each time step and the actual termination label. Construct sequence length weights based on the length of the sample sequences; Based on the length of the robotic arm's motion trajectory corresponding to each time step in the sample sequence, a trajectory length weight is constructed. Based on the regression loss, the termination loss, the sequence length weight, and the trajectory length weight, a total loss function is constructed. The initial handover generation network is optimized based on the total loss function to obtain the target handover generation network.

6. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 5, characterized in that, The regression loss is constructed based on the difference between the predicted handover unit at each time step and the corresponding handover pose and grasping pose, including: For each time step, calculate the first difference between the predicted handover pose in the predicted handover unit and the corresponding real handover pose, and calculate the second difference between the predicted grasping pose in the predicted handover unit and the corresponding real grasping pose. Based on the first difference and the second difference, the regression loss for each time step is constructed.

7. The method for optimizing the handover and grasping posture based on an autoregressive strategy according to claim 1, characterized in that, The process of planning and executing the dual-arm motion trajectory based on the handover pose and grasping posture in the handover unit at the current time step continues until the termination probability in the handover unit at the current time step exceeds a preset termination probability threshold, at which point generation stops, thus completing the optimization of the object grasping posture. This includes: Based on the grasping posture in the handover unit of the first time step, the trajectory lengths of the two robotic arms are planned respectively, and the trajectory length comparison results are obtained. Based on the trajectory length comparison results, the gripping arm is determined, and the motion trajectory of the gripping arm is planned to obtain the initial gripping trajectory; Based on the initial grasping trajectory, the grasping arm is controlled to perform grasping, thereby obtaining the grasped object; For each subsequent time step, the gripping arm of the current time step is determined based on the robotic arm that has grasped the object at the end of the previous time step, and the receiving arm of the current time step is determined based on another robotic arm. Based on the handover pose in the handover unit of the current time step, the handover trajectory of the gripping arm and the gripping trajectory of the receiving arm are planned respectively. Based on the handover trajectory of the grasping arm and the grasping trajectory of the receiving arm, the grasping arm and the receiving arm are controlled to perform movements to obtain the handed-over object; Based on the handed-over object, the original gripping arm releases its gripper and returns to a preset safe position after the handover is completed, thus completing the execution of the current time step. The termination probability in the handover unit of the current time step is compared with a preset termination probability threshold. If the termination probability is greater than the preset termination probability threshold, generation stops, thus completing the optimization of the object gripping posture.

8. A dual-arm handover and grasping posture optimization system based on an autoregressive strategy, characterized in that, include: An initial network construction module is used to construct an initial handover generation network. The initial handover generation network is used to generate handover units based on the input object point cloud and task description. The handover unit includes handover pose, grasping pose, and termination probability. The target network training module is used to acquire historical object point clouds, task descriptions, and corresponding handover poses and grasping postures to construct a training set, and to train the initial handover generation network based on the training set and an autoregressive strategy to obtain the target handover generation network. The handover unit generation module is used to collect object point cloud and task description at the current time step, input them into the target handover generation network, and obtain the handover unit at the current time step. The trajectory planning and execution module is used to plan and execute the movement trajectory of both arms based on the handover pose and grasping posture in the handover unit of the current time step, until the termination probability in the handover unit of the current time step is greater than a preset termination probability threshold, at which point generation stops, thus completing the optimization of the object grasping posture.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7, which is a method for optimizing the hand-arm handover and grasping posture based on an autoregressive strategy.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7, for optimizing the bi-arm handover and grasping posture based on an autoregressive strategy.

Images