A branch linear flexible body dynamics modeling method for robot operation

Abstract

The application belongs to the technical field of robot operation. The application provides a branch linear flexible body dynamics modeling method for robot operation. The embodiment of the disclosure solves the problem of difficult construction of the relationship between state changes and robot actions caused by high dimension, nonlinearity, dynamics and anisotropy of BDLO through BDLO state segmentation, local feature aggregation, branch interaction construction in different degrees and state joint solving, realizes accurate and reliable BDLO dynamics modeling, and further improves the accuracy and reliability of robot autonomous operation BDLO, and promotes the wide application and subsequent development of robot autonomous operation technology in the BDLO scene.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of robot operation, and particularly relate to a branched linear flexible body dynamics modeling method for robot operation. BACKGROUND

[0002] A branched linear flexible body (BDLO) refers to an object that appears linear and has multiple branches, and whose state can change continuously with external operations, such as branched cables, multi-channel pipelines, branched fibers, and the like.

[0003] A system for robot operation of a BDLO generally needs to include three parts, namely BDLO state perception, BDLO dynamics modeling, and robot operation control. Among them, the dynamics modeling of the BDLO is mainly to construct the relationship between the state change of the BDLO and the robot action. Specifically, it calculates the action required to be performed by the robot to achieve the expected state according to the gap between the BDLO state obtained by BDLO state perception and the expected BDLO state of the user, and provides guidance for subsequent robot operation control, which is one of the core contents of autonomous BDLO operation of the robot.

[0004] For example, in the task of automatic wiring and maintenance of an electrical control cabinet robot, for the branched cables widely existing in the electrical control cabinet, the robot needs to calculate the difference between the initial state of the branched cables in the cabinet and the expected state of the branched cables after assembly according to the dynamics model of the branched cables, and calculate the grabbing and moving actions required to be performed by the robot in real time to cope with the complex morphological changes of the branched cables in the cabinet, and finally realize the stable and accurate arrangement of the branched cables into the target morphology.

[0005] When the BDLO is subjected to external forces, its continuous deformation behavior has the typical characteristic of "infinite degrees of freedom", so that the relationship between the state change of the BDLO and the robot action in the operation process presents complex high-dimensional and nonlinear characteristics. In addition, the BDLO has a complex topology and variable shape, and is subjected to different effects of connection force and friction, so that the propagation mechanism of local deformation in the BDLO also presents dynamic and anisotropic characteristics. These complex characteristics of the BDLO make its state change difficult to predict, thereby significantly increasing the difficulty of control of the BDLO robot operation process.

[0006] The patent "Robot dynamic operation cloth method and system based on reinforcement and imitation learning" (application number 202211191648.2) and the patent "Force simulation method and device for flexible object, electronic equipment and storage medium" (application number 202310334834.5) both use the spring particle method to model the dynamics of cloth. However, this modeling method has the problem of over-simplifying BDLO, which makes the dynamics prediction distorted and difficult to apply in scenarios where BDLO has dynamic and anisotropic behavior.

[0007] The patent "Separation type spacecraft flexible cable modeling and micro-vibration compensation method" (application number 202411397408.7) uses time-domain segmented differential quadrature method to model the dynamics of the cable. This physical modeling method is a reliable solution, but it relies on the mechanical properties of the cable, which is costly and less applicable in scenarios with complex topological structure like BDLO.

[0008] The patent "Multi-robot flexible object servo method based on deep learning" (application number 202411133433.4) uses a dynamics model based on convolution and fully connected operations for cloth. This method, which introduces a neural network, can be well applied to complex nonlinear dynamics model construction. However, its network structure is too simple and cannot be applied to scenarios where BDLO has dynamic and anisotropic behavior.

[0009] The patent "Method and device for manipulating deformable objects based on visual and language models" (application number 202411640054.4) and the patent "Flexible object manipulation method and device based on key point detection" (application number 202410184129.6) use the advantages of attention mechanisms in deep learning Transformer architecture to represent the global interaction ability of key features of deformable objects, achieving good results. However, when applied to the dynamics modeling scenario of BDLO, there is a lack of understanding of the dynamic and anisotropic behavior of BDLO.

[0010] The document "Chen Y, Wu X, Zong Y, et al. DEFT: Differentiable Branched Discrete Elastic Rods for Modeling Furcated DLOs in Real-Time[J]. arXiv preprint arXiv:2502.15037, 2025." completes the dynamic modeling of BDLO by combining the Cosserat rod model and the GCN residual network. This method can adaptively optimize sensitive physical parameters in offline training, thereby improving the accuracy of the dynamic model. However, due to its high computational complexity, the robustness and generalization ability of this method in BDLOs of different structures are still limited.

[0011] In general, the current flexible body dynamic modeling method mainly targets single linear flexible bodies or cloth, and there is less discussion on the dynamic and anisotropic characteristics of BDLO, which limits the robot operation for this more widely existing flexible body.

[0012] Therefore, it is necessary to improve one or more problems existing in the above related technical solutions.

[0013] It should be noted that this part aims to provide background or context for the technical solutions of the disclosure stated in the claims. The description herein is not admitted to be prior art because it is included in this part. SUMMARY

[0014] The purpose of the embodiments of the present disclosure is to provide a branched linear flexible body dynamic modeling method for robot operation, thereby at least overcoming one or more problems caused by the limitations and defects of the related art.

[0015] According to the embodiments of the present disclosure, a branched linear flexible body dynamic modeling method for robot operation is provided, comprising:

[0016] Step S1: extracting a plurality of key points from a BDLO image collected by a camera, segmenting and optimizing all key points to obtain a plurality of optimized key point sets; N

[0017] Step S2: for each optimized key point set, constructing a key point tensor and an adjacency tensor, using a GCN algorithm to aggregate features of each key point tensor and adjacency tensor respectively to obtain an aggregated feature tensor, and splicing all aggregated feature tensors to obtain a spliced aggregated feature tensor;

[0018] ​Step S3: After dimension expansion of the spliced and aggregated feature tensor by the first multi-layer perception, the different degrees of interaction relationships between the key points are represented by a soft mask, and the branch interaction relationship features of the spliced and aggregated feature tensor are constructed by combining the encoding of the first Transformer encoder, to obtain a global information feature tensor of the current state;

[0019] Step S4: According to the global information feature tensor of the current state and the global information feature tensor of the expected state, the robot's grasping position and placing position are obtained by encoding and decoding of the second Transformer encoder and the second multi-layer perception, to realize the dynamics modeling from the BDLO state change to the robot action.

[0020] Further, step S1 specifically comprises:

[0021] Step S1-1: acquiring a BDLO image collected by a camera, extracting a plurality of key points and their topological relationships of the BDLO image, and identifying branch points and overlapping points in the BDLO image; N

[0022] Step S1-2: constructing an undirected graph with each key point as a vertex set and the topological relationship between the key points as an edge set;

[0023] Step S1-3: segmenting the key points according to the branch points, overlapping points and undirected graph, to obtain a key point set;

[0024] Step S1-4: optimizing the key point set to exclude duplicate points, to obtain a plurality of optimized key point sets.

[0025] Further, step S1-3 specifically comprises:

[0026] Step S1-3-1: setting the initial value of the serial number of the key point set as 1; k

[0027] Step S1-3-2: selecting the leftmost leaf node in the undirected graph as the starting node of the depth-first traversal, and recording it as ;

[0028] Step S1-3-3: starting from the starting node, traversing each key point along the path of the undirected graph in turn, and continuously assigning an increasing serial number to each key point according to the order in which it is accessed and recording it as ;

[0029] Step S1-3-4: when a segmentation point is reached, the segmented key points are constructed into a key point set , and then the value of k is set to 1; wherein the segmentation points include branch points and overlapping points; ​​

[0030] Step S1-3-5: Store the current segmentation point;

[0031] Step S1-3-6: Based on the stored segmentation points, traverse the leftmost unselected branch path in the horizontal direction from left to right in the current image coordinate system; if all path branches are selected, jump to step S1-3-10.

[0032] Step S1-3-7: Enter the branch path selected in step S1-3-6, continue to execute the depth-first traversal strategy, continuously assign subsequent sequence numbers to the key points on this branch and record them;

[0033] Step S1-3-8: If the traversal of nodes reaches the split point, then jump to step S1-3-4;

[0034] Step S1-3-9: If the traversal of nodes reaches a leaf node, construct a set of key points. Then set k Increment the value by 1;

[0035] Step S1-3-10: Read the previously stored split point and jump to step S1-3-6; if the previously stored split point does not exist, the splitting ends, and all key points have been visited and assigned sequence numbers. And all key points are divided into A set of key points .

[0036] Furthermore, steps S1-4 specifically include:

[0037] Step S1-4-1: Set the first set of optimization key points With the first set of key points same;

[0038] Step S1-4-2: Set the optimization key point set and the key point set index. The initial value is 2;

[0039] Step S1-4-3: Read the set of key points The key elements in the key set are removed, and those that are already in the key set are deleted. To the set of key points The key elements that appeared in the text are used to obtain the optimized key point set. ;

[0040] Step S1-4-4: Settings Add 1 to the value; if Then proceed to step S1-4-3; if Then the process ends, and the set of optimization key points is obtained. .

[0041] Further, the step S2 specifically comprises:

[0042] The optimized key point set is constructed as a tensor respectively.

[0043] For each optimized key point tensor , , a corresponding adjacency tensor is constructed.

[0044] Taking each optimized key point tensor and its corresponding adjacency tensor as input, an aggregated feature tensor of each optimized key point tensor is obtained by a multi-layer GCN algorithm.

[0045] Each aggregated feature tensor is spliced in the dimension direction of the number of key points to obtain a spliced aggregated feature tensor ; wherein the aggregated feature tensor contains a first dimension and a second dimension, the first dimension representing the number of key points, and the second dimension representing the dimension of information of each key point.

[0046] Further, the step S3 specifically comprises:

[0047] Step S3-1: The second dimension of the aggregated feature tensor is expanded by a linear layer and a ReLU activation function of the first multi-layer perceptron to extract a deeper hidden feature tensor from it.

[0048] Step S3-2: A position encoding is generated for each key point element in the hidden feature tensor by a sine-cosine position encoding method, and is added to the feature vector of the key point to obtain an updated hidden feature tensor .

[0049] Step S3-3: A soft mask tensor is created, and each element of the soft mask tensor is initially set to negative infinity.

[0050] Step S3-4: The value of each element in the soft mask tensor is updated by using an undirected graph; wherein is the i-th row of the soft mask tensor i , is the j-th column of the soft mask tensor i , j is the i-th row and the j-th column of the adjacency tensor , and is the i-th row and the j-th column of the hidden feature tensor .j column;

[0051] Step S3-5: update the hidden feature tensor and the updated soft mask tensor , and the interaction feature between the concatenated aggregated feature tensor is characterized by using a multi-layer first Transformer encoder, and the global information feature tensor is output.

[0052] Further, step S3-4 specifically includes:

[0053] Step S3-4-1: for all set pairs satisfying and , construct a tuple ; wherein, ;

[0054] Step S3-4-2: construct all tuples into a set ; wherein, is the total number of tuples, represents the th tuple, represents the sequence number of the first set in the th tuple, represents the sequence number of the second set in the th tuple;

[0055] Step S3-4-3: set the initial value of the sequence number of the tuple to 1, and set the weight of the edge in the undirected graph to 1;

[0056] Step S3-4-4: set the minimum value of the key point sequence number in the optimized key point set to ;

[0057] Step S3-4-5: set the minimum value of the key point sequence number in the optimized key point set to ;

[0058] Step S3-4-6: according to the undirected graph and the edge weight, the shortest distance between the key point and each intersection key point is calculated by Dijkstra algorithm, and the minimum is selected as the intra-fragment distance of the key point , and the corresponding intersection key point is recorded Key point Segment matching key points ;

[0059] Step S3-4-7: If the segment matches the key points If it is an overlap point, then set the coefficient. If the segment matches the key points If it is a branch point, then set the coefficient. If the segment matches the key points If it is not an overlapping point or a branch point, set the coefficient. ;in, It is the first positive constant. It is the second positive constant;

[0060] Step S3-4-8: Calculate key points using Dijkstra's algorithm based on the undirected graph and its edge weights. Key points for matching segments shortest distance ;

[0061] Step S3-4-9: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation and Add them together to get the key points. With key points geodesic distance ;

[0062] Step S3-4-10: Based on key points With key points geodesic distance and coefficients Update the first element of the corresponding mask tensor. and the second element of the mask tensor ;

[0063] Step S3-4-11: If ,set up Increment the value by 1 and jump to step S3-4-6; where, Represents the set of optimization key points The number of key elements in the text;

[0064] Step S3-4-12: If ,set up Increment the value by 1 and jump to step S3-4-5; where Represents a set The number of key elements in the text;

[0065] Step S3-4-13: If ,set up The value is added by 1, and jump to step S3-4-4;

[0066] Step S3-4-14: when , the update is ended, and the update of the element in the soft mask tensor is completed.

[0067] Further, the step S4 specifically comprises:

[0068] segmenting the expected BDLO state, locally aggregating features, and constructing a degree branch interaction to obtain a global information feature tensor of the expected state;

[0069] splicing the global information feature tensor of the current state and the global information feature tensor of the expected state to obtain a spliced global feature tensor;

[0070] performing position coding on the spliced global feature tensor to obtain an updated spliced global feature tensor;

[0071] creating a hard mask tensor for isolating attention calculation between the key point of the current moment and the key point of the expected moment;

[0072] inputting the updated spliced global feature tensor and the hard mask tensor to process by using a second multi-layer Transformer encoder to represent the features of the BDLO cross-time domain relationship, and obtain a feature encoding tensor that fuses the current state and the user's expected state;

[0073] decoding the feature encoding tensor by using a second multi-layer perceptron to obtain a score tensor;

[0074] selecting the grabbing position and the placing position of the robot according to the score tensor, and realizing the dynamics modeling from the BDLO state change to the robot action.

[0075] The technical scheme provided by the embodiments of the present disclosure can include the following beneficial effects:

[0076] ​In the embodiments of this disclosure, the above method, on the one hand, treats the BDLO image as a whole and extracts its key points and topological structure using image recognition methods; and divides the BDLO image into several segments with branch points and overlapping points as boundaries. In the local feature aggregation method, for the separated single linear flexible body, an adjacency tensor is established according to its key point connection relationship; and the key points and adjacency tensors within the segments are processed using the GCN method, thereby aggregating and representing the features within each segment. Using this simple aggregation method to simulate the deformation propagation mechanism between key points of a single linear flexible body, the computational complexity and response time of the model are reduced while ensuring the accuracy of dynamic modeling. In the method of constructing branch interaction with varying degrees, corresponding soft mask values ​​are set based on the geodesic distance between key points of different segments, and the mask values ​​for the relationships between the remaining key points are set to negative infinity; the soft mask tensor and key point features are processed using a multi-layer first Transformer encoder to express the relationship features of the interaction between each segment of the BDLO. Using this method of setting the degree of interaction with soft masks can reflect the different characteristics of the interaction between segments connected by branch points or overlapping points, and complete the modeling of the nonlinearity and anisotropy of different deformation propagation modes. In the joint state solution method, the processing of the current BDLO state and the desired BDLO state are combined. Through encoding by a second Transformer encoder and decoding by a second multilayer perceptron, the robot's grasping and placement positions are obtained, achieving dynamic modeling from BDLO state changes to robot actions. On the other hand, when the robot operates on the BDLO under the guidance of the constructed dynamic model, each robot operation changes the BDLO state, affecting overlap points and segmentation segments, thus influencing the transmission of connection forces and friction forces, making BDLO dynamic modeling dynamic. This method addresses this dynamic nature of BDLO by resegmenting segments at each time step and recalculating soft masks. By effectively handling the inherent complexities of high dimensionality, nonlinearity, dynamics, and anisotropy in BDLO dynamic modeling, this method achieves accurate and reliable modeling of its dynamic behavior. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0078] Figure 1 The diagram illustrates the steps of a branched linear flexible body dynamics modeling method for robot manipulation in an exemplary embodiment of this disclosure;

[0079] Figure 2 A schematic diagram showing one application scenario in the exemplary embodiments of the present disclosure;

[0080] Figure 3 A schematic diagram showing one branched linear flexible body in the exemplary embodiments of the present disclosure;

[0081] Figure 4 A flowchart showing application of a branched linear flexible body dynamics modeling method for robot operation in the exemplary embodiments of the present disclosure;

[0082] Figure 5 A schematic diagram showing all key points, segmented key point sets and optimized segmented key point sets in the exemplary embodiments of the present disclosure;

[0083] Figure 6 A specific flowchart showing aggregation and splicing of optimized key point tensors in the exemplary embodiments of the present disclosure;

[0084] Figure 7 A flowchart showing key point global information feature tensor calculation in the exemplary embodiments of the present disclosure;

[0085] Figure 8 A schematic diagram showing soft mask value updating in the exemplary embodiments of the present disclosure;

[0086] Figure 9 A flowchart showing robot grasping and placing point selection in combination with current state and desired state in the exemplary embodiments of the present disclosure;

[0087] Figure 10 A simulation diagram showing current BDLO state and simulation robot grasping BDLO in the exemplary embodiments of the present disclosure;

[0088] Figure 11 A simulation diagram showing desired BDLO state and simulation robot operation to obtain desired BDLO state in the exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0089] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0090] In addition, the accompanying drawings are only schematic and are non-limiting illustrative of the embodiments of this disclosure; they are not necessarily drawn to scale. Same reference numerals in different drawings denote the same or similar functionalities unless otherwise delineated herein. Some of the blocks in the drawings can be functional blocks that represent functions implemented by a processor, software, or combination thereof for performing the specified functions.

[0091] A robot operation-oriented branched linear flexible body dynamics modeling method is provided in the present example embodiment. Referring to FIG. 11, the robot operation-oriented branched linear flexible body dynamics modeling method can include: Figure 1

[0092] Step S1: Extracting a plurality of key points from a BDLO image collected by a camera, segmenting and optimizing all the key points to obtain a plurality of optimized key point sets; N

[0093] Step S2: For each optimized key point set, constructing a key point tensor and an adjacency tensor, using a GCN algorithm to respectively aggregate features of each key point tensor and adjacency tensor to obtain an aggregated feature tensor, and splicing all the aggregated feature tensors to obtain a spliced aggregated feature tensor;

[0094] Step S3: After dimension expansion of the spliced aggregated feature tensor by a first multi-layer perception, using a soft mask to represent different degrees of interaction between the key points, combining the encoding of a first Transformer encoder to construct branch interaction feature of the spliced aggregated feature tensor, and obtaining a global information feature tensor of a current state;

[0095] Step S4: According to the global information feature tensor of the current state and the global information feature tensor of a desired state, through the encoding of a second Transformer encoder and the decoding of a second multi-layer perception, obtaining a grasping position and a placing position of the robot, and realizing dynamics modeling from a BDLO state change to a robot action.

[0096] ​​By the above branch linear flexible body dynamics modeling method facing robot operation, on the one hand, the BDLO image is taken as a whole, and its key points and topological structure are extracted by image recognition method; and the BDLO image is divided into several segments with branch points and overlapping points as boundaries. In the local feature aggregation method, for the separated single linear flexible body, an adjacency tensor is established according to the connection relationship of the key points; and the key points and the adjacency tensor in the segment are processed by the GCN method, so as to aggregate and represent the features in each segment. Using this simple aggregation means to simulate the deformation propagation mechanism between the key points of the single linear flexible body can reduce the computational complexity and response time of the model while ensuring the accuracy of the dynamics modeling. In the branch interaction construction method, the soft mask value is set based on the geodesic distance between the key points of different segments, and the mask value of the relationship between the remaining key points is set to negative infinity; the soft mask tensor and the key point features are processed by the multi-layer first Transformer encoder to express the relationship features of the interaction of each segment of the BDLO. Using this soft mask setting interaction degree method can reflect the different features of the interaction of each segment connected by the branch points or overlapping points, and complete the nonlinear and anisotropic modeling of different deformation propagation modes. In the state joint solving method, the processing of the current BDLO state and the processing of the expected BDLO state are combined, and the robot's grasping position and placing position are obtained by the coding of the second Transformer encoder and the decoding of the second multi-layer perception machine, so as to realize the dynamics modeling from the BDLO state change to the robot action. On the other hand, when the robot operates the BDLO under the guidance of the constructed dynamics model, the state of the BDLO will be changed by each robot operation, which will affect the overlapping points and the segmented segments, and thus affect the transmission of the connection force and the friction, so that the BDLO dynamics modeling has dynamic nature. And the method realizes the dynamics modeling for the dynamic nature of the BDLO through the re-segmentation of each segment at each time and the recalculation of the soft mask. By effectively processing the inherent complex characteristics such as high dimensionality, nonlinearity, dynamics and anisotropy in the BDLO dynamics modeling, the method realizes the accurate and reliable modeling of the dynamics behavior of the BDLO.

[0097] In the following, reference will be made to Figures 1 to 11 The above branch linear flexible body dynamics modeling method facing robot operation in the present example embodiment will be described in more detail.

[0098] In step S1, the key points in the BDLO image collected by the camera are extracted N The key points will be segmented and optimized to obtain several optimized key point sets.

[0099] Specifically, the BDLO state is divided into several segments according to the branch and overlap conditions. Specifically, N key points of the BDLO are extracted from the BDLO image collected by the camera to represent the BDLO state; the key points are divided into an optimized key point set is further obtained.

[0100] Step S1-1: By image recognition method, the key points and topological structure of the BDLO image are extracted, and the branch points and overlap points are identified.

[0101] Step S1-2: The topological structure of image recognition is constructed into an undirected graph form.

[0102] Step S1-3: Based on the key points and branch points, overlap points and undirected graph, the key points are segmented.

[0103] Step S1-4: The key point segmentation is optimized to exclude the repeated points (generally branch points or overlap points).

[0104] In step S2, for each optimized key point set, a key point tensor and an adjacency tensor are constructed, and a GCN algorithm is used to aggregate the features of each key point tensor and adjacency tensor respectively to obtain an aggregated feature tensor, and all the aggregated feature tensors are spliced to obtain a spliced aggregated feature tensor.

[0105] Specifically, the local feature aggregation of the BDLO segment (i.e. BDLO image): for each segmented BDLO segment, the intra-segment feature aggregation is performed, and the segment aggregation features are spliced. Specifically, the optimized key point set is constructed into a tensor; the GCN algorithm is used to aggregate each optimized key point set tensor to obtain an aggregated feature tensor; and these feature tensors are spliced to obtain a spliced aggregated feature tensor.

[0106] Step S2-1: The edge relationship between the key points in the segment is constructed in the form of adjacency tensor.

[0107] Step S2-2: Input the key points and adjacency tensors in each segment, process in the GCN mode, and aggregate the features in each segment.

[0108] Step S2-3: The aggregated features of each segment are spliced to obtain a spliced aggregated feature tensor.

[0109] In step S3, based on the spliced aggregated feature tensor, the different degrees of interaction relationship between the key points are represented by using a soft mask, and the branch interaction relationship features of the spliced aggregated feature tensor are constructed by combining the encoding of the first Transformer encoder, to obtain a global information feature tensor of the current state.

[0110] Specifically, the branch interaction construction of different degrees: using soft mask to represent the different degree interaction relationship between key points, combining the encoding of the first Transformer encoder, the branch interaction relationship features of the aggregated features of BDLO segment splicing are constructed, and the key point global information feature tensor is obtained.

[0111] Step S3-1: Dimension expansion of the aggregated features of BDLO segment splicing.

[0112] Step S3-2: Position coding is performed on the aggregated features after dimension expansion.

[0113] Step S3-3: Create an initial soft mask tensor with all mask values being negative infinity.

[0114] Step S3-4: Based on the geodesic distance between the key points of different segments, update the mask values in the soft mask tensor, and keep the mask values between the remaining key points as the initial mask values.

[0115] Step S3-5: Input the position coded expanded dimension aggregated features and the soft mask tensor, process it in the way of multi-layer Transformer encoder, characterize the BDLO segment interaction relationship features, and obtain the key point global information feature tensor.

[0116] In step S4, according to the global information feature tensor of the current state and the global information feature tensor of the desired state, the second Transformer encoder coding and the second multi-layer perception decoding are used to obtain the grasping position and the placing position of the robot, and the dynamics modeling from the BDLO state change to the robot action is realized.

[0117] Specifically, the state joint solution: joint processing of the current BDLO state and the desired BDLO state, through the coding of the second Transformer encoder and the decoding of the second multi-layer perception, the grasping position and the placing position of the robot are obtained, and the dynamics modeling from the BDLO state change to the robot action is realized.

[0118] Step S4-1: The user given desired BDLO state is processed by segmentation, local feature aggregation, and branch interaction construction of different degrees.

[0119] Step S4-2: The splicing of the current BDLO state processing result and the desired BDLO state processing result is obtained, and the splicing global feature tensor is obtained.

[0120] Step S4-3: Position coding is performed on the splicing global feature tensor, and the updated splicing global feature tensor is obtained.

[0121] Step S4-4: Create a hard mask tensor to isolate the attention calculation between the key points at the current time and the key points at the desired time.

[0122] Step S4-5: Input the updated concatenated global feature tensor and hard mask tensor, process them using a multi-layer second Transformer encoder, represent the features of the BDLO cross-temporal relationship, and obtain the feature encoding tensor that fuses the current state and the user's expected state.

[0123] Step S4-6: Decode the features of the BDLO cross-temporal relationship using a multilayer perceptron to obtain the scoring tensor.

[0124] Step S4-7: Select the robot's grasping and placement positions based on the scoring tensor.

[0125] In one specific embodiment, an application scenario example of using the dynamic model established in this application to guide a robot's autonomous operation of branch cables is shown, such as... Figure 2 As shown.

[0126] A flowchart illustrating an example of an application method using the dynamic model established in this application to guide the autonomous operation of BDLO by a robot, as shown below. Figure 3 As shown.

[0127] The specific implementation of the dynamic modeling method for branched linear flexible bodies for robot manipulation proposed in this application is as follows:

[0128] Step S1: BDLO image acquired from the camera Extract N key points of BDLO Used to represent BDLO states; these key points are divided into A set of key points And further optimization yields the set of key optimization points. .

[0129] Step S1-1: Use the GNN topological representation method to obtain BDLO images from the camera. Extract N key points of BDLO and their topological relationships, and mark the branch points and intersection points.

[0130] Step S1-2: Construct an undirected graph using each key point as the vertex set and the topological relationships between key points as the edge set.

[0131] Step S1-3: Based on the undirected graph in Step S1-2, use a depth-first traversal method to assign unique and continuously increasing global indices to all key points, thus obtaining... At the same time, these key points are divided into multiple sets using the branch points and intersection points marked in step S1-1 as the dividing points. wherein the split points include branch points and overlap points.

[0132] Step S1-3-1: Set the initial value of split set index k to 1.

[0133] Step S1-3-2: Select the leftmost leaf node in the undirected graph as the starting node of the depth-first traversal, and record it as .

[0134] Step S1-3-3: Starting from the starting node, traverse each key point along the undirected graph path in turn, and assign an increasing sequence number in the order in which they are accessed and record (i.e. ).

[0135] Step S1-3-4: When a branch point or overlap point (collectively referred to as a split point) is reached, these key points are constructed into a set , and then the value of k is set to 1.

[0136] Step S1-3-5: Store the current split point.

[0137] Step S1-3-6: According to the stored split points, select the leftmost branch path that has not been selected in the order of the horizontal direction (X-axis coordinate) of each branch path in the current image coordinate system from left to right. If all path branches are selected, jump to step S1-3-10.

[0138] Step S1-3-7: Enter the branch path selected in step S1-3-6, continue to execute the depth-first traversal strategy, and assign subsequent sequence numbers to the key points on this branch in turn and record.

[0139] Step S1-3-8: If the traversal node reaches a split point, jump to step S1-3-4.

[0140] Step S1-3-9: If the traversal node reaches a leaf node, these key points are constructed into a set , and then the value of k is set to 1.

[0141] Step S1-3-10: Read the last stored split point, and jump to step S1-3-6. If there is no last stored split point, it is determined that step S1-3 is ended, at this time all key points have been accessed and assigned sequence numbers to obtain , and these key points are split into .

[0142] Step S1-4: Remove duplicate key points in the key point set to obtain an optimized key point set .

[0143] Step S1-4-1: Set the first optimized keypoint set . .

[0144] Step S1-4-2: Set the optimized keypoint set sequence number .

[0145] Step S1-4-3: Read the keypoint elements in the keypoint set , and delete the keypoint elements that have appeared in the keypoint sets to , to obtain the optimized keypoint set .

[0146] Step S1-4-4: Set . If , jump to step S1-4-3, otherwise, step S1-4 ends, and the optimized keypoint set .

[0147] As shown in Figure 4 , an example of a branched linear flexible body involved in the present application.

[0148] As shown in Figure 5 , a diagram of an example of a keypoint set, a segmented keypoint set, and an optimized segmented keypoint set. Among them, Figure 5 (a) in is all the key points extracted from the BDLO image, the red and blue points are key points, and the blue points are identified branch key points; Figure 5 (b) in is the 3 keypoint sets segmented; Figure 5 (c) in is the 3 optimized keypoint sets obtained by excluding the repeated key points in the 3 keypoint sets .

[0149] Step S2: Construct the optimized keypoint set into a tensor respectively, use the GCN algorithm to aggregate each optimized keypoint set tensor to obtain an aggregated feature tensor , and splice these feature tensors to obtain a spliced aggregated feature tensor ; wherein the aggregated feature tensor contains a first dimension and a second dimension, the first dimension represents the number of key points, and the second dimension represents the dimension of the information of each key point.

[0150] Step S2-1: constructing an optimization key point set is constructed as a tensor respectively. For each optimization key point tensor , , a corresponding adjacency tensor is constructed.

[0151]

[0152] wherein is the number of key points in the mth optimization key point set .

[0153] Step S2-2: taking each and its corresponding adjacency tensor as input, performing aggregation calculation by a multi-layer GCN method to obtain respective aggregation feature tensors . .

[0154] An example of the architecture of the GCN is as follows:

[0155] 1) The number of layers of the multi-layer GCN method is set to 3 layers;

[0156] 2) The first GCN layer: the input dimension is consistent with the node feature dimension (for example, 2 dimensions), and the output dimension is set to 128 dimensions;

[0157] 3) The second GCN layer: the input dimension is 128 dimensions, and the output dimension is 64 dimensions;

[0158] 4) The third GCN layer: the input dimension is 64 dimensions, and the output dimension is 2 dimensions;

[0159] 5) A ReLU activation function is applied after each GCN layer;

[0160] Step S2-3: each is spliced in the dimension direction of the number of key points to obtain a spliced aggregation feature tensor .

[0161]

[0162] As shown in Figure 6 , it is a flowchart of an example of the aggregation and splicing of the optimization key point tensors in step S2.

[0163] Step S3: based on the spliced aggregation feature tensor , uses a soft mask to represent the different degrees of interaction between key points, and combines the encoding of the first Transformer encoder to construct the branch interaction feature of the aggregated features of the BDLO segment splicing, and obtains the key point global information feature tensor .

[0164] Step S3-1: Through the processing of the linear layer of the first multi-layer perception and the ReLU activation function, the second dimension (another dimension in addition to the number of key points) of is expanded, from which a deeper hidden feature tensor is extracted.

[0165] An example of the architecture of the multi-layer perception is as follows:

[0166] 1) Set the number of fully connected layers of the multi-layer perception method to 3 layers;

[0167] 2) The first fully connected layer: the input dimension is consistent with the GCN output dimension (for example, 2 dimensions), and the output dimension is set to 32 dimensions;

[0168] 3) The second fully connected layer: the input dimension is 32 dimensions, and the output dimension is 64 dimensions;

[0169] 4) The third fully connected layer: the input dimension is 64 dimensions, and the output dimension is 64 dimensions;

[0170] 5) Apply the ReLU activation function after each perception layer;

[0171] Step S3-2: generate a position encoding for each key point element in by using the Sinusoidal Positional Encoding method, and add it to the key point feature vector to obtain an updated hidden feature tensor .

[0172] Step S3-3: create a soft mask tensor , and the initial value of each element of the soft mask tensor is negative infinity.

[0173]

[0174] Step S3-4: update the value of each element in the soft mask tensor according to the undirected graph obtained in step S1-2 .

[0175] Step S3-4-1: for all set pairs in the key point set family that satisfy and , construct a tuple ;

[0176] Step S3-4-2: Construct all tuples into a set ; wherein is the total number of found tuples; represents the first tuple; represents the second tuple; represents the first set in the first tuple; represents the second set in the first tuple; represents the first set in the second tuple; represents the second set in the second tuple;

[0177] Step S3-4-3: Set the initial value of the tuple number to 1; set the weight of each edge in the undirected graph obtained in step S1-2 to 1;

[0178] Step S3-4-4: Set the initial value of the key point number to the minimum value of the key point numbers in the optimized key point set ;

[0179] Step S3-4-5: Set the initial value of the key point number to the minimum value of the key point numbers in the optimized key point set ;

[0180] Step S3-4-6: According to the undirected graph obtained in step S1-2 and the edge weight obtained in step S3-4-3, calculate the shortest distance between the key point and each intersection key point using the Dijkstra algorithm, and select the minimum as the intra-fragment distance of the key point , and record the corresponding as the intra-fragment key point of the key point ;

[0181] Step S3-4-7: If the intra-fragment key point is an overlapping point, set the coefficient ; if the intra-fragment key point is a branch point, set the coefficient ; if the intra-fragment key point is neither an overlapping point nor a branch point, set the coefficient ; wherein , are the first positive constant and the second positive constant set by a person;

[0182] Step S3-4-8: According to the undirected graph obtained in step S1-2 and the edge weight obtained in step S3-4-3, the shortest distance between the key point and is calculated by Dijkstra algorithm ;

[0183] Step S3-4-9: The is added to , and the geodesic distance between the key point and the key point is obtained ;

[0184] Step S3-4-10: Based on the geodesic distance between the key point and the key point and the coefficient , the corresponding mask tensor element is updated ;

[0185]

[0186] wherein, is a positive constant set by man, and its value needs to meet , is a number that approaches 0 from positive infinity;

[0187] Step S3-4-11: If , the value of is set to 1, and jump to step S3-4-6; wherein represents the number of key point elements in the set ;

[0188] Step S3-4-12: If , the value of is set to 1, and jump to step S3-4-5; wherein, represents the number of key point elements in the set ;

[0189] Step S3-4-13: If , the value of is set to 1, and jump to step S3-4-4;

[0190] Step S3-4-14: When , step S3-4 ends, and the update of the element in the soft mask tensor has been completed.

[0191] Step S3-5: The is added to the soft mask tensor The BDLO is processed using a multi-layer Transformer encoder, employing a multi-layer self-attention mechanism and a feedforward neural network to represent the interaction features of each segment, outputting a feature tensor that fuses global information. .

[0192] An example of the architecture of a multi-layer Transformer encoder is as follows:

[0193] 1) The number of layers in the first multi-layer Transformer encoder is set to 6;

[0194] 2) The encoder contains 16 attention heads, the model dimension and the feedforward network dimension are 64, and layer normalization is used.

[0195] 3) Input dimension and updated hidden feature tensor Consistent, with an output dimension of 32.

[0196] like Figure 7 The diagram shown is a flowchart of an example of calculating the global information feature tensor of key points in step S3.

[0197] like Figure 8 The diagram illustrates an instance of updating a soft mask value in steps S3-4. Figure 8 (a) shows the distance d6 of the 6th keypoint within a fragment of the first set of a certain tuple, and the distance d of the 37th keypoint within a fragment of the first set of a certain tuple. 37 , Figure 8 (b) shows the geodesic distance between the 6th and 37th key points. Update the corresponding mask value in the soft mask (here) The value is 2. (Value is 70) .

[0198] Step S4: Combine the current state with the global information feature tensor With the global information feature tensor of the user's expected state By encoding and decoding based on an attention mechanism, the robot's grasping key points are extracted from the current state. Extracting the robot's placement key points from the desired state This enables dynamic modeling of robot actions from BDLO state changes.

[0199] Step S4-1: Based on the desired state image given by the user , execution and The same steps S1 to S3 are performed to obtain the global information feature tensor for the user's desired state. .

[0200] Step S4-2: Concatenate the global feature tensor with along the first dimension (dimension of the number of key points) to obtain a concatenated global feature tensor .

[0201] Step S4-3: Generate a position encoding for each key point element in by a sine-cosine position encoding method, and add it to the key point feature vector to obtain an updated concatenated global feature tensor .

[0202] Step S4-4: Create a hard mask tensor to isolate the attention calculation inside the current time key point and the expected time key point.

[0203]

[0204] Step S4-5: Input and the hard mask tensor to a multi-layer second Transformer encoder module, and perform sequence modeling through a multi-layer self-attention mechanism and a feedforward neural network to represent the features of the BDLO cross-time domain relationship, and output a feature encoding tensor that fuses the current state and the user's expected state.

[0205] An example of the architecture of the multi-layer second Transformer encoder is as follows:

[0206] 1) The number of layers of the multi-layer second Transformer encoder is set to 6 layers.

[0207] 2) The encoder contains 16 attention heads, the model dimension and the feedforward network dimension are 64, and layer normalization is used.

[0208] 3) The input dimension is consistent with the concatenated global feature tensor , and the output dimension is 32.

[0209] Step S4-6: Decode the feature encoding tensor by processing through the linear layer of the second multi-layer perceptron and the ReLU activation function to obtain a score tensor .

[0210]

[0211] wherein is the score of the corresponding key point in the current state; is the score of the corresponding key point the score of the current state.

[0212] One example of the architecture of the multi-layer perceptron is as follows:

[0213] 1) Set the number of fully connected layers of the multi-layer perceptron method to 3 layers;

[0214] 2) The first fully connected layer: the input dimension is consistent with the output of the multi-layer second Transformer encoder module in step S4-5 (for example, 32 dimensions), and the output dimension is set to 128 dimensions;

[0215] 3) The second fully connected layer: the input dimension is 128, and the output dimension is 64;

[0216] 4) The third fully connected layer: the input dimension is 64, and the output dimension is 1;

[0217] 5) Apply the ReLU activation function after each perceptron layer;

[0218] Step S4-7: Select the maximum The corresponding as the robot's grasp key point ; select the maximum The corresponding as the robot's placement key point .

[0219] As Figure 9 shown, a flowchart of one example of selecting a robot grasping and placing point in step S4 in combination with the current state and the desired state.

[0220] In one specific embodiment, as Figure 10 shown, Figure 10 (a) in is the state of the current BDLO, Figure 10 (b) in is the simulation diagram of the simulation robot grasping the BDLO.

[0221] As Figure 11 shown, Figure 11 (a) in is the state of the desired BDLO; Figure 11 (b) in is the simulation diagram of the BDLO desired state obtained after the simulation robot operates.

[0222] By the above-mentioned branch linear flexible body dynamics modeling method for robot operation, on the one hand, the BDLO image is taken as a whole, and key points and topological structures thereof are extracted by an image recognition method; and the BDLO image is divided into a plurality of segments with branch points and overlapping points as boundaries. In the local feature aggregation method, for the separated single linear flexible body, an adjacency tensor is established according to the connection relationship of the key points; and the key points and the adjacency tensor in the segment are processed by a GCN method, so as to aggregate and represent the features in each segment. Using this simple aggregation means to simulate the deformation propagation mechanism between the key points of the single linear flexible body can reduce the calculation complexity and response time of the model while ensuring the accuracy of the dynamics modeling. In the branch interaction construction method, the soft mask values are set based on the geodesic distance between the key points of different segments, and the mask values of the relationships between the remaining key points are set as negative infinity; the soft mask tensor and the key point features are processed by a multi-layer first Transformer encoder to express the relationship features of the interactions of the BDLO segments. Using this soft mask setting interaction degree method can reflect the different features of the interactions of the segments connected by the branch points or the overlapping points, and complete the nonlinear and anisotropic modeling of different deformation propagation modes. In the state joint solving method, the processing of the current BDLO state and the processing of the expected BDLO state are combined, and the robot grasping position and placing position are obtained by coding by a second Transformer encoder and decoding by a multi-layer perception machine, so as to realize the dynamics modeling from the BDLO state change to the robot action. On the other hand, when the robot operates the BDLO under the guidance of the constructed dynamics model, the state of the BDLO is changed by each robot operation, which further affects the overlapping points and the segmented segments, and thus affects the transmission of the connection force and the friction, so that the BDLO dynamics modeling has dynamics. The method realizes the dynamics modeling for the dynamics of the BDLO by re-segmenting the segments at each time and recalculating the soft mask. By effectively processing the inherent complex characteristics such as high dimensionality, nonlinearity, dynamics and anisotropy in the BDLO dynamics modeling, the method realizes accurate and reliable modeling of the dynamics behavior of the BDLO.

[0223] In addition, the terms "first", "second", "third", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0224] In the description of the disclosure, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the disclosure. In the description of the disclosure, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0225] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the disclosure that follow the general principles thereof and include the general principles thereof and include other known or customary features not specifically mentioned herein. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the disclosure are indicated by the appended claims.

Claims

1. A method for modeling the dynamics of a branched linear flexible body for robot manipulation, characterized in that, Comprise: Step S1: Extract from BDLO images captured by the camera N Each key point will be segmented and optimized to obtain several optimized key point sets; specifically including: Step S1-1: Obtain the BDLO image collected by the camera, extract the key points and their topological relations of the BDLO image, and identify the branch points and intersection points in the BDLO image. N Step S1-1: Obtain the BDLO image collected by the camera, extract the key points and their topological relations of the BDLO image, and identify the branch points and intersection points in the BDLO image. Step S1-2: taking each key point as a vertex set and the topological relationship between the key points as an edge set to construct an undirected graph; Step S1-3: segmenting the key points according to the branch points, the overlapping points and the undirected graph to obtain a key point set; Step S1-4: optimizing the key point set to exclude duplicate points to obtain a plurality of optimized key point sets; Step S2: for each optimized key point set, constructing a key point tensor and an adjacency tensor, using a GCN algorithm to aggregate features of each key point tensor and adjacency tensor respectively to obtain an aggregated feature tensor, and splicing all the aggregated feature tensors to obtain a spliced aggregated feature tensor; specifically comprising: Step S2-1: optimizing the key point set are respectively constructed as tensors ; Step S2-2: For each optimized keypoint tensor , , construct a corresponding adjacency tensor ; Step S2-3: Optimize the tensor of each key point and its corresponding adjacency tensor As input, the multi-layer GCN algorithm is used for aggregation calculation to obtain the tensors of each optimization key point. Aggregate feature tensor ; convert each aggregated feature tensor By concatenating the features along the dimension of the number of key points, we obtain the concatenated aggregated feature tensor. Among them, the aggregated feature tensor It includes a first dimension and a second dimension. The first dimension represents the number of key points, and the second dimension represents the information dimension of each key point. Step S3: using a first multi-layer perception to expand the dimension of the spliced aggregated feature tensor, using a soft mask to represent different degrees of interaction relationship between the key points, combining the encoding of the first Transformer encoder to construct branch interaction relationship features of the spliced aggregated feature tensor to obtain a global information feature tensor of the current state; specifically comprising: Step S3-1: expanding the aggregated feature tensor through a linear layer of the first multi-layer perceptron and a ReLU activation function from the second dimension thereof, from which a deeper level of hidden feature tensor is extracted ; Step S3-2: Encode the hidden feature tensor using sine and cosine position encoding. Each keypoint element in the dataset generates a positional code, which is then added to the keypoint's feature vector to obtain the updated hidden feature tensor. ; Step S3-3: Create a soft mask tensor , the initial value of each element of the soft mask tensor is negative infinity; Step S3-4: Update the soft-mask tensor with the undirected graph of element values ; wherein, i is the row of the soft-mask tensor i , j is the column of the soft-mask tensor j ; Step S3-5: updating the hidden feature tensor with the updated soft mask tensor , processing by using the first Transformer encoder with multiple layers, representing the interaction features between the concatenated and aggregated feature tensors by using the multi-layer self-attention mechanism and the feedforward neural network, and outputting the global information fusion feature tensor ; Step S4: according to the global information feature tensor of the current state and the global information feature tensor of the expected state, through the encoding of the second Transformer encoder and the decoding of the second multi-layer perception, obtaining the grasping position and the placing position of the robot, realizing the dynamics modeling from the BDLO state change to the robot action.

2. The method of claim 1, wherein, Step S1-3 specifically comprises: Step S1-3-1: Set the sequence number of the key point set k is 1; Step S1-3-2: Select the leftmost leaf node in the undirected graph as the starting node of the depth-first traversal, and record it as ; Step S1-3-3: Starting from the starting node, each key point is sequentially traversed along the undirected graph path, and an increasing serial number is continuously assigned in the order in which it is accessed and recorded as ; Step S1-3-4: when traversing to a split point, the split key points are constructed as a key point set , and then setting k value + 1; wherein the split point includes a branch point and an overlap point; Step S1-3-5: store the current segmentation point; Step S1-3-6: according to the stored segmentation point, select the leftmost branch path that has not been selected in the order of the horizontal direction from left to right of each branch path in the current image coordinate system; if all path branches are selected, jump to step S1-3-10; Step S1-3-7: enter the branch path selected in step S1-3-6, continue to execute the depth-first traversal strategy, and continuously assign subsequent serial numbers to the key points on the branch and record them; Step S1-3-8: if the traversal node reaches the segmentation point, jump to step S1-3-4; Step S1-3-9: If the traversal node reaches a leaf node, the key point is constructed as a set , and then the value is set k value + 1; Step S1-3-10: read the last stored split point, jump to step S1-3-6; if there is no last stored split point, the splitting ends, all key points are visited and assigned serial numbers to get , and all key points are split into key point sets .

3. The method of claim 2, wherein the robot operation oriented branched linear flexible body dynamics modeling method is characterized by, Step S1-4 specifically comprises: Step S1-4-1: Set the first optimized key point set The same as the first key point set ​ Step S1-4-2: Set the sequence number of the optimization key point set and the key point set is 2; Step S1-4-3: reading the key point elements in the key point set and deleting the key point elements in which the key point elements have already appeared in the key point set to the key point set , to obtain the optimized key point set ; Step S1-4-4: setting value + 1; if jump to step S1-4-3; if end, and get the optimized key point set .

4. The method of claim 3, wherein, Step S3-4 specifically comprises: Step S3-4-1 : For the set of key points all satisfying and , construct it as a tuple ; wherein ; Step S3-4-2: Construct all tuples into sets ; wherein, is the total number of tuples, represents the first tuple, represents the second tuple, represents the ordinal number of the first set in the first tuple, represents the ordinal number of the second set in the first tuple, represents the ordinal number of the first set in the second tuple, and represents the ordinal number of the second set in the second tuple. Step S3-4-3: Set the sequence number of the tuple The initial value of the weight of the edge in the undirected graph is set to 1. Step S3-4-4: Set the optimized key point set The value of the key point sequence number is the minimum value ; Step S3-4-5: Set the optimized key point set The value of the key point sequence number is the smallest ; Step S3-4-6: According to the undirected graph and its edge weight, calculate the key point by Dijkstra algorithm The shortest distance of each intersection key point And select the smallest As the intra-fragment distance of the key point And record the corresponding intersection key point As the fragment matching key point of the key point ;​​​ Step S3-4-7: If the segment matches a key point is an overlap point, set the coefficient ; If the segment matches a key point is a branch point, set the coefficient ; If the segment matches a key point is neither an overlap point nor a branch point, set the coefficient ; wherein is a first positive constant, is a second positive constant; Step S3-4-8: According to the undirected graph and its edge weights, calculate the key points by Dijkstra algorithm Key points matching with segments Shortest distance ; Step S3-4-9: add up the geodesic distances between the key points and the key points to obtain the geodesic distance between the key points and the key points ; Step S3-4-10: updating the first element of the mask tensor based on the geodesic distance of the key point and the coefficient and the second element of the mask tensor ;​​​ Step S3-4-11: If , set value plus 1, and jump to step S3-4-6; wherein, represents the number of key point elements in the optimized key point set . Step S3-4-12: If , set value plus 1, and jump to step S3-4-5; wherein denotes the number of key point elements in the set . Step S3-4-13: If , set value plus 1, and jump to step S3-4-4; Step S3-4-14: When the update is finished, the update of the elements in the soft mask tensor is completed. ​ 5. The method of claim 4, wherein, Step S4 specifically comprises: Segmenting the expected BDLO state, locally aggregating features, and constructing branch interactions of different degrees to obtain a global information feature tensor of the expected state; Splicing the global information feature tensor of the current state and the global information feature tensor of the expected state to obtain a spliced global feature tensor; Position encoding is performed on the spliced global feature tensor to obtain an updated spliced global feature tensor; A hard mask tensor for isolating attention calculation between the key points at the current time and the key points at the expected time is created; The updated spliced global feature tensor and the hard mask tensor are input, and a multi-layer second Transformer encoder is used for processing to represent the features of the BDLO cross-time domain relationship, and a feature encoding tensor that fuses the current state and the user's expected state is obtained; A second multi-layer perception is used to decode the feature encoding tensor to obtain a score tensor; According to the score tensor, the grasping position and the placing position of the robot are selected, and the dynamics modeling from the BDLO state change to the robot action is realized.

Images