Dexterous hand multi-modal motion trajectory prediction model generation method and related device

Abstract

The application relates to a dexterous hand multi-modal motion trajectory prediction model generation method and related devices, which are used for simulating a dexterous hand grabbing process, and the method comprises the following steps: generating a motion trajectory of the dexterous hand according to initial position information of the dexterous hand and end position information of a target object; acquiring a multi-modal data set of the dexterous hand in the motion trajectory based on a plurality of motion trajectory points, wherein the multi-modal data set comprises environmental image data, first multi-modal data and second multi-modal data; performing multi-modal feature fusion on the environmental image data, the first multi-modal data and the second multi-modal data to obtain an input sequence; inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to train the self-attention model and obtain a motion trajectory prediction model. According to the scheme, the real-time performance and the accuracy of dexterous hand motion trajectory prediction can be improved through the fusion of multi-modal data, and the grabbing success rate of the dexterous hand is improved.

Description

Technical Field

[0001] This application relates to the field of robot simulation technology, and in particular to a method and related apparatus for generating a multimodal motion trajectory prediction model of a dexterous hand. Background Technology

[0002] Humanoid dexterous hands possess extremely high flexibility and human-like structural features, enabling them to perform delicate operations such as tool handling and various hand manipulation tasks. The grasping motion planning of dexterous hands is a crucial robotics task, essential for the efficient and safe movement and interaction of robots in real-world environments. Its goal is to generate feasible robot trajectories within a configuration space and achieve the objective in the optimal manner according to given criteria.

[0003] Currently, dexterous hands mainly rely on conventional trajectory planning algorithms in simulated grasping processes, which are difficult to adapt to the needs of different object shapes and grasping scenarios, resulting in unstable grasping success rates and affecting the operational performance and adaptability of dexterous hands in complex scenarios. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a method and related apparatus for generating a multimodal motion trajectory prediction model for dexterous hands. By fusing multimodal data, the real-time performance and accuracy of dexterous hand motion trajectory prediction can be improved, thereby increasing the success rate of dexterous hand grasping.

[0005] The first aspect of this application provides a method for generating a multimodal motion trajectory prediction model for a dexterous hand, used in a dexterous hand simulation grasping process, comprising: generating a motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object, the motion trajectory including a plurality of motion trajectory points; acquiring a multimodal dataset of the dexterous hand moving in the motion trajectory based on the plurality of motion trajectory points, the multimodal dataset including environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand to move in the motion trajectory; performing multimodal feature fusion on the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model.

[0006] In conjunction with the first aspect, in one possible implementation of the first aspect, the step of generating the movement trajectory of the dexterous hand based on the initial position information of the dexterous hand and the end position information of the target object, the movement trajectory including a plurality of movement trajectory points, includes: generating a first movement trajectory point, a second movement trajectory point, and a third movement trajectory point of the dexterous hand based on the initial position information of the dexterous hand, the initial position information of the target object, and the end position information; acquiring movement information of the dexterous hand at the first movement trajectory point, the second movement trajectory point, and the third movement trajectory point, the movement information including the position information, posture information, and finger closure information of the dexterous hand; generating a plurality of first trajectory transition points between the first movement trajectory point and the second movement trajectory point, and a plurality of second trajectory transition points between the second movement trajectory point and the third movement trajectory point, based on a preset optimized interpolation algorithm and the movement information; and determining the movement trajectory of the dexterous hand based on the first movement trajectory point, the first trajectory transition point, the second movement trajectory point, the second trajectory transition point, and the third movement trajectory point.

[0007] In conjunction with the first aspect, in one possible implementation of the first aspect, generating the second motion trajectory points of the dexterous hand based on the initial position information of the target object includes: acquiring point cloud data of the target object at the initial position based on the initial position information of the target object; and generating the second motion trajectory points of the dexterous hand based on a preset shape matching algorithm and the point cloud data.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, the first multimodal data includes the first joint information and the first fingertip contact force information of the dexterous hand, the second multimodal data includes the second joint information of the robotic arm, and the first joint information includes the first finger joint information; the step of multimodal data fusion of the environmental image data, the first multimodal data, and the second multimodal data to obtain the input sequence includes: extracting features from the first joint information, the first fingertip contact force information, the second joint information, and the environmental image data to obtain multiple sets of feature tensors, the multiple sets of feature tensors including a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environmental image data, the first feature tensor including the first finger joint feature tensor; fusing the multiple sets of feature tensors belonging to continuous time steps to obtain the input sequence.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, the step of inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model includes: dividing the input sequence into several input sub-sequences, the input sub-sequences corresponding to different motion stages of the dexterous hand; selecting target feature tensors from the input sub-sequences based on motion stage constraints and generating a target feature tensor set; performing feature learning on the target feature tensor set based on the sparse self-attention mechanism and a preset modality locking strategy to obtain a predicted motion sequence, the predicted motion sequence being used to characterize the multimodal prediction data of the dexterous hand and the robotic arm at the motion trajectory points, the preset modality locking strategy being used to control the second feature tensor to interact only with the first finger joint feature tensor; and adjusting the weight parameters of the self-attention model to be trained based on the loss function, the predicted motion sequence, and the input sequence to obtain the motion trajectory prediction model.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, the fourth feature tensor includes a top image feature tensor, a bottom image feature tensor, and a wrist image feature tensor. The step of selecting a target feature tensor from the input sub-sequence based on motion phase constraints and generating a target feature tensor set includes: determining the motion phase of the dexterous hand based on the motion trajectory points of the dexterous hand, wherein the motion phase includes a movement phase, a grasping phase, and a post-grasping movement phase; when the input sub-sequence corresponds to the movement phase, selecting the first feature tensor, the top image feature tensor, the bottom image feature tensor, and the third feature tensor from the input sub-sequence as the target. The first feature tensor, the bottom image feature tensor, the wrist image feature tensor, the second feature tensor, and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated; and / or, when the input subsequence corresponds to the grasping stage, the first feature tensor, the top image feature tensor, the wrist image feature tensor, the second feature vector, and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated.

[0011] A second aspect of this application provides a motion trajectory prediction method for dexterous hand simulation grasping, characterized in that the motion trajectory prediction method includes: acquiring first initial modal data of the dexterous hand at an initial position, second initial modal data of the robotic arm at an initial position, and initial environmental image data; inputting the first initial modal data, the second initial modal data, and the initial environmental image data into a motion trajectory prediction model to obtain a predicted motion trajectory sequence of the dexterous hand, the predicted motion trajectory sequence including predicted multimodal data of the dexterous hand and predicted multimodal data of the robotic arm.

[0012] A third aspect of this application provides a device for generating a multimodal motion trajectory prediction model for a dexterous hand, used for simulating a grasping process. The device comprises: a motion trajectory generation module, used to generate a motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object, the motion trajectory including a plurality of motion trajectory points; a dataset acquisition module, used to acquire a multimodal dataset of the dexterous hand's motion in the motion trajectory based on the plurality of motion trajectory points, the multimodal dataset including environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand's motion in the motion trajectory; a fusion module, used to fuse the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and a training module, used to input the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model.

[0013] In conjunction with the third aspect, in one possible implementation of the second aspect, the motion trajectory generation module is further configured to generate a first motion trajectory point, a second motion trajectory point, and a third motion trajectory point of the dexterous hand based on the initial position information of the dexterous hand, the initial position information of the target object, and the final position information of the target object; acquire motion information of the dexterous hand at the first motion trajectory point, the second motion trajectory point, and the third motion trajectory point, respectively, wherein the motion information includes the position information, posture information, and finger closure information of the dexterous hand; generate a plurality of first trajectory transition points between the first motion trajectory point and the second motion trajectory point, and a plurality of second trajectory transition points between the second motion trajectory point and the third motion trajectory point, based on a preset optimized interpolation algorithm and the motion information; and determine the motion trajectory of the dexterous hand based on the first motion trajectory point, the first trajectory transition point, the second motion trajectory point, the second trajectory transition point, and the third motion trajectory point.

[0014] In conjunction with the third aspect, in one possible implementation of the second aspect, the motion trajectory generation module is further configured to obtain point cloud data of the target object at its initial position based on the initial position information of the target object; and generate the second motion trajectory points of the dexterous hand based on a preset shape matching algorithm and the point cloud data.

[0015] In conjunction with the third aspect, in one possible implementation of the second aspect, the training module is further configured to extract features from the first joint information, the first fingertip contact force information, the second joint information, and the environmental image data to obtain multiple sets of feature tensors. These multiple sets of feature tensors include a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environmental image data. The first feature tensor includes a first finger joint feature tensor. The multiple sets of feature tensors belonging to consecutive time steps are then fused to obtain the input sequence.

[0016] In conjunction with the third aspect, in one possible implementation of the second aspect, the training module is further configured to divide the input sequence into several input sub-sequences, the input sub-sequences corresponding to different movement stages of the dexterous hand; based on movement stage constraints, select target feature tensors from the input sub-sequences and generate a set of target feature tensors; perform feature learning on the set of target feature tensors based on a sparse self-attention mechanism and a preset modality locking strategy to obtain a predicted motion sequence, the predicted motion sequence being used to characterize the multimodal prediction data of the dexterous hand and the robotic arm at the motion trajectory points, the preset modality locking strategy being used to control the second feature tensor to interact only with the first finger joint feature tensor; and adjust the weight parameters of the self-attention model to be trained based on the loss function, the predicted motion sequence, and the input sequence to obtain the motion trajectory prediction model.

[0017] In conjunction with the third aspect, in one possible implementation of the second aspect, the training module is further configured to determine the movement phase of the dexterous hand based on the movement trajectory points of the dexterous hand, the movement phase including a movement phase, a grasping phase, and a post-grasping movement phase; when the input subsequence corresponds to the movement phase, the first feature tensor, the top image feature tensor, the bottom image feature tensor, and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated; and / or, when the input subsequence corresponds to the grasping phase, the first feature tensor, the bottom image feature tensor, the wrist image feature tensor, the second feature tensor, and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated; and / or, when the input subsequence corresponds to the post-grasping movement phase, the first feature tensor, the top image feature tensor, the wrist image feature tensor, the second feature vector, and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated.

[0018] A fourth aspect of this application provides an electronic device, comprising:

[0019] Processor; and

[0020] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0021] A fifth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0022] The technical solution provided in this application may include the following beneficial effects:

[0023] This application discloses a method and related apparatus for generating a multimodal motion trajectory prediction model for a dexterous hand, used for simulating the grasping process of a dexterous hand. The motion trajectory prediction model generation method includes: generating the motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object, the motion trajectory including several motion trajectory points; acquiring a multimodal dataset of the dexterous hand's movement in the motion trajectory based on the several motion trajectory points, the multimodal dataset including environmental image data, first multimodal data corresponding to each motion trajectory point of the dexterous hand, and second multimodal data of the robotic arm controlling the dexterous hand's movement in the motion trajectory; fusing multimodal features of the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model. By fusing multimodal data, the real-time performance and accuracy of the dexterous hand's motion trajectory prediction can be improved, thereby increasing the grasping success rate of the dexterous hand.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0025] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0026] Figure 1 This is a flowchart illustrating the method for generating a multimodal motion trajectory prediction model for a dexterous hand, as shown in an embodiment of this application.

[0027] Figure 2 This is another flowchart illustrating the method for generating a multimodal motion trajectory prediction model for a dexterous hand, as shown in the embodiments of this application.

[0028] Figure 3 This is a schematic diagram of the structure of the dexterous hand multimodal motion trajectory prediction model generation device shown in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram illustrating the dexterity hand motion trajectory prediction method of the dexterity hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram of the top image of the dexterous hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application;

[0032] Figure 7 This is a schematic diagram of the bottom image of the dexterous hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application;

[0033] Figure 8 This is a schematic diagram of a wrist image used in the dexterity hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application;

[0034] Figure 9 This is a schematic diagram of the robotic arm and dexterous hand in the dexterous hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application;

[0035] Figure 10 This is a partial structural diagram of the dexterous hand in the dexterous hand multimodal motion trajectory prediction model generation method shown in the embodiments of this application. Detailed Implementation

[0036] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a” and “the” as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0039] As a crucial component of biomimetic robotic systems, dexterous hands hold significant value in areas such as complex object manipulation, precision assembly, and human-robot collaboration. Compared to traditional rigid grippers, dexterous hands typically possess multi-degree-of-freedom joint structures, enabling more flexible and diverse grasping and manipulation. However, due to factors such as high-dimensional control, complex dynamics, and environmental uncertainties, achieving stable and efficient autonomous grasping still presents numerous challenges.

[0040] Grasping motion planning is a crucial robotics task, essential for the efficient and safe movement and interaction of robots in real-world environments. Its goal is to generate feasible robot trajectories within a configuration space and achieve the objective optimally according to given criteria. Trajectory feasibility typically refers to collision avoidance and constraint compliance, such as robot joint limitations, while optimality usually refers to trajectory smoothness, acceleration minimization, execution path length, or similar qualitative metrics.

[0041] Currently, dexterous hand grasping control mainly relies on rule-based planning methods and reinforcement learning techniques. The former typically achieves its goal by pre-setting grasping strategies or force control schemes, but it suffers from poor versatility when dealing with objects of different shapes, materials, and constraints, making it difficult to adapt to complex tasks. The latter employs deep reinforcement learning algorithms to learn the optimal strategy through interactive trial and error. However, these methods often require a large amount of online interactive data, resulting in high training time costs. Furthermore, when transferred to real robots, they are susceptible to simulation-reality bias, leading to insufficient generalization ability.

[0042] To address the aforementioned issues, this application provides a method for generating a motion trajectory prediction model, a prediction method, and related devices. By fusing multimodal data, the real-time performance and accuracy of dexterous hand motion trajectory prediction are improved, thereby increasing the success rate of dexterous hand grasping.

[0043] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This is a flowchart illustrating the method for generating a multimodal motion trajectory prediction model for a dexterous hand, as shown in an embodiment of this application.

[0045] See Figure 1 , Figure 5 , Figure 9 and Figure 10 A method for generating a multimodal motion trajectory prediction model for a dexterous hand, used to simulate the grasping process of a dexterous hand, includes:

[0046] S110: Generate the motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object. The motion trajectory includes several motion trajectory points.

[0047] Specifically, the process can begin by designing a 3D model of the target object, such as a robotic arm or dexterous hand, using design software. This ensures that the model's dimensions, mass distribution, and joint connection methods meet the requirements of the actual application. The designed model is then imported into a simulation model in a suitable format. The simulation model can be configured with a suitable simulation environment, allowing for the preset initial position information of the dexterous hand and the final position information of the target object. Based on these information, the motion estimate of the dexterous hand is generated. The motion trajectory can be a continuous path containing key points and transition points constructed from the initial and final positions, and includes multiple motion trajectory points.

[0048] S120: Based on several motion trajectory points, obtain a multimodal dataset of the dexterous hand moving in the motion trajectory. The multimodal dataset includes environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand to move in the motion trajectory.

[0049] Specifically, a platform can be set up in the simulation environment, on which the dexterous hand and the target object can be placed. Joint velocity sensors and torque sensors for the robotic arm and dexterous hand, tactile sensors for the dexterous fingertips, and visual sensors for the top, bottom, and wrist of the platform can be added to the simulation environment to provide feedback information during the grasping process. Based on multiple motion trajectory points on the motion trajectory, multimodal data of the dexterous hand and robotic arm can be acquired. The multimodal dataset integrates visual perception, joint motion, and tactile feedback information. For example, a camera located at the top of the platform can collect object positioning information from a top-down perspective, a wrist camera can capture images of the grasping dexterous hand's wrist, and various sensors can be used to record joint information such as joint angles, joint velocities, and joint torques of the robotic arm and dexterous hand, contact force information of the dexterous fingertips, and environmental image data of the top, bottom, and wrist of the platform.

[0050] S130: Perform multimodal feature fusion on environmental image data, first multimodal data, and second multimodal data to obtain the input sequence.

[0051] Specifically, feature fusion can be performed on environmental image data, first multimodal data, and second multimodal data to obtain the input sequence. Feature fusion can be to convert time-series data of different modalities into a unified tensor format, such as by stitching together ten consecutive frames of data into an input sequence through time step stacking, or by using a lightweight convolutional network to process images to obtain 256-dimensional visual features.

[0052] S140: Based on the preset sparse self-attention mechanism, the input sequence is input into the self-attention model to be trained to obtain the motion trajectory prediction model.

[0053] Specifically, sparse self-attention mechanism refers to an attention calculation method based on dynamically selecting effective modalities during the motion phase. By inputting the input sequence into the self-attention model to be trained, the sparse attention mechanism masks the attention weights of non-activated modalities according to the current motion phase. For example, in the movement phase after grasping, only the top visual and joint information interaction is retained, which can obtain a trained motion trajectory prediction model. By adding sparse self-attention mechanism to the model, the learning efficiency of key features can be improved while reducing the amount of computation.

[0054] In related technologies, dexterous hand grasping control methods mainly rely on rule-based policies or deep reinforcement learning algorithms. Rule-based policies are poorly adaptable to objects of different shapes or materials, making them difficult to handle complex task requirements. While deep reinforcement learning methods can learn strategies through trial and error, the training process requires a large amount of interactive data and suffers from transfer bias between simulation and reality. Traditional imitation learning techniques, although utilizing expert demonstration trajectories, are prone to low data utilization and insufficient action space learning efficiency when processing high-dimensional multimodal sensor information. For example, in dynamic grasping scenarios, existing methods struggle to effectively integrate visual, tactile, and joint information, leading to inaccurate grasping trajectory prediction.

[0055] This application discloses a method for generating a multimodal motion trajectory prediction model for a dexterous hand, used in the simulation of a dexterous hand grasping process. The method includes: generating a motion trajectory of the dexterous hand based on its initial position information and the final position information of the target object; the motion trajectory includes several motion trajectory points; based on these motion trajectory points, obtaining a multimodal dataset of the dexterous hand's movement within the motion trajectory, including environmental image data, first multimodal data corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand's movement within the motion trajectory; fusing multimodal features of the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model. This method improves the real-time performance and accuracy of dexterous hand motion trajectory prediction through the fusion of multimodal data, thereby increasing the success rate of dexterous hand grasping.

[0056] Figure 2 This is another flowchart illustrating the method for generating a multimodal motion trajectory prediction model for a dexterous hand, as shown in the embodiments of this application.

[0057] See Figure 2 A method for generating a multimodal motion trajectory prediction model for a dexterous hand, used to simulate the grasping process of a dexterous hand, includes:

[0058] S210: Generate the motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object. The motion trajectory includes several motion trajectory points.

[0059] Specifically, after importing the designed model into the simulation software in a suitable format, you can set the relevant environmental parameters in the simulation software, such as setting the gravitational acceleration to 9.81 m / s²; setting friction parameters, including the coefficient of friction between the dexterous hand surface and the target object surface, to match the actual friction of different materials; setting motor parameters, including maximum torque and speed limit, to ensure that the dexterous hand's movements are reasonable; and setting object properties, including density, mass distribution, surface material, etc., to ensure that the physical response in the simulation is reasonable.

[0060] In one possible implementation, a motion trajectory of the dexterous hand is generated based on the initial position information of the dexterous hand and the final position information of the target object. The motion trajectory includes several motion trajectory points, including: generating a first motion trajectory point, a second motion trajectory point, and a third motion trajectory point of the dexterous hand based on the initial position information of the dexterous hand, the initial position information of the target object, and the final position information of the target object; acquiring motion information of the dexterous hand at the first motion trajectory point, the second motion trajectory point, and the third motion trajectory point, respectively, the motion information including the position information, posture information, and finger closure information of the dexterous hand; generating several first trajectory transition points between the first motion trajectory point and the second motion trajectory point, and several second trajectory transition points between the second motion trajectory point and the third motion trajectory point, based on a preset optimized interpolation algorithm and the motion information; and determining the motion trajectory of the dexterous hand based on the first motion trajectory point, the first trajectory transition points, the second motion trajectory point, the second trajectory transition points, and the third motion trajectory point.

[0061] Specifically, the first trajectory point refers to the trajectory point at the starting position of the dexterous hand, which can be defined by the initial position coordinates and attitude angles, and is used to establish the starting reference for trajectory planning. The second trajectory point is the trajectory point at the target object's grasping position, which can be calculated by matching the target object's initial position coordinates and surface geometric features. The third trajectory point is the placement point of the target object, which can be determined based on the target object's ending position coordinates and placement attitude, ensuring that the target object is accurately transferred to the target area.

[0062] Specifically, in the trajectory generation process, three key trajectory points are first determined based on the initial position coordinates of the dexterous hand and the initial and final position information of the target object. At each key point, multi-dimensional motion information including position, posture, and finger closure is simultaneously collected. Then, an optimized interpolation algorithm is introduced between the key points. By dynamically adjusting the interpolation coefficients and introducing mechanical feedback compensation, a sequence of transition points that satisfy the motion constraints of the robotic arm is generated. The complete motion trajectory is constructed by the orderly connection of the motion trajectory points and the trajectory transition points. The optimized interpolation algorithm refers to a nonlinear interpolation method based on physical constraints. For example, cubic spline interpolation combined with dynamic feedback can be used to adjust the curvature of the motion trajectory to ensure that the generated transition points meet the joint velocity limits and collision avoidance conditions, thereby improving the smoothness of the trajectory.

[0063] Specifically, the first motion trajectory point can be set. Second trajectory point The third trajectory point At each motion trajectory point Includes the following information: Timestamp: (Indicates the time corresponding to this trajectory point), Dexterity hand position information: Postural information of dexterous hands: (Usually represented as a quaternion), finger closure information of a dexterous hand: (Range 0-1, representing the degree of finger opening or closing of the dexterous hand), position information is represented by three-dimensional spatial coordinates to indicate the position state of the dexterous hand's end effector; posture information is described by quaternions to indicate the spatial orientation of the dexterous hand; finger closure information is represented by normalized values ​​to indicate the degree of finger opening or closing. By embedding physical constraints during keypoint interpolation and through continuous optimization, multiple trajectory transition points are obtained, ensuring that the generated motion trajectory is both smooth and meets mechanical stability requirements. Location information of dexterous hands at all times The following formula can be used for calculation:

[0064]

[0065] in, It can be obtained using the following formula:

[0066]

[0067]

[0068] in, It is an offset vector dynamically generated based on feedback from the force sensor; It compensates for the tilt direction of the object after assessing the risk of sliding or falling while in a gripping state. The value is set based on practical experience. and It can be obtained directly. and Values ​​can be selected according to actual needs, for example and There is a certain correspondence, which can be based on dynamically obtain the value The value of .

[0069] Specifically, regarding the posture information of dexterous hands To ensure smooth changes in posture during the grasping process, spherical linear interpolation is used, and quaternions are defined. , Indicating posture, this refers to the posture information of the dexterous hand. The calculation formula can be:

[0070]

[0071] in yes , The angle between them.

[0072] Information on finger closure for dexterous hands A method of dynamically adjusting the closing speed can be adopted, and the finger closing process can be adaptively adjusted according to factors such as the surface material of the object and the contact force, so that the finger closing process is smooth and meets the actual needs of grasping objects. This allows for the efficient use of finger closing information in dexterous hands. The calculation formula can be:

[0073]

[0074] in, The damping coefficient is... The calculation formula is:

[0075]

[0076] in, This is the basic damping factor, which can be set based on practical experience; It is the material resistance coefficient, which can be obtained directly; It is the expected fingertip contact force information of a dexterous hand, which can be set based on practical experience; It is skillful hands Real-time fingertip contact force information can be obtained through fingertip tactile sensors. By generating key points in stages and combining optimized interpolation with physical constraints, the trajectory shape can be dynamically adjusted according to the actual shape and contact state of the target object. This solves the problem of uneven motion trajectory and easy collision in complex grasping tasks. Furthermore, by dynamically adjusting finger closure information, it can avoid the grasping instability caused by the dexterous hand closing too fast or too slow.

[0077] Specifically, due to the physical limitations of the robotic arm's movement (such as joint speed and torque), constraint optimization is required when obtaining trajectory transition points to ensure that the motion trajectory is executable. For example, the joint speed of the robotic arm is constrained, the angular velocity of each joint is limited to avoid exceeding the maximum safety value, and a collision detection algorithm is used to detect whether the motion trajectory will cause the robotic arm to collide with the environment. If a collision occurs, the trajectory points are readjusted.

[0078] In one possible implementation, generating the second motion trajectory points of the dexterous hand based on the initial position information of the target object includes: acquiring point cloud data of the target object at the initial position based on the initial position information of the target object; and generating the second motion trajectory points of the dexterous hand based on a preset shape matching algorithm and the point cloud data.

[0079] Specifically, point cloud data refers to the collection of three-dimensional coordinate points on the surface of a target object, acquired through 3D scanning or depth sensors, used to accurately describe the object's geometry. Shape matching algorithms are mathematical methods that determine the optimal gripping position through geometric feature comparison. Specifically, they can be implemented using iterative nearest-point algorithms or feature descriptor matching algorithms. By calculating the geometric fit between the dexterous hand's gripping surface and the object's surface, a stable gripping contact area is determined.

[0080] Specifically, after point cloud data acquisition, a 3D surface model of the target object is generated. The shape matching algorithm compares the pre-set grasping posture of the dexterous hand with the geometric features of the object's surface. First, the curvature features, normal direction, and concave and convex areas in the point cloud data are extracted. Then, the contact area, contact force distribution, and posture matching degree between the dexterous fingertip contact surface and the object surface are calculated. Candidate areas that meet the grasping stability threshold are selected. The optimal grasping area can be calculated based on the point cloud data of the target object. The angle between the finger and the object surface is ensured to be appropriate through normal vector calculation to prevent grasping failure.

[0081] S220: Based on several motion trajectory points, obtain a multimodal dataset of the dexterous hand moving in the motion trajectory. The multimodal dataset includes environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand to move in the motion trajectory.

[0082] Specifically, it can record and store data on the movement of a dexterous hand along its trajectory to obtain a multimodal dataset. Data can then be filtered from the first multimodal data, environmental image data, and second multimodal data, removing data with particularly large errors.

[0083] S230: Perform multimodal feature fusion on environmental image data, first multimodal data, and second multimodal data to obtain the input sequence.

[0084] Specifically, feature regions can be extracted from environmental image data. For example, using EfficientNet-Lite0 as a feature extractor can convert the original RGB image into a low-dimensional feature representation, thereby reducing the amount of computation and extracting effective information for fusion with other sensor data.

[0085] In one possible implementation, the first multimodal data includes information on the first joint of the dexterous hand and information on the first fingertip contact force; the second multimodal data includes information on the second joint of the robotic arm; and the first joint information includes information on the first finger joint. Multimodal data fusion is performed on the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence, including: feature extraction from the first joint information, the first fingertip contact force information, the second joint information, and the environmental image data to obtain multiple sets of feature tensors. These multiple sets of feature tensors include a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environmental image data. The first feature tensor includes the first finger joint feature tensor. The multiple sets of feature tensors belonging to consecutive time steps are then fused to obtain the input sequence.

[0086] Specifically, the first joint information refers to the joint angles, joint velocities, and joint torque data of each joint of the dexterous hand. The first fingertip contact force information refers to the three-dimensional force vector generated when the dexterous fingertip contacts an object, which can be measured using a flexible tactile sensor array. The second joint information refers to the joint angles, joint velocities, and joint torque data of each driven joint of the robotic arm, used to represent the overall motion state of the robotic arm. Features can be extracted from these data separately to obtain the corresponding feature tensors. When the feature tensors are stacked along the time dimension, a time alignment strategy is used to ensure the correspondence between the modal data at the same time step. For example, in three consecutive time steps, the feature tensors of each modality are extracted and then concatenated along the time axis to form an input sequence containing historical state information. The input sequence can be a three-dimensional data structure formed by stacking the feature tensors of consecutive time steps along the time dimension.

[0087] S240: Divide the input sequence into several input subsequences, which correspond to different movement stages of the dexterous hand.

[0088] Specifically, an input subsequence refers to a local sequence of input data in continuous time steps divided into motion stages. Multiple sub-blocks are formed by grouping the input data by time steps. For example, the motion stage can be divided into the movement stage, the grasping stage, and the grasping-after-movement stage. Each stage can correspond to several input subsequences.

[0089] S250: Based on the motion phase constraints, select the target feature tensor from the input subsequence and generate a set of target feature tensors.

[0090] Specifically, the motion stage constraints can be constraints set according to different motion stages, and the target feature tensor set refers to the combination of modal features activated according to different motion stages, ensuring that the selected features are relevant to the current motion stage.

[0091] See Figures 6-8 In one possible implementation, the fourth feature tensor includes a top image feature tensor, a bottom image feature tensor, and a wrist image feature tensor. Based on motion phase constraints, a target feature tensor is selected from the input subsequence, and a target feature tensor set is generated. This includes: determining the motion phase of the dexterous hand based on its motion trajectory points, where the motion phase includes a movement phase, a grasping phase, and a post-grasping movement phase; when the input subsequence corresponds to the movement phase, selecting the first feature tensor, the top image feature tensor, the bottom image feature tensor, and the third feature tensor from the input subsequence as... Target feature tensors are generated, and a set of target feature tensors is generated; and / or, when the input subsequence corresponds to the grasping stage, a first feature tensor, a bottom image feature tensor, a wrist image feature tensor, a second feature tensor, and a third feature tensor are selected from the input subsequence as target feature tensors, and a set of target feature tensors is generated; and / or, when the input subsequence corresponds to the moving stage after grasping, a first feature tensor, a top image feature tensor, a wrist image feature tensor, a second feature vector, and a third feature tensor are selected from the input subsequence as target feature tensors, and a set of target feature tensors is generated.

[0092] Specifically, the top image feature tensor refers to the data representation of the global view image of the environment collected by the visual sensor at the top of the platform after feature extraction; the bottom image feature tensor refers to the object bottom surface feature data collected by the visual sensor at the bottom of the platform; and the wrist image feature tensor refers to the data of the close-range operation view data collected by the visual sensor installed on the dexterous wrist after feature extraction. By dividing the movement stages into regions, the system can focus on the multimodal data activated in the current stage when processing complex action sequences, avoiding interference from irrelevant features on attention calculation, reducing the computational complexity of the attention mechanism, and ensuring effective interaction between key modalities.

[0093] Specifically, the movement phase is the process of the dexterous hand approaching the target object, the grasping phase corresponds to the process of the fingers closing to form a stable contact, and the post-grasping movement phase corresponds to the process of the target object being lifted and transferred to the target position. Before the dexterous hand moves, the movement phase corresponding to the trajectory points can be preset. For example, trajectory points 1-200 are used as the movement phase, trajectory points 201-400 are used as the grasping phase, and trajectory points 401-600 are used as the post-grasping movement phase.

[0094] S260: Based on a sparse self-attention mechanism and a preset modality locking strategy, feature learning is performed on the target feature tensor set to obtain a predicted motion sequence. The predicted motion sequence is used to represent the multimodal prediction data of the dexterous hand and the robotic arm at the motion trajectory points. The preset modality strategy is used to control the second feature tensor to interact only with the first finger joint feature tensor.

[0095] Specifically, attention calculations can be performed on the target feature tensor through sparse self-attention mechanisms, which can reduce the amount of computation. Modality locking strategies can be used to constrain the interaction relationships between specific modalities. For example, tactile features can be forced to interact only with finger joint features. The sparse self-attention mechanism performs attention calculations only on the activated modal features at each stage and restricts the interaction path between tactile features and finger joint features through preset modality locking strategies.

[0096] Specifically, the input sequence can be:

[0097]

[0098] Wherein, each in the input sequence It can belong to one of the following modes:

[0099] The first feature tensor corresponding to the first joint information and the third feature tensor corresponding to the second joint information: The second feature tensor corresponding to the first fingertip contact force information The fourth feature tensor corresponding to environmental image data , , The first finger joint feature tensor corresponding to the finger joint information The trajectory of motion can be divided into multiple phases. Each motion stage corresponds to a different set of target feature tensors, which in turn corresponds to a different set of activation modes:

[0100]

[0101] In different stages of movement In this process, only the feature tensors in the active state are selected to form the target feature tensor set. :

[0102]

[0103] Then, attention is calculated for each target feature tensor set according to different motion stages:

[0104]

[0105] in, , , From The mapped Query, Key, and Value This is the dimension normalization factor.

[0106] Specifically, the feature tensors in different motion phases and activation states can be:

[0107] Movement phase : crawling phase : After grabbing, move phase : Based on this motion constraint, different sets of target feature tensors can be obtained.

[0108] Specifically, a preset modality locking strategy can be adopted to make the second feature tensor corresponding to the first fingertip contact force information... The first finger joint feature tensor corresponding to finger joint information Maintaining attentional connections throughout and not engaging in modal interactions with other feature tensors can be represented as:

[0109]

[0110]

[0111] S270: Based on the loss function, the predicted motion sequence, and the input sequence, the weight parameters of the self-attention model to be trained are adjusted to obtain the motion trajectory prediction model.

[0112] Specifically, during training, model parameters can be optimized based on action prediction loss and temporal smoothing loss to predict action sequences. These predicted action sequences can be the joint information of the robotic arm and the fingertip contact force information of the dexterous hand. The predicted action sequences can be a set of action states, and the combined loss function can be:

[0113]

[0114]

[0115]

[0116] in, It is the loss from action prediction; It is the action predicted by the model, which can include joint information of the dexterous hand and the robotic arm, as well as the contact force information of the fingertips of the dexterous hand; It can be the multimodal data corresponding to the robotic arm and dexterous hand at each trajectory point; These are weight parameters set based on practical experience; It is a time-series smoothing loss.

[0117] In one possible implementation, after acquiring the motion trajectory prediction model, the model can be validated. For example, it can be tested to verify the generalization ability and grasping success rate in different scenarios. Simulation testing can be used to set up a dexterous hand and a grasped object with different initial poses in a simulation environment, testing the model's performance under different grasping tasks. Evaluation metrics include: Grasping success rate: calculating whether the robotic arm and dexterous hand can stably grasp the target object and complete the task; Grasping stability: detecting whether the object slips or falls during the grasping process; Execution efficiency: analyzing the smoothness of the action sequence to avoid unnecessary jitter or adjustments, thereby judging the prediction effect of the motion trajectory prediction model.

[0118] This application discloses a method for generating a multimodal motion trajectory prediction model for a dexterous hand, used for simulating the grasping process of a dexterous hand. The method includes: generating the motion trajectory of the dexterous hand based on its initial position information and the final position information of the target object; the motion trajectory includes several motion trajectory points; based on these several motion trajectory points, acquiring a multimodal dataset of the dexterous hand's motion within the motion trajectory, the multimodal dataset including environmental image data, first multimodal data corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand's motion within the motion trajectory; fusing multimodal features of the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and dividing the input sequence into several input sub-sequences. Corresponding to different movement stages of the dexterous hand; based on the constraints of the movement stage, target feature tensors are selected from the input sub-sequences, and a set of target feature tensors is generated; based on a sparse self-attention mechanism and a preset modality locking strategy, feature learning is performed on the set of target feature tensors to obtain a predicted motion sequence. The predicted motion sequence is used to represent the multimodal prediction data of the dexterous hand and the robotic arm at the motion trajectory points. The preset modality strategy is used to control the second feature tensor to interact only with the first finger joint feature tensor; based on the loss function, the predicted motion sequence, and the input sequence, the weight parameters of the self-attention model to be trained are adjusted to obtain the motion trajectory prediction model. Through the fusion of multimodal data, the real-time performance and accuracy of dexterous hand motion trajectory prediction are improved, thereby increasing the grasping success rate of the dexterous hand.

[0119] This application also provides a motion trajectory prediction method for dexterous hand simulation grasping. The motion trajectory prediction method includes: acquiring first initial modal data of the dexterous hand at the initial position, second initial modal data of the robotic arm at the initial position, and initial environmental image data; inputting the first initial modal data, second initial modal data, and initial environmental image data into a motion trajectory prediction model to obtain a predicted motion trajectory sequence of the dexterous hand, the predicted motion trajectory sequence including predicted multimodal data of the dexterous hand and predicted multimodal data of the robotic arm.

[0120] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a motion trajectory prediction model generation device, electronic device, and corresponding embodiments.

[0121] Figure 3 This is a schematic diagram of the structure of the dexterous hand multimodal motion trajectory prediction model generation device shown in the embodiments of this application.

[0122] See Figure 3 A dexterous hand multimodal motion trajectory prediction model generation device 300, used for simulating the grasping process of a dexterous hand, includes:

[0123] The motion trajectory generation module 310 is used to generate the motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object. The motion trajectory includes several motion trajectory points.

[0124] In one possible implementation, the motion trajectory generation module 310 is further configured to generate a first motion trajectory point, a second motion trajectory point, and a third motion trajectory point of the dexterous hand based on the initial position information of the dexterous hand, the initial position information of the target object, and the end position information; acquire motion information of the dexterous hand at the first motion trajectory point, the second motion trajectory point, and the third motion trajectory point, the motion information including the position information, posture information, and finger closure information of the dexterous hand; generate a plurality of first trajectory transition points between the first motion trajectory point and the second motion trajectory point, and a plurality of second trajectory transition points between the second motion trajectory point and the third motion trajectory point, based on a preset optimized interpolation algorithm and the motion information; and determine the motion trajectory of the dexterous hand based on the first motion trajectory point, the first trajectory transition point, the second motion trajectory point, the second trajectory transition point, and the third motion trajectory point.

[0125] In one possible implementation, the motion trajectory generation module 310 is further configured to acquire point cloud data of the target object at its initial position based on the initial position information of the target object; and generate a second motion trajectory point of the dexterous hand based on a preset shape matching algorithm and the point cloud data.

[0126] The dataset acquisition module 320 is used to acquire a multimodal dataset of the dexterous hand moving in the motion trajectory based on several motion trajectory points. The multimodal dataset includes environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand to move in the motion trajectory.

[0127] The fusion module 330 is used to perform multimodal feature fusion on environmental image data, first multimodal data, and second multimodal data to obtain an input sequence.

[0128] The training module 340 is used to train the self-attention model to obtain the motion trajectory prediction model by inputting the input sequence into the model to be trained based on the preset sparse self-attention mechanism.

[0129] In one possible implementation, the training module 340 is further configured to extract features from the first joint information, the first fingertip contact force information, the second joint information, and the environmental image data to obtain multiple sets of feature tensors. The multiple sets of feature tensors include a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environmental image data. The first feature tensor includes the first finger joint feature tensor. The multiple sets of feature tensors belonging to continuous time steps are fused to obtain the input sequence.

[0130] In one possible implementation, the training module 340 is further configured to divide the input sequence into several input sub-sequences, each sub-sequence corresponding to a different movement stage of the dexterous hand; based on movement stage constraints, select target feature tensors from the input sub-sequences and generate a set of target feature tensors; perform feature learning on the set of target feature tensors based on a sparse self-attention mechanism and a preset modality locking strategy to obtain a predicted motion sequence, which is used to characterize the multimodal prediction data of the dexterous hand and the robotic arm at the motion trajectory points; the preset modality strategy is used to control the second feature tensor to interact only with the first finger joint feature tensor; and adjust the weight parameters of the self-attention model to be trained based on the loss function, the predicted motion sequence, and the input sequence to obtain a motion trajectory prediction model.

[0131] In one possible implementation, the training module 340 is further configured to determine the movement phase of the dexterous hand based on the movement trajectory points of the dexterous hand, the movement phase including a movement phase, a grasping phase, and a post-grasping movement phase; when the input subsequence corresponds to the movement phase, select a first feature tensor, a top image feature tensor, a bottom image feature tensor, and a third feature tensor from the input subsequence as target feature tensors, and generate a target feature tensor set; and / or, when the input subsequence corresponds to the grasping phase, select a first feature tensor, a bottom image feature tensor, a wrist image feature tensor, a second feature tensor, and a third feature tensor from the input subsequence as target feature tensors, and generate a target feature tensor set; and / or, when the input subsequence corresponds to the post-grasping movement phase, select a first feature tensor, a top image feature tensor, a wrist image feature tensor, a second feature vector, and a third feature tensor from the input subsequence as target feature tensors, and generate a target feature tensor set.

[0132] This application discloses a dexterous hand multimodal motion trajectory prediction model generation device for simulating the grasping process of a dexterous hand. The motion trajectory prediction model generation method includes: generating the motion trajectory of the dexterous hand based on the initial position information of the dexterous hand and the final position information of the target object, the motion trajectory including several motion trajectory points; acquiring a multimodal dataset of the dexterous hand's movement in the motion trajectory based on the several motion trajectory points, the multimodal dataset including environmental image data, first multimodal data of the dexterous hand corresponding to each motion trajectory point, and second multimodal data of the robotic arm controlling the dexterous hand's movement in the motion trajectory; fusing multimodal features of the environmental image data, the first multimodal data, and the second multimodal data to obtain an input sequence; and inputting the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model. This device can improve the real-time performance and accuracy of dexterous hand motion trajectory prediction through the fusion of multimodal data, thereby increasing the grasping success rate of the dexterous hand.

[0133] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0134] This application also provides an electronic device. Figure 4 This is a schematic diagram of the hardware structure of an embodiment of the electronic device of this application. The electronic device includes a memory 410 and at least one processor 420. The memory 410 is electrically connected to the at least one processor 420. The memory 410 stores instructions. The at least one processor 420 calls the instructions in the memory 410 to cause the electronic device to execute the dexterous hand multimodal motion trajectory prediction model generation method according to any of the foregoing embodiments of this application.

[0135] Specifically, the processor 420 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0136] Memory 410 may include a large-capacity memory 410 for data or instructions. For example, and not limitingly, memory 410 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 410 may include removable or non-removable (or fixed) media. Where appropriate, memory 410 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 410 is non-volatile solid-state memory. In a particular embodiment, memory 410 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0137] In one example, the control device may also include a communication interface 430 and a bus 440. The processor 420, memory 410, and communication interface 430 are connected via the bus 440 and communicate with each other.

[0138] The communication interface 430 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0139] Bus 440 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a Memory 410 bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 440 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0140] Furthermore, in conjunction with the dexterous hand multimodal motion trajectory prediction model generation method in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores instructions that, when executed by a processor, implement any of the dexterous hand multimodal motion trajectory prediction model generation methods in the above embodiments.

[0141] This application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0142] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0143] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0144] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for generating a multimodal motion trajectory prediction model for a dexterous hand, used to simulate the grasping process of a dexterous hand, characterized in that... The motion trajectory prediction model generation method comprises: According to the initial position information of the dexterous hand and the end position information of the target object, the motion trajectory of the dexterous hand is generated, and the motion trajectory comprises a plurality of motion trajectory points; Based on a plurality of the motion trajectory points, a multi-modal data set of the dexterous hand moving in the motion trajectory is obtained, the multi-modal data set comprising environmental image data, first multi-modal data of the dexterous hand at each motion trajectory point, and second multi-modal data of a robot arm controlling the dexterous hand to move in the motion trajectory; Multi-modal feature fusion is performed on the environmental image data, the first multi-modal data, and the second multi-modal data to obtain an input sequence; the first multi-modal data comprises first joint information of the dexterous hand and first fingertip contact force information, and the second multi-modal data comprises second joint information of the robot arm; the first joint information comprises first finger joint information; wherein, feature extraction is performed on the first joint information, the first fingertip contact force information, the second joint information, and the environmental image data to obtain a plurality of feature tensors, the plurality of feature tensors comprising a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environmental image data; the first feature tensor comprises a first finger joint feature tensor; the plurality of feature tensors belonging to consecutive time steps are fused to obtain the input sequence; The input sequence is input into a self-attention model to be trained based on a preset sparse self-attention mechanism for training to obtain a motion trajectory prediction model; wherein, the input sequence is divided into a plurality of input subsequences, the input subsequences corresponding to different motion stages of the dexterous hand; target feature tensors are selected from the input subsequences based on a motion stage constraint condition, and a target feature tensor set is generated; feature learning is performed on the target feature tensor set based on the sparse self-attention mechanism and a preset modal locking strategy to obtain a predicted motion sequence, the predicted motion sequence being used to represent multi-modal prediction data of the dexterous hand and the robot arm at the motion trajectory points, the preset modal locking strategy being used to control the second feature tensor to interact only with the first finger joint feature tensor; the weight parameters of the self-attention model to be trained are adjusted based on a loss function, the predicted motion sequence, and the input sequence to obtain the motion trajectory prediction model.

2. The method of claim 1, wherein, The motion trajectory prediction model generation method comprises: According to the initial position information of the dexterous hand and the end position information of the target object, the motion trajectory of the dexterous hand is generated, and the motion trajectory comprises a plurality of motion trajectory points; According to the initial position information of the dexterous hand, the initial position information and the end position information of the target object, first, second, and third motion trajectory points of the dexterous hand are respectively generated; Obtain motion information of the dexterous hand at the first motion trajectory point, the second motion trajectory point and the third motion trajectory point respectively, wherein the motion information comprises position information, attitude information and finger closing information of the dexterous hand; According to a preset optimization interpolation algorithm and the motion information, generate a plurality of first trajectory transition points between the first motion trajectory point and the second motion trajectory point, and a plurality of second trajectory transition points between the second motion trajectory point and the third motion trajectory point; According to the first motion trajectory point, the first trajectory transition point, the second motion trajectory point, the second trajectory transition point and the third motion trajectory point, determine the motion trajectory of the dexterous hand.

3. The method of claim 2, wherein, The second motion trajectory point of the dexterous hand is generated according to the initial position information of the target object, comprising: According to the initial position information of the target object, obtain point cloud data of the target object at the initial position; According to a preset shape matching algorithm and the point cloud data, generate the second motion trajectory point of the dexterous hand.

4. The method of claim 1, wherein, The fourth feature tensor comprises a top image feature tensor, a bottom image feature tensor and a wrist image feature tensor, and the target feature tensor set is generated by selecting the target feature tensor from the input subsequence based on the motion stage constraint condition, comprising: According to the motion trajectory point of the dexterous hand, determine the motion stage of the dexterous hand, wherein the motion stage comprises a moving stage, a grasping stage and a post-grasping moving stage; When the input subsequence corresponds to the moving stage, the first feature tensor, the top image feature tensor, the bottom image feature tensor and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated; And / or, when the input subsequence corresponds to the grasping stage, the first feature tensor, the bottom image feature tensor, the wrist image feature tensor, the second feature tensor and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated; And / or, when the input subsequence corresponds to the post-grasping moving stage, the first feature tensor, the top image feature tensor, the wrist image feature tensor, the second feature tensor and the third feature tensor are selected from the input subsequence as the target feature tensor, and the target feature tensor set is generated.

5. A dexterous hand multi-modal motion trajectory prediction model generation apparatus for dexterous hand simulation of a grasping process, characterized in that, The motion trajectory prediction model generation device comprises: A motion trajectory generation module is configured to generate a motion trajectory of a dexterous hand according to initial position information of the dexterous hand and end position information of a target object, wherein the motion trajectory comprises a plurality of motion trajectory points; A data set acquisition module is configured to acquire a multi-modal data set of the dexterous hand moving in the motion trajectory based on the plurality of motion trajectory points, wherein the multi-modal data set comprises environmental image data, first multi-modal data of the dexterous hand at each motion trajectory point, and second multi-modal data of a robot arm controlling the dexterous hand to move in the motion trajectory; a fusion module configured to perform multi-modal feature fusion on the environment image data, the first multi-modal data, and the second multi-modal data to obtain an input sequence, wherein the first multi-modal data comprises first joint information of the dexterous hand and first fingertip contact force information, the second multi-modal data comprises second joint information of the robot arm, and the first joint information comprises first finger joint information; wherein the fusion module is configured to: perform feature extraction on the first joint information, the first fingertip contact force information, the second joint information, and the environment image data to obtain a plurality of groups of feature tensors, wherein the plurality of groups of feature tensors comprise a first feature tensor corresponding to the first joint information, a second feature tensor corresponding to the first fingertip contact force information, a third feature tensor corresponding to the second joint information, and a fourth feature tensor corresponding to the environment image data, and the first feature tensor comprises a first finger joint feature tensor; and fuse a plurality of groups of feature tensors belonging to consecutive time steps to obtain the input sequence; a training module configured to input the input sequence into a self-attention model to be trained based on a preset sparse self-attention mechanism to obtain a motion trajectory prediction model, wherein the training module is configured to: divide the input sequence into a plurality of input subsequences, wherein the plurality of input subsequences correspond to different motion stages of the dexterous hand; select target feature tensors from the plurality of input subsequences based on a motion stage constraint condition to generate a target feature tensor set; perform feature learning on the target feature tensor set based on the sparse self-attention mechanism and a preset modality locking strategy to obtain a predicted motion sequence, wherein the predicted motion sequence is used to represent multi-modal prediction data of the dexterous hand and the robot arm at the motion trajectory point, and the preset modality locking strategy is used to control the second feature tensor to interact only with the first finger joint feature tensor; and adjust weight parameters of the self-attention model to be trained based on a loss function, the predicted motion sequence, and the input sequence to obtain the motion trajectory prediction model.

6. An electronic device, comprising: comprise: a processor; and a memory having executable code stored thereon that, when executed by the processor, causes the processor to perform the method of any one of claims 1-4. a memory having executable code stored thereon that, when executed by the processor, causes the processor to perform the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: ​

Images