A robot motion generation method and system

Abstract

The present disclosure provides a robot action generation method and system, wherein the method comprises: acquiring action reference data containing human action, and extracting the human action contained in the action reference data into corresponding textual description segments through a preset large model; synthesizing action segments representing the movement of corresponding human body parts in space according to the text information contained in the description segments; performing trajectory synthesis on multiple action segments according to a preset matching algorithm to obtain multiple action trajectories composed of action segments; and converting the action trajectories into movement trajectories for guiding the target robot to perform actions according to the mapping relationship between the corresponding human body parts and the robot skeletal structure. The present disclosure has low difficulty and cost in collecting action reference data; avoids the reinforcement learning training and parameter adjustment process, reduces the computing power and time cost, and improves the efficiency; and follows the mapping relationship of the robot skeleton, improving the fluency and usability of the movement trajectory.

Description

Technical Field

[0001] This disclosure relates to the field of embodied intelligence, and more particularly to a method and system for generating robot actions. Background Technology

[0002] The commonly used methods for robot trajectory generation and control in the industry today mainly include two approaches: imitation learning and teleoperation.

[0003] One approach, imitation learning, involves collecting human demonstration data and combining it with reinforcement learning to train a model in a simulation environment to generate control strategies. However, this approach has the following drawbacks: First, data collection is costly and cumbersome, and high-difficulty actions require highly skilled motion capture personnel, making it difficult to popularize. Second, model training demands significant computing power and has a long cycle, with uncertainties requiring repeated parameter adjustments, resulting in high time and financial costs and failing to meet the needs for rapid, low-cost development.

[0004] Teleoperation schemes rely on manually conceived trajectories, which are then recorded and guided by specific equipment. These primarily include isomorphic arm teleoperation (controlled via mapping through a miniature robotic arm) and VR teleoperation (recording position and calculating posture using VR devices). However, this approach also has significant shortcomings: firstly, isomorphic arm schemes struggle to coordinate the large degrees of freedom of humanoid robots (e.g., more than 40), leading to poor synchronization and difficulty in achieving complex movements due to multi-person collaborative control; secondly, VR schemes can only acquire positional information, resulting in poor calculation quality and consequently, poor trajectory smoothness and usability.

[0005] In summary, the relevant technologies have shortcomings in terms of data acquisition, training costs, and the quality of generated motion trajectories, making it difficult to meet the needs of humanoid robots for efficient, low-cost, and high-quality motion trajectory generation. Summary of the Invention

[0006] This disclosure provides a method and system for generating robot motions to address the problems of inefficiency and high cost in related technologies.

[0007] In view of the above problems, firstly, embodiments of this disclosure provide a method for generating robot actions, including: Obtain motion reference data containing human movements, and extract the human movements contained in the motion reference data into corresponding textual description fragments using a preset large model; Based on the text information contained in the description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized; Multiple action segments are combined into a trajectory using a preset matching algorithm to obtain multiple action trajectories composed of action segments; Based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure, the motion trajectory is converted into a motion trajectory that guides the target robot to perform actions.

[0008] In conjunction with the first aspect, in one possible implementation, based on the text information contained in the descriptive fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized, including: For each description segment, semantic features of the text information contained in the description segment are extracted, and based on the semantic features, a first preset number of action segments corresponding to the description segment are generated; wherein, each action segment contains continuous pose data of the corresponding human body part in space; Action segments are filtered based on continuous pose data to determine those that meet the physical constraints of the corresponding human body parts.

[0009] In conjunction with the first aspect, in one possible implementation, multiple action segments are synthesized into trajectories according to a preset matching algorithm to obtain multiple action trajectories composed of action segments, including: Based on the temporal order of the action segments in the corresponding human body parts, multiple action segments are divided into segment sets, such that each segment set contains at least one candidate action segment with the same temporal order. Starting from the initial action segment determined by the timing sequence, for the current action segment that needs to be matched, based on the timing sequence corresponding to the current action segment, the next action segment is matched from the candidate action segments contained in the segment set corresponding to the next timing sequence, and the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0010] In conjunction with the first aspect, in one possible implementation, for the current action segment requiring a matching operation, based on the timing sequence corresponding to the current action segment, the next action segment is matched from the candidate action segments contained in the set of segments corresponding to the next timing sequence, and trajectory synthesis is performed between the next action segment and the current action segment until the synthesis of the corresponding action trajectory is completed, including: For the current action segment that requires matching, determine the first pose data of the corresponding human body part at the preset length of the end of the current action segment; Based on the timing of the current action segment, determine the candidate action segments included in the set of segments corresponding to the next timing segment; Determine the second pose data of the corresponding human body part with the preset length of the beginning of each candidate action segment; The first pose data and the second pose data are matched. The candidate action segment corresponding to the second pose data that meets the preset matching conditions is determined as the next action segment of the current action segment, and the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0011] In conjunction with the first aspect, in one possible implementation, based on the mapping relationship between the corresponding human body part and the robot's skeletal structure, the motion trajectory is converted into a motion trajectory to guide the target robot to perform actions, including: The corresponding human body parts are mapped to the corresponding robot skeleton structure, so as to convert the pose data of the corresponding human body parts contained in the motion trajectory into the pose information of the robot skeleton structure. Based on the pose information of the robot's skeletal structure, the motion path of the robot's skeletal structure is planned in chronological order to obtain the initial motion trajectory of the target robot. The smoothness of the initial motion trajectory is determined based on the physical constraints of the robot's skeletal structure. If the smoothness judgment result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory.

[0012] In conjunction with the first aspect, in one possible implementation, determining the smoothness of the initial motion trajectory based on the physical constraints of the robot's skeletal structure includes: Based on the pose information corresponding to the trajectory points in the initial motion trajectory, calculate the motion parameters of the target nodes in the robot's skeletal structure; The motion parameters of the target node are compared with the physical constraint thresholds set for the target node; If the motion parameters are higher than the physical constraint threshold, the initial motion trajectory is determined to be unsmooth. If the motion parameters are not higher than the physical constraint threshold, the initial motion trajectory is determined to be smooth.

[0013] In conjunction with the first aspect, in one possible implementation, if the smoothness determination result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory, including: When the initial motion trajectory is not smooth, a quadratic programming algorithm is used to interpolate data between the trajectory points of the initial motion trajectory to obtain a smoothed target motion trajectory. The method further includes: If the initial motion trajectory is smooth, the initial motion trajectory is output as the target motion trajectory.

[0014] In conjunction with the first aspect, in one possible implementation, it further includes: The obtained motion trajectory is simulated and evaluated, and the motion trajectory is scored according to the degree of adaptation between the physical characteristics of the target robot and the motion trajectory during the simulation process, and the simulation reward score of each motion trajectory is obtained. The motion trajectory is compared with the corresponding action reference trajectory, and the motion trajectory is scored according to the degree of deviation between the motion trajectory and the action reference trajectory to obtain the degree of compliance score of each motion trajectory; the action reference trajectory includes at least one of the following: action reference data, textual description fragments, action fragments or motion trajectories; Based on the simulation reward score and the compliance score, the final trajectory score of each motion trajectory is calculated, and based on the final trajectory score, a second preset number of motion trajectories are selected from each motion trajectory as candidate motion trajectories to guide the target robot to perform actions.

[0015] In conjunction with the first aspect, in one possible implementation, the motion reference data includes at least one of the following: video data, image data, and text data; The human movements contained in the motion reference data are extracted into corresponding textual description fragments using a pre-defined large model, including: The video data is sampled to obtain continuous video segments with a preset frame rate; a preset large model is called to perform inference on the continuous video segments, and a textual description of human actions contained in the continuous video segments is output in a preset format. For image data, a preset large model is invoked to perform inference on the image data, and a textual description of the human body movements contained in the image data is output in a preset format. For text data, a preset large model is invoked to convert the text data into a textual description fragment in a preset format.

[0016] In conjunction with the first aspect, in one possible implementation, the description fragment includes at least: action duration and action description; The human movements contained in the motion reference data are extracted into corresponding textual description fragments using a pre-defined large model, including: For the same action reference data, the human actions contained in the action reference data are extracted into corresponding textual description fragments multiple times through a preset large model; The method further includes: The extracted descriptive fragments are filtered based on action duration and action description respectively to determine the target descriptive fragments that match the corresponding action reference data; Based on the text information contained in the description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized, including: Based on the text information contained in the target description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized.

[0017] A second aspect of this disclosure provides a robot motion generation system, comprising: The extraction module is used to acquire motion reference data containing human movements, and extract the human movements contained in the motion reference data into corresponding textual description fragments through a preset large model. The synthesis module is used to synthesize action segments representing the movement of corresponding human body parts in space based on the text information contained in the description fragment; The matching module is used to synthesize the trajectories of multiple action segments according to a preset matching algorithm, so as to obtain multiple action trajectories composed of action segments. The conversion module is used to convert the motion trajectory into a motion trajectory that guides the target robot to perform actions, based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure.

[0018] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a method and system for generating robot motion. The method includes: acquiring motion reference data containing human motions, and extracting the human motions contained in the motion reference data into corresponding textual description fragments using a preset large model; synthesizing motion fragments representing the spatial movement of corresponding human body parts based on the text information contained in the description fragments; synthesizing multiple motion fragments into trajectories using a preset matching algorithm to obtain multiple motion trajectories composed of motion fragments; and converting the motion trajectories into motion trajectories that guide the target robot to perform actions based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure. This disclosure generates motion trajectories based on motion reference data, which has lower data acquisition difficulty and cost. During the motion generation process, it is driven by a preset large model, avoiding long-term reinforcement learning training and parameter adjustment processes, reducing computing power and time costs, and improving efficiency. Furthermore, the motion generation process follows the mapping relationship between human body parts and the robot's skeletal structure, avoiding the problem of poor matching between the generated motion trajectory and the robot structure, and improving the smoothness and usability of the output motion trajectory. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a robot motion generation method provided in this embodiment of the disclosure; Figure 2 A schematic diagram of the matching process provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of a robot motion generation system provided in an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the robot motion generation system architecture provided in an embodiment of this disclosure. Detailed Implementation

[0020] This disclosure provides a method and system for generating robot actions. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0021] This disclosure provides a method for generating robot actions, such as... Figure 1 As shown, it can be implemented as follows: S101. Obtain motion reference data containing human movements, and extract the human movements contained in the motion reference data into corresponding textual description fragments through a preset large model. S102. Based on the text information contained in the description fragment, synthesize an action fragment representing the movement of the corresponding human body part in space; S103. Combine multiple action segments into a trajectory according to a preset matching algorithm to obtain multiple action trajectories composed of action segments; S104. Based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure, the motion trajectory is converted into a motion trajectory that guides the target robot to perform actions.

[0022] In this embodiment, the motion reference data can be a dataset characterizing the motion features of a humanoid creature. Specifically, this data mainly records features such as the limb outlines, spatial location, and posture changes of the target humanoid creature. The humanoid creature here can include, but is not limited to, humans, humanoid robots, or virtual digital humans—objects with humanoid forms. In one possible implementation, the motion reference data can be images containing human motion (such as RGB, Red, Green, Blue images, depth images, etc.), or it can be an image sequence or video stream composed of multiple frames containing continuous human motion, or it can be a textual description of human motion.

[0023] Furthermore, to obtain descriptions of human movements from the aforementioned action reference data, this action reference data can be input into a pre-defined large model. Based on the contextual understanding capabilities of the pre-defined large model, the action features of the target object (i.e., the aforementioned humanoid creature) in the action reference information can be identified, and then a textual description fragment containing the semantics of the action features can be output. This pre-defined large model can be implemented as a multi-modal model based on a neural network architecture. This model can support input from multiple modalities (e.g., text, images, audio, video, etc.) and perform cross-modal reasoning interaction to understand and output the input information. As an example, this pre-defined large model can be a similar model such as GPT-4o or Qwen3-Omni.

[0024] Within the pre-defined large model, different preprocessing operations can be performed based on the different modalities of the input motion reference data. For example, for motion reference data of single-frame images, it can be divided into multiple fixed-size patches, and then these patches can be converted into fixed-dimensional vectors by the visual encoder within the large model; for motion reference data of video composed of continuous image sequences, it is also necessary to perform processing on the time axis and time-stamp each image in the sequence.

[0025] Furthermore, based on the attention mechanism of a pre-defined large model, motion features in the preprocessed motion reference data can be captured and extracted. Through the language model within the pre-defined large model, the extracted motion features are semantically aligned, mapping these features to corresponding semantic concepts, thereby obtaining textual descriptions of human movements in the motion reference data. It should be noted that the motion reference data input to the large model can be single-modal data or a combination of multiple modalities. For example, it could be a combination of an image of human movement at a specific time point and a textual description of the human movement at that time point.

[0026] Furthermore, the format of the descriptive fragments output by the pre-defined large model can be restricted through cue word engineering. In one possible implementation, the descriptive fragments need to include not only motion description text based on human movements, but also the pre-defined large model's understanding of the original motion reference data. For example, the modality of the motion reference data, the duration of the movements in the motion reference data, and the original frame rate for video modalities. The motion description text can also be restricted by time-based output guide words in the cue word engineering, to attach corresponding time information (e.g., timestamp, duration, etc.) to the description of each individual movement output. The cue word engineering can also include JSON-structured output guide words to constrain the pre-defined large model to output descriptive fragments in JSON data format.

[0027] To ensure the accuracy of the description of motion reference data, a pre-defined large model can generate corresponding description fragments multiple times for the same motion reference data, and the description fragment that best reflects the characteristics of human motion in the motion reference data can be selected from these description fragments as the target description fragment.

[0028] Then, corresponding motion segments can be generated based on the selected description segments. These motion segments can be a set of motion states of parts of the human body (e.g., joints and bones) within the time period represented by the description segment. In one possible implementation, the motion segment can be implemented as pose data. This pose data can be obtained by 3D modeling software or synthesized through a 3D motion generation model.

[0029] The 3D motion generation model here can be a generative algorithm architecture based on deep learning. Its mechanism is to map data from modalities such as textual descriptions and visual sequences into continuous pose data representing the human body's motion state. As an example, the 3D motion generation model can be implemented as a Motion Diffusion Model, MotionGPT, or HY-Motion, or similar models.

[0030] In a 3D motion generation model, a human skeleton template can be set. Based on the architecture of the 3D motion generation model itself (e.g., a denoised diffusion probability model architecture or an autoregressive architecture based on discretized sequences), the relevant content in the description segment is converted into a set of data describing the motion state of each joint and bone in the human skeleton template within a certain action time. These data sets can be packaged into files of a uniform format (e.g., FBX or BVH format) for output. These files containing pose data are the motion segments.

[0031] It should be noted that the action fragments here can be representations of a time-controllable portion of a descriptive fragment. For example, multiple action fragments can be obtained from a single descriptive fragment, each covering a portion of the human movement within a certain time interval (e.g., 3 to 8 seconds). Performing multiple synthesis operations on the same descriptive fragment yields a large number of action fragments, upon which subsequent motion trajectory synthesis can be performed. Furthermore, validity checks can be performed on these action fragments to ensure that the synthesized pose data conforms to the constraints of real-world physical laws.

[0032] For these obtained action fragments, a preset matching algorithm can be executed in a certain order. This order can be based on the temporal sequence of the actions represented by the action fragments in the description fragment. The preset matching algorithm can be a greedy matching algorithm or a full matching algorithm; there is no restriction. Based on the preset matching algorithm, multiple complete continuous action trajectories can be combined from these action fragments.

[0033] These motion trajectories can then be converted into motion trajectories to guide the robot's movement. Since motion trajectories are synthesized based on human skeletal templates, and the number of joints, bone length ratios, and degrees of freedom of a real robot differ structurally from those of the human body, a General Motion Retargeting (GMR) model can be used to map between skeletons of different architectures to match them. This GMR model, through its internal mapping function, converts the kinematic features of human movements represented in the motion trajectories into executable joint angle sequences and displacement vectors, while satisfying the physical constraints of the target robot. This allows for the mapping of human body parts to the robot skeleton, converting motion trajectories into motion trajectories.

[0034] The physical constraints of the target robot can be defined using the Unified Robot Description Format (URDF) model. The URDF model can include definitions of the robot's rigid body components (e.g., arms, torso, and wheels) and the constraint relationships between different rigid body components. Specifically, the definition of rigid body components can include parameters used for kinematic calculations, such as mass, shape, and inertia tensor; the definition of constraint relationships can include the linkage relationships, relative poses, rotation axes, motion types, and motion limits between different rigid body components.

[0035] In summary, the pre-defined large model disclosed in this paper supports multimodal data input and has a wide range of data sources. Motion trajectories can be generated from various types of videos, images, and text, eliminating the need for motion capture personnel to provide professional data and thus obtaining complex motion trajectories, lowering the threshold and cost of data acquisition. This method can execute the trajectory generation process by calling existing models, avoiding the reinforcement learning training process in a simulation environment. It has lower computational requirements and avoids parameter tuning, reducing time and computational costs. The generation process is based on the mapping relationship between human body parts and the robot's skeletal structure, avoiding problems such as motion stuttering and distortion caused by mismatch between the human and robot, ensuring the smoothness and usability of the trajectory.

[0036] In another embodiment provided in this disclosure, step S102 above, "synthesizing an action segment representing the movement of the corresponding human body part in space based on the text information contained in the description segment," can be implemented as follows: S1021. For each description segment, extract the semantic features of the text information contained in the description segment, and generate a first preset number of action segments corresponding to the description segment based on the semantic features. Each action segment contains continuous pose data of the corresponding human body part in space; S1022. Based on continuous pose data, filter action segments to determine those that meet the physical constraints of the corresponding human body parts.

[0037] In this embodiment of the disclosure, the 3D motion generation model can extract text information describing human body movements from descriptive fragments and identify their specific semantic features through its internal semantic understanding capabilities. Using the decoding mechanism within the 3D motion generation model, motion fragments including continuous pose data of human body parts are synthesized based on the descriptive fragments.

[0038] Continuous pose data here refers to temporal data describing the motion state of various parts of the human body (e.g., bones and joints). This can include relative position data and relative rotation data between bones. Based on this data, the position and posture changes of human body parts in space over a period of time can be described, thereby obtaining the corresponding motion segments.

[0039] A 3D motion generation model can generate multiple motion fragments from a given description segment. These motion fragments can represent a portion of the information in the description segment. The resulting motion fragments can also undergo validity checks, the purpose of which is to remove motion fragments that do not conform to the physical constraints of human body parts.

[0040] Validity checks can be performed based on continuous pose data of different human body parts in a motion segment. By checking whether there are conflicts between body parts (e.g., different bones overlap and cross) or whether the relationship between body parts violates physiological characteristics or physical laws (e.g., joints are twisted to an angle that the human body cannot reach) when the body parts move according to the pose data, the validity of the motion segment can be checked, and then motion segments that cannot pass the above validity checks can be eliminated.

[0041] In another embodiment provided in this disclosure, the step S103 above, "synthesizing multiple action segments according to a preset matching algorithm to obtain multiple action trajectories composed of action segments", can be implemented as follows: S1031. Based on the temporal sequence of the action segments in the corresponding human body parts, divide multiple action segments into segments sets, such that each segment set contains at least one candidate action segment with the same temporal sequence. S1032. Starting from the initial action segment determined according to the timing sequence, for the current action segment that needs to perform the matching operation, according to the timing sequence corresponding to the current action segment, match the next action segment from the candidate action segments contained in the segment set corresponding to the next timing sequence, and perform trajectory synthesis with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0042] In this embodiment of the disclosure, the action segment can correspond to different stages of a complete action in the action reference data. Assuming the action reference data is a video of "taking a water cup and placing it on a coaster," the resulting description segment can include the correspondence between timestamps and decomposed action descriptions. For example: [00:00]-[00:03] The left arm leads the left hand to move towards the water cup; [00:03]-[00:05] The left hand reaches the target position, the fingers close, and grasp the water cup; [00:05]-[00:06] The left hand lifts, raising the water cup by 10 centimeters; [00:06]-[00:08] The left hand moves to the coaster; [00:08]-[00:10] The left hand descends until the water cup touches the coaster, the fingers release, and the left arm leads the left hand to retract.

[0043] Different action segments can represent the movements of human body parts corresponding to different timestamps in a descriptive segment. Based on the temporal sequence of the action segments in the descriptive segment, they can be grouped. Action segments representing the same temporal sequence are grouped into the same segment set as candidate action segments for that temporal sequence.

[0044] Action segments can be synthesized into trajectories using a matching algorithm to obtain an action trajectory. This matching algorithm can be a greedy matching algorithm. First, a candidate action segment is randomly selected from the set of segments with the earliest timing sequence as the starting action segment, thus determining the pose data at the end of this action segment. Based on this pose data, it can be compared with the average pose data of the preceding frames (e.g., the first 15 frames of the candidate action segment) in the next time-series segment set. The candidate action segment with the closest average pose data is then selected as the matched action segment, and this matched action segment is used as the current action segment. The matching process continues in the next time-series segment set until all time-series segment sets are matched. After matching is complete, all matched action segments can be connected in temporal order to obtain a complete action trajectory corresponding to the action reference data.

[0045] It should be noted that an action segment can also include representations for multiple consecutive timestamps. That is, an action segment can exist in multiple segment sets. The generated action segment can be divided into multiple segments according to the time sequence, and each segment is assigned to a different segment set.

[0046] In another embodiment provided in this disclosure, step S1032, "for the current action segment that needs to perform a matching operation, according to the timing sequence corresponding to the current action segment, match the next action segment from the candidate action segments contained in the segment set corresponding to the next timing sequence, and perform trajectory synthesis with the current action segment until the synthesis of the corresponding action trajectory is completed," can be implemented as follows: Step 1: For the current action segment that needs to be matched, determine the first pose data of the corresponding human body part at the preset length of the end of the current action segment; Step 2: Based on the timing sequence corresponding to the current action segment, determine the candidate action segments included in the segment set corresponding to the next timing sequence; Step 3: Determine the second pose data of the corresponding human body part with the preset length of the beginning of each candidate action segment; Step 4: Match the first pose data and the second pose data, determine the candidate action segment corresponding to the second pose data that meets the preset matching conditions as the next action segment of the current action segment, and perform trajectory synthesis with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0047] In this embodiment of the disclosure, for the current action segment that requires matching operation, the final pose data of the end of the action segment can be used as the first pose data.

[0048] Furthermore, the average pose data of multiple frames at the beginning of each candidate action segment in the next time sequence segment set is obtained (i.e., the second pose data). Based on preset matching conditions, the first pose data and the second pose data are matched. Here, the preset matching conditions can be the degree of deviation between the first pose data and the second pose data, and thus the candidate action segment containing the second pose data with the lowest degree of deviation can be selected as the next matched action segment.

[0049] like Figure 2The diagram illustrates a possible matching process. In the diagram, squares with the same pattern represent the same action segment. For example, A1 and B1, A3, B2 and C1, and A2 and B3 represent different time periods of the same action segment. Based on the action content represented by different time periods, they are divided into different segment sets. The arrows in the diagram represent matching relationships. It can be seen that squares originally belonging to the same action segment can be matched together because their pose data is the most coherent. The pose data at the end of B1 and B3 are closest to the average pose data of the first 15 frames of C2. Therefore, the action segments formed by A1, B1 and C2, and A2, B3 and C2 respectively, are matched together, forming two new action trajectories.

[0050] In another embodiment provided in this disclosure, the step S104 above, "converting the motion trajectory into a motion trajectory to guide the target robot to perform actions based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure," can be implemented as follows: S1041. Map the corresponding human body part to the corresponding robot skeleton structure to convert the pose data of the corresponding human body part contained in the motion trajectory into pose information of the robot skeleton structure. S1042. Based on the pose information of the robot skeleton structure, plan the motion path of the robot skeleton structure in chronological order to obtain the initial motion trajectory of the target robot. S1043. Determine the smoothness of the initial motion trajectory based on the physical constraints of the robot's skeletal structure; S1044. If the smoothness judgment result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory.

[0051] In this embodiment of the disclosure, each part of the human skeletal structure can be mapped and bound to the structure of the target robot using the GMR model, according to the definition of the robot skeletal structure in the URDF model.

[0052] Furthermore, the pose data representing various parts of the human body in the motion trajectory obtained based on human anatomy can be converted into pose information describing the positional and orientational relationships between various parts of the robot. According to the temporal sequence in the motion trajectory, the changes of this pose information over time can be integrated into the complete motion paths of each skeletal structure of the target robot. The set of all these motion paths of skeletal structures is the motion trajectory of the target robot, which represents the motion of the target robot.

[0053] However, these motion trajectories are still constructed based on the logic of human movement. There may be abrupt changes in motion between path points within these trajectories that are unsuitable for robot execution. Therefore, it is necessary to determine the smoothness of these generated initial motion trajectories. This smoothness determination process involves calculating the motion parameters of specific nodes of the robot (e.g., the robot's joints and bones) when executing the motion trajectory and comparing them with constraints based on the physical constraints of the robot's skeletal structure (e.g., the power and maximum speed of the robot's motors). The smoothness is determined based on the comparison results. If the motion parameters exceed the aforementioned physical constraints, it indicates that the initial motion trajectory is not smooth, and the execution of the initial motion trajectory needs to be corrected and replanned to ultimately obtain a target motion trajectory suitable for the target robot.

[0054] By determining the smoothness of motion as described above, we can avoid physical impacts on the target robot caused by sudden changes in motion, protect the robot's mechanical structure, improve the feasibility of motion trajectories, and enhance the robot's operational accuracy.

[0055] In another embodiment provided in this disclosure, step S1043 above, "determining the smoothness of the initial motion trajectory based on the physical constraints of the robot's skeletal structure," includes: Step 1: Calculate the motion parameters of the target nodes in the robot's skeletal structure based on the pose information corresponding to the trajectory points in the initial motion trajectory. Step 2: Compare the motion parameters of the target node with the physical constraint thresholds set for the target node; Step 3: If the motion parameters are higher than the physical constraint threshold, determine that the initial motion trajectory is not smooth; Step 4: If the motion parameters are not higher than the physical constraint threshold, determine that the initial motion trajectory is smooth.

[0056] In this embodiment of the disclosure, a trajectory point can refer to the pose information of a target node in the motion trajectory at a specific moment. The target node is a movable component (e.g., joints and bones) of the target robot's skeletal structure. By traversing all trajectory points in the motion trajectory and based on the pose changes therein, motion parameters such as the velocity and acceleration of the target node between each trajectory point can be calculated.

[0057] These motion parameters can then be compared with the physical constraint thresholds set for the corresponding target nodes. These physical constraint thresholds can be set based on the actuators driving these target nodes in the robot (for example, the maximum speed of the corresponding joint can be determined based on the running speed of the motors driving the robot joints). If the motion parameters between two trajectory points exceed the physical constraint thresholds, it indicates that the hardware resources of the target robot cannot support the smooth execution of the trajectory between the above trajectory points, the motion trajectory is not smooth, and corrections need to be made between trajectory points to adapt to the hardware resources of the target robot.

[0058] In another embodiment provided in this disclosure, step S1044 above, "when the smoothness judgment result indicates that the initial motion trajectory is not smooth, perform a corresponding trajectory correction strategy on the initial motion trajectory to obtain the target motion trajectory", can be implemented as follows: When the initial motion trajectory is not smooth, a quadratic programming algorithm is used to interpolate data between the trajectory points of the initial motion trajectory to obtain a smoothed target motion trajectory. The robot motion generation method also includes the following steps: If the initial motion trajectory is smooth, the initial motion trajectory is output as the target motion trajectory.

[0059] In this embodiment, the trajectory correction strategy can be a quadratic programming-based algorithm. First, the pose information of the beginning and end of the interval of unsmooth trajectory points is calculated into coordinates in a robot coordinate system centered on the robot using forward kinematics. The coordinates corresponding to the beginning and end of the interval are input into the quadratic programming solver as the initial and final states, respectively, and constraints (e.g., velocity or acceleration limits) are established. Then, a quadratic objective function is used to solve the problem, regenerating a series of coordinates and corresponding timing sequences between trajectory points according to a preset step size, and interpolating them between the two trajectory points. This results in a smoothed target motion trajectory.

[0060] In yet another embodiment provided in this disclosure, the robot motion generation method further includes the following steps: Step 1: Simulate and evaluate the obtained motion trajectory, and score the motion trajectory according to the degree of adaptation between the physical characteristics of the target robot and the motion trajectory during the simulation process, and obtain the simulation reward score for each motion trajectory. Step 2: Compare the motion trajectory with the corresponding action reference trajectory, and score the motion trajectory according to the degree of deviation between the motion trajectory and the action reference trajectory to obtain the degree of compliance score of each motion trajectory; the action reference trajectory includes at least one of the following: action reference data, textual description fragments, action fragments or action trajectories; Step 3: Based on the simulation reward score and the compliance score, calculate the final trajectory score for each motion trajectory, and based on the final trajectory score, select a second preset number of motion trajectories from each motion trajectory as candidate motion trajectories to guide the target robot to perform actions.

[0061] In this embodiment of the disclosure, for each motion trajectory, the motion trajectory can be executed based on a simulation model of the target robot using a physical simulation tool. In one possible implementation, the physical simulation tool can be implemented as Mujoco or other similar tools.

[0062] During the simulation, the simulation model of the target robot can execute the obtained motion trajectory and monitor the operating status of each component in the simulation model (e.g., the speed, acceleration, and power of each joint motor can be monitored). Based on the monitored operating status, each motion trajectory can be scored to evaluate the feasibility of the target robot executing the motion trajectory and the structural stress of the motion trajectory on the robot.

[0063] In one possible implementation, the scoring rules for the simulation reward score can include several baseline conditions. If certain conditions occur, it indicates that the motion trajectory is completely infeasible in practical applications, and the trajectory can be directly eliminated. For example, conditions that would damage the robot or the manipulated object, such as robot collisions, motor speeds exceeding the rated range, or end-effector acceleration or angular acceleration exceeding the range, can be set as baseline conditions in the scoring rules.

[0064] If none of the above situations occur, the motion trajectory can be scored 100 points directly. If the robot does not experience a high load during operation (for example, the speed of each motor is below 75% of its rated value throughout the entire process), an additional 50 points can be added.

[0065] The smoothness of the robot's movements can also be scored, with smoothness characterized by end-effector torque. During simulation, the robot's end-effector acceleration and angular acceleration can be monitored. Combined with information such as the robot's mass and center of mass defined in URDF, the end-effector torques of each actuator during movement can be calculated. The motion trajectories are then sorted according to the magnitude of their corresponding end-effector torques. The simulation reward score for the lowest quartile of motion trajectories in the end-effector torque ranking is increased by 25 points, while the simulation reward score for the highest quartile is decreased by 25 points. Finally, after the above simulation and scoring process, a simulation reward score can be obtained for each motion trajectory. It should be noted that the above scoring rules and score settings can be adjusted according to actual circumstances; this is for illustrative purposes only and is not a limitation.

[0066] The degree of adherence to human actions can be scored for each motion trajectory. This adherence assessment can be based on motion reference data, textual descriptions, motion segments, or motion trajectories. Specifically, the adherence score evaluation process assesses the differences between the motion trajectory and the inputs and outputs of its preceding steps. To quantify these differences, human and robot actions can be compared within the same coordinate system. For motion reference data, textual descriptions, and motion segments, the methods provided in this disclosure can be used to obtain the corresponding motion trajectories. Then, based on the pose information of the human body parts contained in the motion trajectories, these trajectories are transformed into the same coordinate system as the simulation model of the target robot. The coordinate values ​​of corresponding trajectory points in the motion trajectory and the motion trajectory are then compared, and the degree of adherence is scored based on the differences in the changes in coordinate values.

[0067] For example, if the difference in coordinate values ​​between the trajectory points of a robot's motion trajectory has the opposite sign to the difference in coordinate values ​​between the human body's motion trajectory in any coordinate axis direction (i.e., the direction of the robot's motion on a certain axis between trajectory points is opposite to the direction of the human body's motion), then points can be deducted from the score for the degree of adherence to the trajectory.

[0068] In one possible implementation, the initial score for compliance is 150 points. Five points are deducted for each instance of the opposite direction, with a maximum deduction of 50 points for each actuator's end effector. For a dual-arm robot, the maximum deduction for both arms is 100 points. It should be noted that the scoring rules and point values ​​described above can be adjusted according to actual circumstances; this is for illustrative purposes only and is not a limitation.

[0069] After scoring the compliance score and simulation reward score, a weighted fusion can be performed to calculate the final trajectory score. In one possible implementation, the weight values ​​for the compliance score and simulation reward score in the above weighted fusion process can each be 0.5. These weight values ​​can be adjusted according to the focus of the robot task, and are not limited here.

[0070] Based on the final trajectory scores, the motion trajectories can be ranked, and the one or more highest-scoring trajectories can be selected as candidate trajectories. These candidate trajectories can then be further manually screened and evaluated by professional technicians to select the most suitable trajectory for the target robot; alternatively, the applicable scenarios for each candidate trajectory can be determined, serving as the target robot's trajectory in different scenarios, thus improving the robot's scene adaptability; furthermore, the target robot can randomly select a trajectory during execution to increase operational flexibility.

[0071] In another embodiment provided in this disclosure, the step S101 above, "extracting the human movements contained in the action reference data into corresponding textual description fragments using a preset large model," can be implemented as follows: S1011. The video data is sampled to obtain continuous video segments with a preset frame rate; a preset large model is called to perform inference on the continuous video segments, and a textual description of human actions contained in the continuous video segments is output in a preset format. S1012. For the image data, call the preset large model to perform inference on the image data, and output the textual description fragment of the human body movements contained in the image data in a preset format. S1013. For text data, call a preset large model to convert the text data into a textual description fragment in a preset format.

[0072] In this embodiment, to facilitate the extraction of information from the motion reference data by the pre-defined large model, preprocessing is required before inputting the motion reference data into the large model. For example, the duration of the motion reference data of the input video data type can be controlled. As an example, the duration of the video data can be controlled to be around 10-20 seconds to control the total number of tokens in the input data. Downsampling processing can also be performed to extract frames from the video data, reduce the frame rate of the video data, and further control the number of tokens, so that the amount of data received by the pre-defined large model remains within its normal processing range. Furthermore, to ensure that the input video data still retains sufficient semantics, the frame rate of the video data can be 1 frame per second. Then, the video data with the adjusted frame rate can be input into the pre-defined large model, which extracts and organizes the content of the video data to generate structured descriptive segments of human motion processes.

[0073] Similarly, resolution control and other operations can be performed on image data, ultimately outputting a description of the human body's movements within the image data through a pre-set large model. Furthermore, the reasoning capabilities of the pre-set large model can be used to infer the preceding and subsequent actions in the image data, yielding a complete action flow.

[0074] For text data, it can be directly input into a large model, which will then format and output a structured descriptive fragment corresponding to the text data based on its semantics. For text data that does not provide a comprehensive description of the action process, the inference capabilities of the large model can be used to complete the entire human action process, allowing complex full-process actions to be obtained based solely on simple text input.

[0075] The data from different modalities describing the same action process can also be simultaneously input into a pre-defined large model, so that the large model can better understand the action process and generate descriptive fragments more accurately.

[0076] In yet another embodiment provided in this disclosure, the description fragment includes at least: action duration and action description; In step S101 above, "extracting the human movements contained in the action reference data into corresponding textual description fragments using a preset large model" can be implemented as follows: For the same action reference data, the human actions contained in the action reference data are extracted into corresponding textual description fragments multiple times through a preset large model; Robot motion generation methods also include: The extracted descriptive fragments are filtered based on action duration and action description respectively to determine the target descriptive fragments that match the corresponding action reference data; The step S102 above, "synthesizing an action segment representing the movement of the corresponding human body part in space based on the text information contained in the description segment," can be implemented as follows: Based on the text information contained in the target description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized.

[0077] In this embodiment of the disclosure, in order to improve the accuracy of the description of human motion content in the motion reference data, the description fragments can be generated multiple times and then the description fragments can be filtered for effectiveness.

[0078] This process can be based on the action duration and action description in the description fragment. Specifically, action duration refers to the textual description in the description fragment of the duration taken to complete the entire human action process in the action reference data, that is, the pre-defined large model's understanding of the duration taken for the action process; action description is the description of the specific action process, that is, the pre-defined large model's understanding of the human action process.

[0079] For action duration, it can be compared with the duration originally indicated in the action reference data (e.g., the duration of video data or the duration described in text data). The action duration is compared with the original duration, and then description segments that do not match the two (e.g., the action duration is 75% less than the original duration) are removed based on the comparison results.

[0080] Furthermore, the descriptive segments that have undergone the aforementioned duration verification process can be further verified through action description. By judging the differences between the action description and the original action reference data, it can be determined whether hallucinations occur in the action description. This process can involve human experts to review the content, or it can incorporate other assessment models to review the content.

[0081] In one possible implementation, the above-mentioned evaluation model can be a discriminant model based on consistency scoring. By taking action reference data as a premise as input, generating action descriptions as hypotheses, and using a discriminant model (e.g., VideoBERT) to measure the semantic space consistency between text and video, the difference between the two can be determined (e.g., by performing cosine similarity calculation). If the difference is greater than a certain threshold, it is judged that a hallucination has occurred.

[0082] Finally, through the above process, the description fragment most similar to the original motion reference data can be extracted from the multiple obtained description fragments and input into the 3D motion generation model as the target description fragment to synthesize motion fragments.

[0083] This disclosure has at least the following beneficial effects: First, it significantly reduces the cost of obtaining specific robot movements, eliminating the need for expensive motion capture equipment, as well as time-consuming model training and data collection.

[0084] Secondly, complex motion movements can be obtained simply by inputting text, without the need for professional actors.

[0085] Third, it significantly reduces the time required to obtain specific robot actions, decreasing the time consumption from days or even weeks to hours.

[0086] Fourth, for robots with different structures, the URDF file can be easily adjusted for complete reuse.

[0087] Furthermore, compared to models that generate specific robot actions (e.g., Pi-0), the method provided in this disclosure is not limited by the type and structure of the robot itself, and can provide high human-like action for various target robots. Compared to models that obtain coordinates in a specific coordinate system based on video (e.g., GVHMR), the method provided in this disclosure can support input from multiple modalities, solving the problem of difficulty in obtaining videos for complex and difficult actions.

[0088] This disclosure also provides a robot motion generation system, such as Figure 3 As shown, it includes: The extraction module 301 is used to acquire motion reference data containing human motions, and extract the human motions contained in the motion reference data into corresponding textual description fragments through a preset large model. The synthesis module 302 is used to synthesize an action segment representing the movement of a corresponding human body part in space based on the text information contained in the description segment. Matching module 303 is used to synthesize the trajectories of multiple action segments according to a preset matching algorithm to obtain multiple action trajectories composed of action segments; The conversion module 304 is used to convert the motion trajectory into a motion trajectory that guides the target robot to perform actions, based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure.

[0089] In another embodiment provided in this disclosure, the synthesis module 302 is used to extract semantic features of the text information contained in each description segment, and generate a first preset number of action segments corresponding to the description segment based on the semantic features; wherein each action segment contains continuous pose data of the corresponding human body part in space; the action segments are filtered based on the continuous pose data to determine the action segments that meet the physical constraints of the corresponding human body part.

[0090] In another embodiment provided in this disclosure, the matching module 303 is used to divide multiple action segments into segments based on the timing of the action segments in the execution sequence of actions in the corresponding human body parts, such that each segment set contains at least one candidate action segment with the same timing; starting from the starting action segment determined according to the timing, for the current action segment that needs to be matched, according to the timing corresponding to the current action segment, the next action segment is matched from the candidate action segments contained in the segment set corresponding to the next timing, and the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0091] In another embodiment provided in this disclosure, the matching module 303 is used to determine, for the current action segment that needs to be matched, the first pose data of the corresponding human body part at a preset length of the end of the current action segment; determine the candidate action segments included in the segment set corresponding to the next time sequence according to the time sequence corresponding to the current action segment; determine the second pose data of the corresponding human body part at a preset length of the beginning of each candidate action segment; match the first pose data and the second pose data; determine the candidate action segment corresponding to the second pose data that meets the preset matching conditions as the next action segment of the current action segment, and perform trajectory synthesis with the current action segment until the synthesis of the corresponding action trajectory is completed.

[0092] In another embodiment provided in this disclosure, the conversion module 304 is used to map the corresponding human body part to the corresponding robot skeleton structure, so as to convert the pose data of the corresponding human body part contained in the motion trajectory into the pose information of the robot skeleton structure; according to the pose information of the robot skeleton structure, the motion path of the robot skeleton structure is planned in chronological order to obtain the initial motion trajectory of the target robot; according to the physical constraints of the robot skeleton structure, the smoothness of the initial motion trajectory is determined; if the smoothness determination result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory.

[0093] In another embodiment provided in this disclosure, the conversion module 304 is used to calculate the motion parameters of the target node in the robot skeleton structure based on the pose information corresponding to the trajectory points in the initial motion trajectory; compare the motion parameters of the target node with the physical constraint threshold set for the target node; if the motion parameters are higher than the physical constraint threshold, determine that the initial motion trajectory is not smooth; if the motion parameters are not higher than the physical constraint threshold, determine that the initial motion trajectory is smooth.

[0094] In another embodiment provided in this disclosure, the conversion module 304 is used to perform data interpolation between trajectory points of the initial motion trajectory when the initial motion trajectory is not smooth, using a quadratic programming algorithm to obtain a smoothed target motion trajectory. The conversion module 304 is also used to output the initial motion trajectory as the target motion trajectory if the initial motion trajectory is smooth.

[0095] In another embodiment provided in this disclosure, the robot motion generation system further includes: The simulation verification module 305 is used to perform simulation evaluation on the obtained motion trajectory, and score the motion trajectory according to the degree of adaptation between the physical characteristics of the target robot and the motion trajectory during the simulation process, so as to obtain a simulation reward score for each motion trajectory; compare the motion trajectory with the corresponding action reference trajectory, and score the motion trajectory according to the degree of deviation between the motion trajectory and the action reference trajectory, so as to obtain a compliance score for each motion trajectory; the action reference trajectory includes at least one of the following: action reference data, textual description fragments, action fragments or action trajectories; based on the simulation reward score and the compliance score, calculate the final trajectory score for each motion trajectory, and select a second preset number of motion trajectories from each motion trajectory based on the final trajectory score as candidate motion trajectories to guide the target robot to perform actions.

[0096] In yet another embodiment provided in this disclosure, the motion reference data includes at least one of the following: video data, image data, and text data; Extraction module 301 is used to sample video data to obtain continuous video segments with a preset frame rate; call a preset large model to infer the continuous video segments and output textual descriptions of human actions contained in the continuous video segments in a preset format; for image data, call a preset large model to infer the image data and output textual descriptions of human actions contained in the image data in a preset format; for text data, call a preset large model to convert the text data into textual descriptions in a preset format.

[0097] In yet another embodiment provided in this disclosure, the description fragment includes at least: action duration and action description; Extraction module 301 is used to extract human actions contained in the same action reference data into corresponding textual description fragments multiple times through a preset large model. The robot motion generation system also includes a filtering module 306, which is used to filter the extracted description fragments based on the motion duration and motion description respectively, so as to determine the target description fragments that match the corresponding motion reference data. The synthesis module 302 is used to synthesize an action segment representing the movement of the corresponding human body part in space based on the text information contained in the target description segment.

[0098] like Figure 4 The diagram shown illustrates one architecture of the aforementioned robot motion generation system. It demonstrates that motion reference data for various modalities can be input into a pre-defined large model in the extraction module, from which semantics are extracted to obtain multiple descriptive fragments. It should be noted that depth stream data needs to be converted into a depth map or depth video before being input into the pre-defined large model.

[0099] These descriptive fragments can be filtered by the filtering module to obtain the target descriptive fragments that best match the motion reference data. These fragments are then input into the 3D motion synthesis model of the synthesis module to generate multiple motion fragments. These motion fragments are matched using the matching algorithm of the matching module, forming multiple motion trajectories. After the motion trajectories are input into the conversion module, they are converted into motion trajectories adapted to the target robot based on the GMR and URDF models. Then, the dynamic interpolation submodule in the conversion module performs smoothing determination and data interpolation operations, resulting in multiple smoothed target motion trajectories.

[0100] These target motion trajectories can be output to the simulation verification module for further simulation verification, and scores can be assigned for simulation reward points and compliance scores, respectively. Based on the scoring results, the optimization and evaluation submodule of the simulation verification module outputs one or more candidate motion trajectories with the highest scores to guide the robot to perform the corresponding actions.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0102] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0103] Those skilled in the art will understand that the modules in the system of the embodiments can be distributed in the system of the embodiments as described in the embodiments, or they can be located in one or more systems different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0104] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0105] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for generating robot actions, characterized in that, include: Obtain motion reference data containing human movements, and extract the human movements contained in the motion reference data into corresponding textual description fragments using a preset large model; Based on the text information contained in the description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized; Multiple action segments are combined into a trajectory using a preset matching algorithm to obtain multiple action trajectories composed of action segments; Based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure, the motion trajectory is converted into a motion trajectory that guides the target robot to perform actions.

2. The method as described in claim 1, characterized in that, Based on the text information contained in the description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized, including: For each description segment, semantic features of the text information contained in the description segment are extracted, and based on the semantic features, a first preset number of action segments corresponding to the description segment are generated; wherein, each action segment contains continuous pose data of the corresponding human body part in space; Action segments are filtered based on continuous pose data to determine those that meet the physical constraints of the corresponding human body parts.

3. The method as described in claim 1, characterized in that, Multiple action segments are combined using a preset matching algorithm to obtain multiple action trajectories composed of action segments, including: Based on the temporal order of the action segments in the corresponding human body parts, multiple action segments are divided into segment sets, such that each segment set contains at least one candidate action segment with the same temporal order. Starting from the initial action segment determined by the timing sequence, for the current action segment that needs to be matched, based on the timing sequence corresponding to the current action segment, the next action segment is matched from the candidate action segments contained in the segment set corresponding to the next timing sequence, and the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed.

4. The method as described in claim 3, characterized in that, For the current action segment that requires matching, based on the timing of the current action segment, the next action segment is matched from the candidate action segments contained in the set of segments corresponding to the next timing segment, and then the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed, including: For the current action segment that requires matching, determine the first pose data of the corresponding human body part at the preset length of the end of the current action segment; Based on the timing of the current action segment, determine the candidate action segments included in the set of segments corresponding to the next timing segment; Determine the second pose data of the corresponding human body part with the preset length of the beginning of each candidate action segment; The first pose data and the second pose data are matched. The candidate action segment corresponding to the second pose data that meets the preset matching conditions is determined as the next action segment of the current action segment, and the trajectory is synthesized with the current action segment until the synthesis of the corresponding action trajectory is completed.

5. The method as described in claim 1, characterized in that, Based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure, the motion trajectory is converted into a motion trajectory to guide the target robot to perform actions, including: The corresponding human body parts are mapped to the corresponding robot skeleton structure, so as to convert the pose data of the corresponding human body parts contained in the motion trajectory into the pose information of the robot skeleton structure. Based on the pose information of the robot's skeletal structure, the motion path of the robot's skeletal structure is planned in chronological order to obtain the initial motion trajectory of the target robot. The smoothness of the initial motion trajectory is determined based on the physical constraints of the robot's skeletal structure. If the smoothness judgment result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory.

6. The method as described in claim 5, characterized in that, Based on the physical constraints of the robot's skeletal structure, the smoothness of the initial motion trajectory is determined, including: Based on the pose information corresponding to the trajectory points in the initial motion trajectory, calculate the motion parameters of the target nodes in the robot's skeletal structure; The motion parameters of the target node are compared with the physical constraint thresholds set for the target node; If the motion parameters are higher than the physical constraint threshold, the initial motion trajectory is determined to be unsmooth. If the motion parameters are not higher than the physical constraint threshold, the initial motion trajectory is determined to be smooth.

7. The method as described in claim 5, characterized in that, If the smoothness judgment result indicates that the initial motion trajectory is not smooth, a corresponding trajectory correction strategy is executed on the initial motion trajectory to obtain the target motion trajectory, including: When the initial motion trajectory is not smooth, a quadratic programming algorithm is used to interpolate data between the trajectory points of the initial motion trajectory to obtain a smoothed target motion trajectory. The method further includes: If the initial motion trajectory is smooth, the initial motion trajectory is output as the target motion trajectory.

8. The method as described in claim 1, characterized in that, Also includes: The obtained motion trajectory is simulated and evaluated, and the motion trajectory is scored according to the degree of adaptation between the physical characteristics of the target robot and the motion trajectory during the simulation process, and the simulation reward score of each motion trajectory is obtained. The motion trajectory is compared with the corresponding action reference trajectory, and the motion trajectory is scored according to the degree of deviation between the motion trajectory and the action reference trajectory to obtain the degree of compliance score of each motion trajectory; the action reference trajectory includes at least one of the following: action reference data, textual description fragments, action fragments or motion trajectories; Based on the simulation reward score and the compliance score, the final trajectory score of each motion trajectory is calculated, and based on the final trajectory score, a second preset number of motion trajectories are selected from each motion trajectory as candidate motion trajectories to guide the target robot to perform actions.

9. The method as described in claim 1, characterized in that, The motion reference data includes at least one of the following: video data, image data, and text data; The human movements contained in the motion reference data are extracted into corresponding textual description fragments using a pre-defined large model, including: The video data is sampled to obtain continuous video segments with a preset frame rate; a preset large model is called to perform inference on the continuous video segments, and a textual description of human actions contained in the continuous video segments is output in a preset format. For image data, a preset large model is invoked to perform inference on the image data, and a textual description of the human body movements contained in the image data is output in a preset format. For text data, a preset large model is invoked to convert the text data into a textual description fragment in a preset format.

10. The method as described in claim 1, characterized in that, The description fragment includes at least: action duration and action description; The human movements contained in the motion reference data are extracted into corresponding textual description fragments using a pre-defined large model, including: For the same action reference data, the human actions contained in the action reference data are extracted into corresponding textual description fragments multiple times through a preset large model; The method further includes: The extracted descriptive fragments are filtered based on action duration and action description respectively to determine the target descriptive fragments that match the corresponding action reference data; Based on the text information contained in the description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized, including: Based on the text information contained in the target description fragment, an action fragment representing the movement of the corresponding human body part in space is synthesized.

11. A robot motion generation system, characterized in that, include: The extraction module is used to acquire motion reference data containing human movements, and extract the human movements contained in the motion reference data into corresponding textual description fragments through a preset large model. The synthesis module is used to synthesize action segments representing the movement of corresponding human body parts in space based on the text information contained in the description fragment; The matching module is used to synthesize the trajectories of multiple action segments according to a preset matching algorithm, so as to obtain multiple action trajectories composed of action segments. The conversion module is used to convert the motion trajectory into a motion trajectory that guides the target robot to perform actions, based on the mapping relationship between the corresponding human body parts and the robot's skeletal structure.

Images