Video-driven bionic robot dance motion generation method and system

Abstract

The application discloses a video driving-based bionic robot dance action generation method and system, and relates to the technical field of computer vision.The method comprises the following steps: acquiring a video containing human dance actions and preprocessing the video to obtain a standardized video frame sequence; adaptively extracting human skeleton key point motion trajectories based on motion consistency constraints and performing time sequence filtering to obtain a key point coordinate sequence; determining three human key points corresponding to each robot joint according to a robot joint mapping table, and outputting a joint angle sequence and an angular velocity sequence corresponding to the robot joint; predefining a bionic robot joint set and kinematic parameters, establishing an adaptive nonlinear mapping function, and calculating a target angle sequence of each joint of the robot; and performing sliding window average filtering on the target angle sequence, and outputting a final executable dance action data.The application can automatically generate dance actions conforming to the kinematic constraints and smooth actions of a bionic robot from a dance video.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method and system for generating video-driven bionic robot dance movements. Background Technology

[0002] With the rapid development of bionic robot technology, enabling robots to dance like humans has become a hot demand in fields such as robot performance, human-computer interaction, and rehabilitation training. Traditional methods for generating robot dance movements mainly rely on two approaches: one is for professional choreographers to manually edit joint angles frame by frame through instructional programming or graphical interfaces, which is time-consuming, labor-intensive, and requires high programming skills; the other is to use expensive optical or inertial motion capture equipment to collect motion data from professional dancers, and then post-process and map it onto the robot, which is costly, has complex equipment deployment, and is difficult to popularize.

[0003] Significant progress has been made in video-based human pose estimation technology, making it possible to extract human skeletal key points from ordinary videos. However, existing video-driven methods still have the following shortcomings: for example, when directly mapping the coordinates of human key points to robot joint angles, the mechanical limits and dynamic constraints of the robot are ignored, which can easily generate movements that exceed the joint range or cause the robot to become unbalanced; the rhythm and coordination of human dance movements are not considered, and the generated robot movements often have obvious shaking and a mechanical feel; and there is a lack of modeling of the limb coupling in dance movements, such as the shoulder leading the elbow when swinging the arm, resulting in low naturalness of the movements.

[0004] Therefore, there is an urgent need for a method to automatically generate natural, smooth dance movements that conform to robot kinematics constraints from ordinary dance videos. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a video-driven method and system for generating bionic robot dance movements.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A video-driven method for generating dance movements for bionic robots includes the following steps:

[0008] Step S1: Obtain a video containing human dance movements and preprocess the video to obtain a standardized video frame sequence;

[0009] Step S2: Based on motion consistency constraints, adaptively extract the motion trajectory of key points of human skeleton from the standardized video frame sequence and perform temporal filtering to obtain the key point coordinate sequence;

[0010] Step S3: Determine the three human key points corresponding to each robot joint according to the robot joint mapping table, extract these key points and calculate their included angle as human joint angle, then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint.

[0011] Step S4: Predefine the set of joints and their kinematic parameters of the bionic robot, establish an adaptive nonlinear mapping function from human joints to robot joints, and calculate the target angle sequence of each joint of the robot. The mapping function is composed of the addition of a linear scaling term and a dynamic compensation term.

[0012] Step S5: Perform sliding window averaging filtering on the robot joint angle sequence generated in step S4 to output the final executable dance motion data.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] Step S1.1: Decode the video into consecutive frame images and record the total number of frames;

[0015] Step S1.2: Unify the video frame rate to 30 frames / second and perform Gaussian filtering to denoise each frame.

[0016] Step S1.3: Scale all frame images to a uniform size of 640×480 pixels.

[0017] Furthermore, step S2 specifically includes the following steps:

[0018] Step S2.1: Calculate the horizontal and vertical gradients for each frame of the image and construct a gradient magnitude map;

[0019] Step S2.2: Calculate the optical flow field between two adjacent frames, and calculate the motion consistency metric for each pixel based on the deviation between the optical flow vector and the global average optical flow and the response of the image Laplacian operator. Pixels with motion consistency metrics higher than a preset threshold are selected as candidate regions for human key points.

[0020] Step S2.3: Construct a joint objective function, which includes motion consistency energy, skeletal geometric constraint energy, and temporal smoothing constraint energy. Optimize the joint objective function to determine the optimal coordinates of each keypoint in each frame.

[0021] Step S2.4: Perform velocity-adaptive exponential smoothing filtering on the coordinate sequence of each key point, where the smoothing coefficient is inversely proportional to the instantaneous motion velocity of the key point.

[0022] Further, in step S2.3, the motion consistency energy is the sum of the negative logarithmic values ​​of the motion consistency metric at each keypoint position in each frame; the skeletal geometric constraint energy is the sum of the squares of the differences between the actual vectors on each bone connection edge in each frame and the preset standard bone vector, multiplied by the geometric constraint weight coefficient; the temporal smoothing constraint energy is the sum of the squares of the displacements of the same keypoint between adjacent frames, multiplied by the temporal smoothing weight coefficient; the three energies are added together to obtain the total energy, and the total energy is minimized by the gradient descent method to obtain the optimal coordinates of each keypoint in each frame.

[0023] Furthermore, step S3 specifically includes the following steps:

[0024] Step S3.1: Based on the joint configuration of the bionic robot, a mapping table is pre-established, and a corresponding human key point triplet is assigned to each robot joint;

[0025] Step S3.2: In the key point sequence obtained in step S2, only the key points involved in the mapping table are retained, and other irrelevant points are ignored;

[0026] Step S3.3: For each robot joint, take the corresponding human joint center key point, calculate the vector from the key point to the near key point and the vector to the far key point, and obtain the human joint angle value corresponding to the joint by the vector dot product and the inverse cosine function.

[0027] Step S3.4: Divide the difference between the human joint angles in the current frame and the previous frame by the inter-frame time interval to obtain the angular velocity value corresponding to the joint. The inter-frame time interval is determined by the reciprocal of the video frame rate.

[0028] Step S3.5: When key points are missing in the triplet, the angle value of the previous frame is used as the estimate of the current frame first. If the same joint is missing for more than 3 consecutive frames, the angle value of the symmetrical joint is multiplied by the direction coefficient for mirror replacement. If the symmetrical joint is also unavailable, the angle of the current frame is set to the historical average of the joint or the robot's neutral position angle. Finally, the human joint angle sequence and angular velocity sequence corresponding one-to-one with the robot joints are output.

[0029] Furthermore, step S4 specifically includes the following steps:

[0030] Step S4.1: Calculate the linear scaling term by subtracting the neutral angle of the human joint from the corresponding human joint angle, multiplying it by the ratio of the robot joint's range of motion to the human joint's range of motion, and then adding the robot joint's neutral angle.

[0031] Step S4.2: Calculate the speed compensation sub-item, which is obtained by multiplying the amplitude coefficient by the hyperbolic tangent function, where the independent variable of the hyperbolic tangent function is the scaling factor multiplied by the human joint angular velocity;

[0032] Step S4.3: Calculate the weighted coupling sub-item of adjacent joints. For each adjacent joint in the set of adjacent joint indices that have motion coupling relationship with the current robot joint, multiply its angle value at the previous moment by the corresponding coupling coefficient and then sum them up.

[0033] Step S4.4: Calculate the secondary velocity sub-term, which is obtained by multiplying the secondary velocity compensation coefficient by the square of the human joint angular velocity and then by the sign function of the human joint angular velocity;

[0034] Step S4.5: Add the three sub-items—velocity compensation, adjacent joint weighted coupling, and secondary velocity—to obtain the dynamic compensation term, and then sum it with the linear scaling term to obtain the target angle of the robot joint.

[0035] Furthermore, in step S4, for the lumbar torsion joint, a weighted linear combination of the angle difference between the left and right hip joints and the angle difference between the left and right shoulder joints is used for mapping, wherein the weight coefficients are obtained by least squares fitting on the demonstration dataset.

[0036] Furthermore, step S5 specifically includes the following steps:

[0037] Step S5.1: For the angle sequence of each robot joint, take the current frame as the center, take 2 frames before and after it, and perform an arithmetic average of 5 frames to obtain the filtered angle value.

[0038] Step S5.2: Apply symmetrical padding to the sequence boundaries, and package the filtered angle sequence into robot control instructions in chronological order, outputting the final executable dance motion data.

[0039] A video-driven bionic robot dance motion generation system is provided to implement any one of the video-driven bionic robot dance motion generation methods described above, including:

[0040] The video acquisition and preprocessing module is used to acquire dance videos, decode the videos into continuous frame images and unify the frame rate, perform Gaussian filtering to remove noise from each frame image, and scale the image size to 640×480 pixels.

[0041] The adaptive key point extraction and filtering module is used to adaptively extract the motion trajectory of human skeleton key points from a standardized video frame sequence based on motion consistency constraints and perform temporal filtering to obtain a key point coordinate sequence.

[0042] The human joint motion calculation module is used to determine the human key points corresponding to each robot joint according to the robot joint mapping table, extract the key points and calculate their included angle as the human joint angle, and then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint.

[0043] A robot kinematics parameter library for storing joint definitions, mechanical limit angles, neutral position angles, and adjacent joint coupling coefficient matrices for biomimetic robots;

[0044] The adaptive nonlinear mapping module is used to establish an adaptive nonlinear mapping function from human joints to robot joints and to calculate the target angle sequence of each robot joint.

[0045] The motion smoothing and output module is used to perform sliding window averaging filtering on the robot's joint angle sequence and output the final executable dance motion data.

[0046] A storage medium storing instructions that, when read by a computer, cause the computer to execute any of the video-driven bionic robot dance motion generation methods described above.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] 1. This invention uses a motion consistency constraint joint optimization method, without relying on existing pose estimation networks, to stably extract key points of the human skeleton from ordinary videos. It is especially suitable for fast-moving scenes such as dance, with high extraction accuracy and strong anti-interference ability.

[0049] 2. This invention proposes a nonlinear mapping function composed of a linear scaling term and a dynamic compensation term, which ensures that the robot's movements do not exceed mechanical limits, and simulates the speed bursts, limb coordination and centrifugal effects in human dance, thereby improving the naturalness and vividness of robot dance.

[0050] 3. This invention establishes an explicit mapping table between robot joints and human key points, calculates only the joint angles required by the robot, has high computational efficiency, and ensures the continuity and stability of the action sequence by handling missing key points.

[0051] 4. This invention further eliminates high-frequency jitter through sliding window averaging filtering while preserving the rhythmic characteristics of the dance, resulting in smooth and executable dance movements. Attached Figure Description

[0052] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0053] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention;

[0055] Figure 3This is a schematic diagram of the mapping from human key points to robot joints in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, the video-driven bionic robot dance motion generation method includes the following steps:

[0058] Step S1: Obtain a video containing human dance movements and preprocess the video to obtain a standardized video frame sequence;

[0059] Step S2: Based on motion consistency constraints, adaptively extract the motion trajectory of key points of human skeleton from the standardized video frame sequence and perform temporal filtering to obtain the key point coordinate sequence;

[0060] Step S3: Determine the three human key points corresponding to each robot joint according to the robot joint mapping table, extract these key points and calculate their included angle as human joint angle, then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint.

[0061] Step S4: Predefine the set of joints and their kinematic parameters of the bionic robot, establish an adaptive nonlinear mapping function from human joints to robot joints, and calculate the target angle sequence of each joint of the robot. The mapping function is composed of the addition of a linear scaling term and a dynamic compensation term.

[0062] Step S5: Perform sliding window averaging filtering on the robot joint angle sequence generated in step S4 to output the final executable dance motion data.

[0063] Step S1 specifically includes:

[0064] Acquire video files or real-time video streams containing human dance movements, decode the video into continuous frame images, and record the total number of frames as T. Then, perform frame rate normalization on the video, setting the frame rate to a uniform 30 frames per second to ensure consistency of the timeline. For each frame image, use Gaussian filtering for noise reduction, with a Gaussian kernel size of 5×5 pixels and a standard deviation of 1.0, to remove sensor noise and environmental interference during the acquisition process. Scale all frame images to a uniform size of 640×480 pixels to reduce subsequent computation and maintain aspect ratio. The preprocessed image sequence is recorded as the normalized video frame sequence.

[0065] Step S2 specifically includes the following steps:

[0066] Step S2.1: Calculate the horizontal and vertical gradients for each frame of the image and construct a gradient magnitude map;

[0067] Step S2.2: Calculate the optical flow field between two adjacent frames, and calculate the motion consistency metric for each pixel based on the deviation between the optical flow vector and the global average optical flow and the response of the image Laplacian operator. Pixels with motion consistency metrics higher than a preset threshold are selected as candidate regions for human key points.

[0068] Step S2.3: Construct a joint objective function, which includes motion consistency energy, skeletal geometric constraint energy, and temporal smoothing constraint energy. Optimize the joint objective function to determine the optimal coordinates of each keypoint in each frame.

[0069] Step S2.4: Perform velocity-adaptive exponential smoothing filtering on the coordinate sequence of each key point, where the smoothing coefficient is inversely proportional to the instantaneous motion velocity of the key point.

[0070] In step S2.3, the motion consistency energy is the sum of the negative logarithmic values ​​of the motion consistency metric at each keypoint position in each frame; the skeletal geometric constraint energy is the sum of the squares of the differences between the actual vectors on each bone connection edge in each frame and the preset standard bone vectors, multiplied by the geometric constraint weight coefficient; the temporal smoothing constraint energy is the sum of the squares of the displacements of the same keypoint between adjacent frames, multiplied by the temporal smoothing weight coefficient; the three energies are added together to obtain the total energy, and the gradient descent method is used to minimize the total energy to obtain the optimal coordinates of each keypoint in each frame.

[0071] The specific formula for the gradient magnitude diagram is as follows:

[0072]

[0073] in, Let represent the gradient magnitude of the image at pixel coordinates (x, y) in frame t. The first derivative of the image at (x, y) in the horizontal direction is represented by the Sobel operator. This represents the first derivative of the image in the vertical direction at (x, y);

[0074] The Farneback algorithm is used to calculate the optical flow field between two adjacent frames, and a motion consistency metric for each pixel is defined, with the following formula:

[0075]

[0076] in, This represents the motion consistency metric for pixel (x, y) in frame t. A higher value indicates that the pixel is more likely to be a human keypoint. This represents the optical flow vector of pixel (x, y) in frame t. Let represent the average value of the optical flow vector for the entire frame t. The Euclidean norm of a vector. This represents the optical flow deviation scale parameter, typically taken as 5 pixels. This represents the weighting coefficient of the Laplace term, typically taken as 0.5. Let the Laplacian operator be the normalized image at (x, y) in frame t. This represents the maximum value of the Laplacian response modulus of all pixels in frame t, used for normalization. Pixels with a value greater than 0.7 are considered as candidate regions for human body key points;

[0077] Let the position of the i-th key point in the t-th frame be... The specific formulas for the three energy sub-items are as follows:

[0078]

[0079] in, Represents the uniformity of motion energy. Indicates the total number of frames. Indicates the total number of key points. Represents the key points of frame t. The motion consistency metric Represents the geometric constraint energy of the skeleton. Represents the set of edges connecting the skeleton. This represents the actual vectors of keypoints i and j in frame t. This represents a preset standard skeletal vector, learned from training data, with its length in pixels. Represents the energy constrained by time series smoothing. This represents the displacement vector of keypoint i from frame (t-1) to frame t.

[0080] The specific formula for the joint objective function is as follows:

[0081]

[0082] in, Describe the joint objective function. This represents the geometric constraint weight, typically set to 0.1. This represents the time-series smoothing weight, typically set to 0.05;

[0083] Specifically, the gradient descent method is used to iteratively solve for the keypoint coordinates that minimize the total energy. This involves: initializing the coordinates of each keypoint in each frame to the center point of the candidate region with the highest motion consistency metric; calculating the gradient of the total energy with respect to the coordinates of each keypoint in each iteration, i.e., the direction and magnitude of the small change in energy with respect to the coordinates; and updating the coordinate values ​​along the direction that causes the total energy to decrease the fastest with a set learning rate; repeating this iterative process until the change in the coordinates of all keypoints is less than the preset convergence threshold, or the maximum number of iterations is reached.

[0084] Adaptive exponential smoothing is applied to the coordinate sequence of each key point. Taking the x-coordinate as an example, the specific formula is as follows:

[0085]

[0086] in This represents the filtered x-coordinate of keypoint i in frame t. Indicates the adaptive smoothing coefficient. This represents the x-coordinate before filtering. This represents the filtered x-coordinate of the previous frame;

[0087] The specific formula for the adaptive smoothing coefficient is as follows:

[0088]

[0089] in, This represents the speed sensitivity factor, typically taken as 0.2. This represents the inter-frame displacement of keypoint i in the x-direction. This represents the inter-frame displacement of keypoint i in the y-direction;

[0090] Similarly, for the y-coordinate, we obtain... The final output is a smoothed sequence of key points.

[0091] Step S3 specifically includes the following steps:

[0092] Step S3.1: Based on the joint configuration of the bionic robot, a mapping table is pre-established, and a corresponding human key point triplet is assigned to each robot joint;

[0093] Step S3.2: In the key point sequence obtained in step S2, only the key points involved in the mapping table are retained, and other irrelevant points are ignored;

[0094] Step S3.3: For each robot joint, take the corresponding human joint center key point, calculate the vector from the key point to the near key point and the vector to the far key point, and obtain the human joint angle value corresponding to the joint by the vector dot product and the inverse cosine function.

[0095] Step S3.4: Divide the difference between the human joint angles in the current frame and the previous frame by the inter-frame time interval to obtain the angular velocity value corresponding to the joint. The inter-frame time interval is determined by the reciprocal of the video frame rate.

[0096] Step S3.5: When key points are missing in the triplet, the angle value of the previous frame is used as the estimate of the current frame first. If the same joint is missing for more than 3 consecutive frames, the angle value of the symmetrical joint is multiplied by the direction coefficient for mirror replacement. If the symmetrical joint is also unavailable, the angle of the current frame is set to the historical average of the joint or the robot's neutral position angle. Finally, the human joint angle sequence and angular velocity sequence corresponding one-to-one with the robot joints are output.

[0097] A table of correspondences between each joint of the bionic robot and key points of the human body is pre-established, and only the joint angles required by the robot are extracted.

[0098] A mapping table is determined based on the joint configuration of the bionic robot. Assume the robot has M joints, including: 2 for neck rotation and pitch, 4 for left and right shoulder pitch and roll, 2 for left and right elbow flexion and extension, 2 for left and right wrist rotation, 1 for lumbar twist, 2 for left and right hip flexion and extension, 2 for left and right knee flexion and extension, and 2 for left and right ankle flexion and extension, totaling 17 joints. Each robot joint is assigned a corresponding human joint triplet: proximal keypoint, central keypoint, and distal keypoint. For example, the robot's left elbow joint corresponds to the human left shoulder (proximal), left elbow (central), and left wrist (distal); the robot's left knee joint corresponds to the human left hip (proximal), left knee (central), and left ankle (distal). For the lumbar twist joint, there is no direct triplet; instead, the angle difference between the left and right hips and shoulders is used.

[0099] For each robot joint k, only three key points related to that joint are retained in the obtained key point set, while other irrelevant points are ignored to improve efficiency;

[0100] For each robot joint k, take the coordinates of the corresponding human joint center key point, calculate the vector from the proximal end to the center and the vector from the center to the distal end. The angle between these two vectors is the human joint angle corresponding to the robot joint. The calculation uses the vector dot product formula, divides the dot product by the product of the magnitudes, and then takes the inverse cosine. The result is in radians. The angular velocity is obtained by dividing the angle difference between adjacent frames by the frame interval in seconds. The angular velocity of the first frame is set to 0.

[0101] The final output is the sequence of human joint angles and angular velocities corresponding to the robot joints.

[0102] Step S4 specifically includes the following steps:

[0103] Step S4.1: Calculate the linear scaling term by subtracting the neutral angle of the human joint from the corresponding human joint angle, multiplying it by the ratio of the robot joint's range of motion to the human joint's range of motion, and then adding the robot joint's neutral angle.

[0104] Step S4.2: Calculate the speed compensation sub-item, which is obtained by multiplying the amplitude coefficient by the hyperbolic tangent function, where the independent variable of the hyperbolic tangent function is the scaling factor multiplied by the human joint angular velocity;

[0105] Step S4.3: Calculate the weighted coupling sub-item of adjacent joints. For each adjacent joint in the set of adjacent joint indices that have motion coupling relationship with the current robot joint, multiply its angle value at the previous moment by the corresponding coupling coefficient and then sum them up.

[0106] Step S4.4: Calculate the secondary velocity sub-term, which is obtained by multiplying the secondary velocity compensation coefficient by the square of the human joint angular velocity and then by the sign function of the human joint angular velocity;

[0107] Step S4.5: Add the three sub-items—velocity compensation, adjacent joint weighted coupling, and secondary velocity—to obtain the dynamic compensation term, and then sum it with the linear scaling term to obtain the target angle of the robot joint.

[0108] In step S4, for the lumbar torsion joint, a weighted linear combination of the angle difference between the left and right hip joints and the angle difference between the left and right shoulder joints is used for mapping, wherein the weight coefficients are obtained by least squares fitting on the demonstration dataset.

[0109] The robot kinematics parameter library pre-stores parameters for each robot joint, including: the maximum and minimum mechanical limit angles of the joint; the neutral angle calculated from the maximum and minimum limits; the maximum and minimum range of motion of the human joint corresponding to the joint, and the corresponding neutral angle of the human joint; the velocity compensation amplitude coefficient; the velocity scaling factor; the secondary velocity compensation coefficient; and the coupling coefficient matrix between adjacent joints. All of the above parameters are obtained through offline calibration or least squares fitting of demonstration data.

[0110] The linear scaling term linearly maps the angles of human joints to the mechanical limits of robot joints. The specific formula is as follows:

[0111]

[0112] in, This represents the linear scaling term of robot joint k in frame t. This represents the maximum and minimum range of motion of robot joint k. This represents the maximum and minimum range of motion of the human joint corresponding to robot joint k. This represents the human joint angle obtained in step S3. This indicates the neutral angle of a human joint. Indicates the neutral position angle of the robot joint;

[0113] The dynamic compensation term is obtained by adding three sub-terms, and the specific formula is as follows:

[0114]

[0115] in, Indicates the speed compensation sub-item. This represents the speed compensation amplitude coefficient, which is 0.05 times the range of motion of the robot joint. This represents the speed scaling factor, with a value of 2.0. This represents the angular velocity of the human joints obtained in step S3. This indicates a weighted coupling sub-item between adjacent joints. This represents the set of indices of adjacent joints that are kinematically coupled to joint k. For example, for the left shoulder joint, the set includes the left elbow joint. The coupling coefficient represents the influence strength of the historical angles of adjacent joints l on the current joint k, and is obtained by fitting dance demonstration data using the offline least squares method. This represents the robot joint angle of adjacent joint l in frame t-1. Indicates the second-order velocity sub-term. This represents the secondary velocity compensation coefficient, with a value of 0.03. Represents a symbolic function;

[0116] The dynamic compensation term is obtained by adding the velocity compensation term, the adjacent joint weighted coupling term, and the secondary velocity term. This dynamic compensation term is then summed with the linear scaling term to obtain the target angle of the robot joint. The specific formula is as follows:

[0117]

[0118] in, This represents the target angle of robot joint k in frame t. This represents the dynamic compensation term for robot joint k in frame t.

[0119] Step S5 specifically includes the following steps:

[0120] Step S5.1: For the angle sequence of each robot joint, take 2 frames before and after the current frame as the center, and perform an arithmetic average of 5 frames to obtain the filtered angle value.

[0121] Step S5.2: Apply symmetrical padding to the sequence boundaries, and package the filtered angle sequence into robot control instructions in chronological order, outputting the final executable dance motion data.

[0122] A video-driven bionic robot dance motion generation system is provided to implement any one of the video-driven bionic robot dance motion generation methods described above, including:

[0123] The video acquisition and preprocessing module is used to acquire dance videos, decode the videos into continuous frame images and unify the frame rate, perform Gaussian filtering to remove noise from each frame image, and scale the image size to 640×480 pixels.

[0124] The adaptive key point extraction and filtering module is used to adaptively extract the motion trajectory of human skeleton key points from a standardized video frame sequence based on motion consistency constraints and perform temporal filtering to obtain a key point coordinate sequence.

[0125] The human joint motion calculation module is used to determine the human key points corresponding to each robot joint according to the robot joint mapping table, extract the key points and calculate their included angle as the human joint angle, and then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint.

[0126] A robot kinematics parameter library for storing joint definitions, mechanical limit angles, neutral position angles, and adjacent joint coupling coefficient matrices for biomimetic robots;

[0127] The adaptive nonlinear mapping module is used to establish an adaptive nonlinear mapping function from human joints to robot joints and to calculate the target angle sequence of each robot joint.

[0128] The motion smoothing and output module is used to perform sliding window averaging filtering on the robot's joint angle sequence and output the final executable dance motion data.

[0129] like Figure 3 As shown, the core mapping relationship is illustrated using the left arm as an example. The left side of the image represents the human posture, with three key points labeled: left shoulder, left elbow, and left wrist. The right side represents the left arm model of the bionic robot, with the left shoulder pitch joint, left shoulder roll joint, and left elbow flexion-extension joint labeled. The mapping rules are as follows:

[0130] The angle of the human left elbow joint is formed by the left shoulder, left elbow and left wrist, which is directly mapped to the target angle of the robot's left elbow flexion and extension joint.

[0131] The angle between the line connecting the left shoulder and left elbow of the human body and the horizontal plane, after scaling and offsetting, is mapped to the angle of the left shoulder pitch joint of the robot.

[0132] The lateral swing angle of the line connecting the left shoulder and left elbow of the human body is estimated by three-dimensional information or the difference between the left and right shoulders and mapped to the angle of the robot's left shoulder roll joint.

[0133] With this explicit mapping table, each robot joint can obtain drive from the most relevant human keypoint information, ensuring semantic consistency of the action and computational efficiency.

[0134] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0135] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

Claims

1. A video-driven method for generating dance movements in a bionic robot, characterized in that, Includes the following steps: Step S1: Obtain a video containing human dance movements and preprocess the video to obtain a standardized video frame sequence; Step S2: Based on motion consistency constraints, adaptively extract the motion trajectory of key points of human skeleton from the standardized video frame sequence and perform temporal filtering to obtain the key point coordinate sequence; Step S3: Determine the three human key points corresponding to each robot joint according to the robot joint mapping table, extract these key points and calculate their included angle as human joint angle, then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint. Step S4: Predefine the set of joints and their kinematic parameters of the bionic robot, establish an adaptive nonlinear mapping function from human joints to robot joints, and calculate the target angle sequence of each joint of the robot. The mapping function is composed of the addition of a linear scaling term and a dynamic compensation term. Step S5: Perform sliding window averaging filtering on the robot joint angle sequence generated in step S4, and output the final executable dance motion data. Specifically, step S4 includes the following steps: Step S4.1: Calculate the linear scaling term by subtracting the neutral angle of the human joint from the corresponding human joint angle, multiplying it by the ratio of the robot joint's range of motion to the human joint's range of motion, and then adding the robot joint's neutral angle. Step S4.2: Calculate the velocity compensation sub-item, which is obtained by multiplying the amplitude coefficient by the hyperbolic tangent function, where the independent variable of the hyperbolic tangent function is the scaling factor multiplied by the human joint angular velocity; Step S4.3: Calculate the weighted coupling sub-item of adjacent joints. For each adjacent joint in the set of adjacent joint indices that have motion coupling relationship with the current robot joint, multiply its angle value at the previous moment by the corresponding coupling coefficient and then sum them up. Step S4.4: Calculate the secondary velocity sub-term, which is obtained by multiplying the secondary velocity compensation coefficient by the square of the human joint angular velocity and then by the sign function of the human joint angular velocity; Step S4.5: Add the three sub-items—velocity compensation, adjacent joint weighted coupling, and secondary velocity—to obtain the dynamic compensation term, and then sum it with the linear scaling term to obtain the target angle of the robot joint. In step S4, for the lumbar torsion joint, a weighted linear combination of the angle difference between the left and right hip joints and the angle difference between the left and right shoulder joints is used for mapping, wherein the weight coefficients are obtained by least squares fitting on the demonstration dataset. Step S5 specifically includes the following steps: Step S5.1: For the angle sequence of each robot joint, take the current frame as the center, take 2 frames before and after it, and perform an arithmetic average of 5 frames to obtain the filtered angle value. Step S5.2: Apply symmetrical padding to the sequence boundaries, and package the filtered angle sequence into robot control instructions in chronological order, outputting the final executable dance motion data.

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: Step S1.1: Decode the video into consecutive frame images and record the total number of frames; Step S1.2: Unify the video frame rate to 30 frames / second and perform Gaussian filtering to denoise each frame. Step S1.3: Scale all frame images to a uniform size of 640×480 pixels.

3. The method according to claim 2, characterized in that, Step S2 specifically includes the following steps: Step S2.1: Calculate the horizontal and vertical gradients for each frame of the image and construct a gradient magnitude map; Step S2.2: Calculate the optical flow field between two adjacent frames, and calculate the motion consistency metric for each pixel based on the deviation between the optical flow vector and the global average optical flow and the response of the image Laplacian operator. Pixels with motion consistency metrics higher than a preset threshold are selected as candidate regions for human key points. Step S2.3: Construct a joint objective function, which includes motion consistency energy, skeletal geometric constraint energy, and temporal smoothing constraint energy. Optimize the joint objective function to determine the optimal coordinates of each keypoint in each frame. Step S2.4: Perform velocity-adaptive exponential smoothing filtering on the coordinate sequence of each key point, where the smoothing coefficient is inversely proportional to the instantaneous motion velocity of the key point.

4. The method according to claim 3, characterized in that, In step S2.3, the motion consistency energy is the sum of the negative logarithmic values ​​of the motion consistency metric at each keypoint position in each frame; the skeletal geometric constraint energy is the sum of the squares of the differences between the actual vectors on each bone connection edge in each frame and the preset standard bone vectors, multiplied by the geometric constraint weight coefficient; the temporal smoothing constraint energy is the sum of the squares of the displacements of the same keypoint between adjacent frames, multiplied by the temporal smoothing weight coefficient; the three energies are added together to obtain the total energy, and the gradient descent method is used to minimize the total energy to obtain the optimal coordinates of each keypoint in each frame.

5. The method according to claim 4, characterized in that, Step S3 specifically includes the following steps: Step S3.1: Based on the joint configuration of the bionic robot, a mapping table is pre-established, and a corresponding human key point triplet is assigned to each robot joint; Step S3.2: In the key point sequence obtained in step S2, only the key points involved in the mapping table are retained, and other irrelevant points are ignored; Step S3.3: For each robot joint, take the corresponding human joint center key point, calculate the vector from the key point to the near key point and the vector to the far key point, and obtain the human joint angle value corresponding to the joint by the vector dot product and the inverse cosine function. Step S3.4: Divide the difference between the human joint angles in the current frame and the previous frame by the inter-frame time interval to obtain the angular velocity value corresponding to the joint. The inter-frame time interval is determined by the reciprocal of the video frame rate. Step S3.5: When key points are missing in the triplet, the angle value of the previous frame is used as the estimate of the current frame first. If the same joint is missing for more than 3 consecutive frames, the angle value of the symmetrical joint is multiplied by the direction coefficient for mirror replacement. If the symmetrical joint is also unavailable, the angle of the current frame is set to the historical average of the joint or the robot's neutral position angle. Finally, the human joint angle sequence and angular velocity sequence corresponding one-to-one with the robot joints are output.

6. A video-driven bionic robot dance motion generation system, used to implement the video-driven bionic robot dance motion generation method as described in any one of claims 1-5, characterized in that, include: The video acquisition and preprocessing module is used to acquire dance videos, decode the videos into continuous frame images and unify the frame rate, perform Gaussian filtering to remove noise from each frame image, and scale the image size to 640×480 pixels. The adaptive key point extraction and filtering module is used to adaptively extract the motion trajectory of human skeleton key points from a standardized video frame sequence based on motion consistency constraints and perform temporal filtering to obtain a key point coordinate sequence. The human joint motion calculation module is used to determine the human key points corresponding to each robot joint according to the robot joint mapping table, extract the key points and calculate their included angle as the human joint angle, and then calculate the angular velocity through inter-frame difference, and output the human joint angle sequence and angular velocity sequence that correspond one-to-one with the robot joint. A robot kinematics parameter library for storing joint definitions, mechanical limit angles, neutral position angles, and adjacent joint coupling coefficient matrices for biomimetic robots; The adaptive nonlinear mapping module is used to establish an adaptive nonlinear mapping function from human joints to robot joints and to calculate the target angle sequence of each robot joint. The motion smoothing and output module is used to perform sliding window averaging filtering on the robot's joint angle sequence and output the final executable dance motion data.

7. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute the video-driven bionic robot dance motion generation method as described in any one of claims 1-5.

Images