A follow-the-dancer robot system and a pose mapping method

Abstract

This invention discloses a follow-along dancing robot system and its posture mapping method. The method constructs user limb vectors and obtains the robot's limb posture mapping relationship based on the projection angles of these vectors in the dynamic coordinate system of the torso. This invention employs cross-morphological lightweight vector computation, completely eliminating the singularity problem inherent in traditional inverse kinematics solutions and thoroughly resolving the technical pain point of "stiff and rigid hands" in humanoid robot dance. The robot can achieve comprehensive motion replication from the human shoulder, elbow, wrist, and hand, accurately reproducing the rhythm of a dancer's fingertips and wrist movements, thus presenting a refined and expressive performance comparable to that of a human dancer.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a dance-following robot system and a posture mapping method. Background Technology

[0002] With the rapid development of robotics and artificial intelligence, robots are increasingly being used in industrial manufacturing, logistics, home services, education, and entertainment. Under the embodied intelligence paradigm, robots are required not only to possess the intelligent cognitive abilities of a "brain," but also to have a "body" to achieve real-time, smooth, and natural interaction with the physical world and humans. Dance, as a complex and continuous movement requiring high levels of physical coordination, rhythm, and emotional expression, is a core indicator for evaluating the level of whole-body motion control and human-computer interaction capabilities of humanoid robots. A robot capable of following a dancer's movements must be able to accurately and in real-time capture and mimic the dancer's actions.

[0003] The human arm possesses a high degree of freedom and flexibility, while commercial humanoid robots and other hardware devices differ fundamentally from the human body in terms of the number of joints, link lengths, and movement limits. When existing robots mimic human dance movements, they generally suffer from distortion due to cross-morphological kinematic mapping. Summary of the Invention

[0004] When robots mimic human dance movements, distortion caused by cross-morphological kinematic mapping arises from two main issues. First, the lack of hand degrees of freedom means that current technologies often neglect the fine movements of the wrist and hand, making the dance appear stiff and rigid. Second, existing pose mapping methods often employ simple 2D joint direct coordinate mapping or brute-force iterative solutions based on inverse kinematics. The former leads to the loss of depth information and severe distortion of movements, while the latter is prone to getting stuck in kinematic singularities when faced with the free movements of the human body, resulting in non-convergence of calculations, robot joint jamming, or unpredictable violent flips.

[0005] To address the aforementioned shortcomings, this invention provides a posture mapping method for a dancing robot, the technical solution of which is as follows:

[0006] A posture mapping method for a dancing robot, characterized by comprising:

[0007] Image acquisition;

[0008] A user digital twin skeleton is constructed based on the image, and the user digital twin skeleton includes human feature points;

[0009] A user baseline is constructed based on the feature points, and a dynamic coordinate system of the torso is established based on the user baseline to obtain the robot torso posture mapping relationship.

[0010] A user limb vector is constructed based on the feature points, and the robot limb posture mapping relationship is obtained based on the projection angle of the user limb vector in the dynamic coordinate system of the torso.

[0011] The robot performs posture mapping based on the torso posture mapping relationship and the limb posture mapping relationship.

[0012] Furthermore, the user benchmark includes:

[0013] A shoulder vector constructed from feature points of the left and right shoulders;

[0014] The hip vector constructed from the feature points of the left hip and the feature points of the right hip;

[0015] The spine vector is constructed from the midpoint of the shoulder and the midpoint of the hip.

[0016] Furthermore, establishing the dynamic coordinate system of the torso includes:

[0017] The origin of the dynamic coordinate system of the torso is the orthogonal intersection point of the shoulder vector and the spine vector.

[0018] The normalized shoulder vector and spine vector are used as the horizontal and vertical axes of the dynamic coordinate system of the torso, respectively.

[0019] By using the cross product of the shoulder vector and the spine vector and normalizing it, a normal axis perpendicular to the front of the chest is generated.

[0020] Furthermore, the torso pose mapping relationship is [θ, Ψ], where:

[0021] θ is the trunk rotation angle, reflecting the degree of trunk twisting;

[0022] Ψ is the trunk tilt angle, reflecting the degree of trunk tilt.

[0023] Furthermore, the limb vectors include upper arm vectors, forearm vectors, and hand vectors; the limb posture mapping relationship includes [α1, α2, β1, β2, r], where:

[0024] α1 is the shoulder roll angle, which reflects the degree to which the arm is raised to the sides of the body;

[0025] α2 is the shoulder pitch angle, which reflects the degree of arm swing back and forth;

[0026] β1 is the elbow roll angle, which reflects the degree of arm flexion;

[0027] β2 is the elbow yaw angle, reflecting the degree of internal and external rotation of the arm;

[0028] r is the wrist yaw angle, which represents the lateral orientation of the palm.

[0029] Furthermore, the attitude mapping method further includes:

[0030] The robot's hand engagement degree H is obtained based on limb feature points, and H is calculated using the following formula:

[0031] H = D / L

[0032] Where D is the three-dimensional Euclidean distance between the thumb tip feature point and the index finger tip feature point, and L is the three-dimensional Euclidean distance from the wrist feature point to the index finger root feature point.

[0033] Furthermore, the attitude mapping method further includes:

[0034] The attitude mapping relationship is optimized to generate an optimized attitude mapping relationship; the optimization process includes at least one of direction correction, safety range correction, straight arm correction, filtering, and smoothing.

[0035] Another aspect of the present invention provides a posture mapping system for a dancing robot, the technical solution of which is as follows:

[0036] A posture mapping system for a dancing robot, characterized in that it includes:

[0037] The image processing module is used to capture the user's motion video stream and obtain the user's human body feature points;

[0038] The morphology calculation module is used to construct user limb vectors, user reference, and torso dynamic coordinate system based on the feature points, and to obtain posture mapping relationships; the user limb vectors include upper arm vectors, forearm vectors, and hand vectors; the morphology calculation module includes: a torso calculation unit, used to obtain torso posture mapping relationships based on the user reference; an upper arm calculation unit, used to obtain upper arm posture mapping relationships based on upper arm vectors; a forearm calculation unit, used to obtain forearm posture mapping relationships based on forearm vectors; and a hand calculation unit, used to obtain hand posture mapping relationships based on hand vectors.

[0039] The optimization module is used to optimize the attitude mapping relationship and generate an optimized attitude mapping relationship.

[0040] The communication module is used to send the optimized posture mapping relationship to the robot.

[0041] Another aspect of the present invention provides a robot system for following a dance, the technical solution of which is as follows:

[0042] A dancing robot system, including

[0043] The client includes the aforementioned attitude mapping system and image acquisition module.

[0044] The robot is used to receive and execute the posture mapping relationship sent by the client.

[0045] In another aspect, the present invention provides a computer-readable storage medium, the technical solution of which is as follows:

[0046] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described posture mapping methods for a dancing robot.

[0047] In another aspect, the present invention provides a user digital twin skeleton, the technical solution of which is as follows:

[0048] A user digital twin skeleton includes multiple human feature points and their three-dimensional coordinates in a world coordinate system, wherein the multiple human feature points include:

[0049] The feature points are: head feature points, shoulder feature points, elbow feature points, wrist feature points, hip feature points, little finger base feature points, index finger base feature points, index finger tip feature points, and thumb tip feature points. These feature points are identified using a neural network based on the user's video stream.

[0050] In another aspect, the present invention provides a method for establishing a user digital twin skeleton, the technical solution of which is as follows:

[0051] A method for establishing a user's digital twin skeleton includes:

[0052] Capture user images, and mirror each image frame to obtain mirrored frame images;

[0053] Construct a world coordinate system in the mirrored flipped frame image;

[0054] In the world coordinate system, a three-dimensional network model of the human body is generated from the image using a neural network.

[0055] In the aforementioned three-dimensional human body network model, neural networks are used to extract human body feature points and their three-dimensional coordinates to form a digital twin skeleton.

[0056] In another aspect, the present invention provides a user reference for mapping robot dance movements, the technical solution of which is as follows:

[0057] A user baseline for mapping robot dance movements is an I-shaped structure composed of the user's shoulder vector, hip vector, and spine vector.

[0058] In another aspect, the present invention also provides a method for establishing user baselines, the technical solution of which is as follows:

[0059] A method for establishing user benchmarks includes:

[0060] Obtain the left and right shoulder feature points from the user's 3D digital twin skeleton and construct a shoulder vector;

[0061] Obtain the feature points of the left and right hips, and construct the hip vector;

[0062] Obtain the midpoints of the shoulder and hip, and construct the spine vector;

[0063] The user reference is formed by the shoulder vector, hip vector, and spine vector.

[0064] In another aspect, the present invention provides a method for correcting attitude mapping relationships, the technical solution of which is as follows:

[0065] A method for correcting attitude mapping relationships includes:

[0066] Based on the symmetry of the robot's left and right structures, the orientation mapping relationship is corrected; and / or,

[0067] Based on the range of motion of the robot joints, the posture mapping relationship is corrected within a safe range; and / or,

[0068] Based on the degree of elbow straightness, the elbow yaw angle in the attitude mapping relationship is corrected for straightness.

[0069] The attitude mapping relationship is F = [θ, Ψ, α1, α2, β1, β2, r, H].

[0070] In another aspect, the present invention also provides an attitude mapping filtering method, the technical solution of which is as follows:

[0071] An attitude mapping filtering method, comprising:

[0072] Obtain the pose mapping parameters of adjacent frames;

[0073] The dynamic cutoff frequency f is calculated based on the attitude mapping parameters of adjacent frames. c :

[0074]

[0075] in,

[0076]

[0077] f min The basic low-pass cutoff frequency is set to 0.8–1 Hz;

[0078] δ is the velocity response coefficient, set to 0.4–0.5;

[0079] x includes θ, Ψ, α1, α2, β1, β2, r, H;

[0080] Compare dynamic cutoff frequencies f c With threshold A;

[0081] Based on the comparison results, the pose mapping relationship of the current frame or the previous frame is selected as the pose mapping relationship of the current frame.

[0082] In another aspect, the present invention also provides a filter, the technical solution of which is as follows:

[0083] An attitude mapping filter includes multiple filtering units, wherein the filtering unit is at least one of a torso rotation angle filtering unit, a torso tilt angle filtering unit, a shoulder roll angle filtering unit, a shoulder pitch angle filtering unit, an elbow roll angle filtering unit, an elbow yaw angle filtering unit, a wrist yaw angle filtering unit, and a hand yaw angle filtering unit; the filtering unit adopts a One-Euro filter.

[0084] The posture mapping system, method, and system for a dancing robot of the present invention have the following significant advantages compared with the prior art:

[0085] Extremely high motion fidelity and expressiveness: This invention constructs a six-degree-of-freedom spatial posture mapping of the shoulder, elbow, wrist, and palm, and proposes a dimensionless forward mapping algorithm based on pure spatial vector geometry. Employing cross-morphological lightweight vector solutions, it completely eliminates the singularity problem inherent in traditional inverse kinematics solutions and thoroughly resolves the technical pain point of "stiff and rigid hands" in humanoid robot dance. The robot can achieve comprehensive motion replication from the human shoulder, elbow, wrist, and hand, accurately reproducing the rhythm of a dancer's fingertips and wrist movements, thus presenting a refined and expressive performance comparable to that of a human dancer.

[0086] Completely free from the constraints of wearing devices and achieve natural interaction without wearing anything: Thanks to efficient full-body topological reasoning and cross-morphological lightweight vector solution, users do not need to wear any expensive and cumbersome motion capture equipment. They can start interacting simply by standing in front of a regular monocular or binocular camera, which greatly reduces the barrier to entry for using the robot.

[0087] Dual improvements in anti-interference capability and hardware lifespan: This invention deeply integrates a dynamically adaptive cutoff frequency filter into the robot's full-body joint spatial control model. This mechanism can automatically determine the dynamic and static states of the dancer's movements within microseconds, perfectly eliminating high-frequency mechanical oscillations caused by monocular visual noise, and greatly reducing the risk of robot motor overheating and burnout. At the same time, combined with the symmetrical truncation of the robot's end structure and joint limit truncation, the absolute stability and safety of the robot hardware are ensured. Attached Figure Description

[0088] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0089] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0090] Figure 1 This is a schematic diagram of the structure of the dancing robot system of the present invention.

[0091] Figure 2 This is a schematic diagram of the client structure of the dancing robot system of the present invention.

[0092] Figure 3 This is a schematic diagram of the user digital twin skeleton of the present invention.

[0093] Figure 4 This is a schematic diagram of the two degrees of freedom of the torso posture according to the present invention.

[0094] Figure 5 This is a schematic diagram of the six degrees of freedom of limb posture according to the present invention.

[0095] Figure 6 This is a schematic diagram of a filter unit structure according to the present invention.

[0096] Figure 7 This is a schematic diagram of another filtering unit structure according to the present invention.

[0097] Figure 8 This is a flowchart of the posture mapping method for the dancing robot of the present invention.

[0098] Figure 9 Flowchart of the method for building a digital twin skeleton for users.

[0099] Figure 10 This is a flowchart of the posture mapping relationship acquisition method of the present invention.

[0100] Figure 11 This is a flowchart of the attitude mapping relationship optimization method of the present invention.

[0101] 100 - Client, 200 - Robot, 300 - User, 300' - Digital twin skeleton;

[0102] 11-Image acquisition module; 12-Processor;

[0103] 121-Image processing module, 122-morphological calculation module, 123-Optimization module, 124-Communication module;

[0104] 1221 - Trunk calculation unit, 1222 - Upper arm calculation unit, 1223 - Forearm calculation unit, 1224 - Hand calculation unit;

[0105] 1231 - Correction unit, 1232 - Filtering unit;

[0106] R - Robot coordinate system, O - World coordinate system, Q - Torso dynamic coordinate system, S - Local coordinate system;

[0107] θ - trunk rotation angle, Ψ - trunk tilt angle, α1 - shoulder roll angle, α2 - shoulder pitch angle, β1 - elbow roll angle, β2 - elbow yaw angle, r - wrist yaw angle, H - hand opening angle.

[0108] F1 - First Attitude Mapping Relationship

[0109] F 11 - Attitude mapping relationship after orientation correction

[0110] F2 - Second attitude mapping relationship (attitude mapping relationship after safety range correction)

[0111] F 21 - Posture mapping relationship after straight arm correction

[0112] F3 - Third Attitude Mapping Relationship (Filtered Attitude Mapping Relationship)

[0113] F4 - Fourth Attitude Mapping Relationship (Smoothed Attitude Mapping Relationship) Detailed Implementation

[0114] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0115] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0116] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0117] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," "head," and "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. In addition, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0118] This embodiment provides a follow-dancing robot system, referring to... Figures 1-2 The robot-following dance system includes a client 100, a robot 200, and a user 300. The client 100 and robot 200 are connected wirelessly. The client 100 is a PC and includes an image acquisition module 11 and a processor 12. The image acquisition module 11 is preferably a monocular RGB camera used to acquire images, specifically a video stream of the user's movements; in other embodiments, it can also be other image acquisition devices capable of acquiring video streams of user movements, such as a binocular RGB camera. The processor 12 processes the video stream acquired by the image acquisition module 11 and generates posture mapping instructions for the robot to follow the user 300 based on the video stream, then sends these instructions to the robot 200.

[0119] The processor 12 includes an image processing module 121, a morphology calculation module 122, an optimization module 123, and a communication module 124. The image processing module 121, morphology calculation module 122, optimization module 123, and communication module 124 are sequentially and communicatively connected. The image acquisition module 11 is connected to the image processing module 121 and is used to transmit the acquired video stream to the image processing module 121. The communication module 124 is wirelessly connected to the robot 200 and is used to transmit the posture mapping relationship to the robot 200. The robot 200 is a NAO robot used to execute posture mapping instructions corresponding to the posture mapping relationship, thereby enabling the robot 200 to follow the user's dance movements.

[0120] In this embodiment, the image processing module 121 is used to construct a user digital twin skeleton based on the video stream acquired by the image acquisition module 11. Specifically:

[0121] The image processing module 121 uses a neural network, such as the MediaPipe Pose model, to process the RGB video stream from the image acquisition module 11 in real time. It reads video frame images from a monocular RGB camera or a binocular RGB camera in a non-blocking manner and performs mirror flipping to conform to the user's "mirror following" intuition. That is, when the user raises his left hand, the robot raises the corresponding right hand, forming a mirror interaction. Afterward, the mirrored frame images in RGB format are input into the neural network for inference.

[0122] Image processing module 121 uses a neural network to perform pose estimation on the RGB mirrored frame image, obtaining the user's three-dimensional key points. (Refer to...) Figure 3 The image processing module 121 constructs a three-dimensional network model of the user 300 in the world coordinate system based on the three-dimensional key points of the human body, and extracts the feature points required for the pose mapping relationship from the three-dimensional network model, and sets their coordinates P in the world coordinate system O. i =(x i , y i , z i The features are labeled and stored in the form of ), where i is the feature point number, and the number of feature points is less than the number of key points. (Refer to...) Figure 3 The user's body feature points include head feature point P0, left shoulder feature point P1, right shoulder feature point P2, left elbow feature point P5, right elbow feature point P6, left wrist feature point P7, right wrist feature point P8, left hip feature point P3, right hip feature point P4, left little finger base feature point P9, and right little finger base feature point P1. 10 Feature point P at the base of the left index finger 11 Feature point P at the base of the right index finger 12 Feature point P of the left index fingertip 13 Feature point P of the right index fingertip 14 Feature point P of the left thumb tip 15 Feature point P of the right thumb tip 16 And so on. Where i is an odd number representing the left side of the human body, i is an even number representing the right side of the human body, and i is an integer greater than or equal to 16.

[0123] The aforementioned human feature points and their coordinates in the world coordinate system together constitute the user's digital twin skeleton. After the image processing module 121 obtains the user's digital twin skeleton using a neural network, it transmits the user's digital twin skeleton to the morphology calculation module 122.

[0124] Thanks to efficient feature point extraction, users do not need to wear any expensive and cumbersome motion capture equipment; they only need to stand in front of a regular monocular or binocular RGB camera to begin interaction, greatly lowering the barrier to entry for using the robot. This invention relies solely on a regular monocular RGB camera, achieving high-frequency extraction of at least 30 frames per second of three-dimensional coordinates of full-body human feature points on ordinary computing devices.

[0125] Reference Figures 3-5 The morphological calculation module 122 is used to parse the first pose mapping relationship F1=[θ, Ψ, α1, α2, β1, β2, r, H] of the robot following the user's pose based on feature points. This pose mapping relationship includes two degrees of freedom [θ, Ψ] of the torso and six degrees of freedom [α1, α2, β1, β2, r, H] of the limbs. Wherein:

[0126] θ is the trunk rotation angle, used to reflect the degree of trunk twisting;

[0127] Ψ is the trunk tilt angle, used to reflect the degree of trunk tilt;

[0128] α1 is the shoulder roll angle, which reflects the degree to which the arm is raised to the sides of the body;

[0129] α2 is the shoulder pitch angle, used to reflect the degree of arm swing back and forth;

[0130] β1 is the elbow roll angle, used to reflect the degree of arm flexion;

[0131] β2 is the elbow yaw angle, used to reflect the degree of internal and external rotation of the arm;

[0132] r is the wrist yaw angle, which represents the lateral orientation of the palm and is related to the user's wrist rotation angle;

[0133] H represents the hand opening degree, which reflects the degree of grip of the palm.

[0134] The morphology calculation module 122 includes a torso calculation unit 1221, an upper arm calculation unit 1222, a forearm calculation unit 1223, and a hand calculation unit 1224. Each unit of the morphology calculation module 122 is interconnected with the image processing module 121 and is used to receive the user digital twin skeleton from the image processing module 121. The user digital twin skeleton includes multiple human feature points and their coordinates P in the world coordinate system O. i =(x i , y i , z i). Each unit of the posture solving module 122 is also interconnected with the optimization module 123, and is configured to transmit the solved posture mapping relationship to the optimization module 123. Wherein, the torso solving unit 1221 is configured to acquire the posture mapping relationship [θ, Ψ] of the robot torso according to torso-related feature points such as the shoulders and hips; the upper arm solving unit 1222 is configured to acquire the posture mapping relationship [α1, α2] of the robot upper arm according to upper arm-related feature points such as the shoulder and elbow; the forearm solving unit 1223 is configured to acquire the posture mapping relationship [β1, β2] of the robot forearm according to forearm-related feature points such as the elbow and wrist; the hand solving unit 1224 is configured to acquire the posture mapping relationship [r, H] of the robot hand according to hand-related feature points such as the wrist feature point and the fingers.

[0135] In this embodiment, the torso solving unit 1221 is communicatively connected to the optimization module 123, and is configured to acquire the posture mapping relationship [θ, Ψ] of the torso posture.

[0136] First, the torso solving unit 1221 establishes the "I"-shaped reference of the user according to the torso-related feature points such as the shoulder feature points and the hip feature points. Refer to Figure 3 , Figure 4 , the torso solving unit 1221 takes the "I"-shaped mark composed of the connecting line of the two shoulder coordinates, the connecting line of the two hip coordinates, and the spine line at the midpoint of the shoulders and hips of the digital twin skeleton 300' of the user 300 as the reference, and this reference will not change due to the inclination or rotation of the user's body; during the dancing process, the orientation and inclination of the user 300 will change continuously, so that the reference is inclined or rotated. Refer to Figure 3 , specifically:

[0137] Construct a shoulder vector according to the coordinates of the left and right shoulder feature points P1 and P2 in the world coordinate system ;

[0138] Construct a hip vector according to the coordinates of the left and right hip bone feature points P3 and P4 in the world coordinate system ;

[0139] Construct a spine vector according to the midpoint of the left and right shoulder feature points P1 and P2 and the midpoint of the left and right hip bone feature points P3 and P4 = ; where P m1 is the midpoint of P1 and P2, and P m2 is the midpoint of P3 and P4.

[0140] The above shoulder vector , hip vector , and spine vector jointly form the "I"-shaped reference of the user.

[0141] Next, refer toFigure 4 , and establish the user's torso dynamic coordinate system based on the "I"-shaped reference. Specifically: taking the shoulder vector , the spine vector as the orthogonal intersection point which is the origin Q of the torso dynamic coordinate system, taking the normalized shoulder vector and the spine vector as the X' axis and Y' axis of the torso dynamic coordinate system respectively; performing cross product (outer product) on the shoulder vector and the spine vector and normalizing the result to generate the Z' axis perpendicular to the front chest: .

[0142] Thus, the torso dynamic coordinate system Q of the user 300 is obtained, and the coordinate P of each feature point in the torso dynamic coordinate system is acquired i =(x' i , y' i , z' i ). The torso dynamic coordinate system serves as a reference basis for subsequent upper limb posture calculation.

[0143] In this embodiment, the robot torso posture mapping relationship [θ, Ψ] is obtained according to the projection from the torso dynamic coordinate system Q to the world coordinate system O. Specifically: the included angle between the projection of the x'-axis base vector x' of the torso dynamic coordinate system on the xz-axis plane of the world coordinate system and the x-axis base vector x of the world coordinate system is defined as the torso rotation angle θ, and the included angle is solved by using the scalar product formula, so as to obtain the torso rotation angle θ; the included angle between the y'-axis base vector y' of the torso dynamic coordinate system and the y-axis base vector y of the world coordinate system is defined as the torso inclination angle Ψ, and the included angle is solved by using the scalar product formula, so as to obtain the torso inclination angle Ψ.

[0144] In this embodiment, the robot coordinate system R is selected to be consistent with the world coordinate system O, or may be only consistent with the world coordinate system in direction but inconsistent in position. Thus, the rotation relationship F between the robot coordinate system R and the world coordinate system O R0 is an identity matrix, and the rotation relationship F between the world coordinate system O and the torso dynamic coordinate system Q 0Q =[θ, Ψ]; therefore, the rotation relationship between the robot coordinate system R and the torso dynamic coordinate system (that is, the mapping relationship from the robot torso posture to the user torso posture) is F 0Q =[θ, Ψ].

[0145] It should be noted that the pose mapping relationship of the robot's torso is not limited to the methods described above. It can also be solved using a seven-parameter Bursa model or Rodrigues matrix model based on multiple feature points, or using a direction cosine matrix model based on the basis vectors of two coordinate systems. Since the robot coordinate system R and the world coordinate system O have the same direction (i.e., the rotation relationship is an identity matrix), and the image processing module 121 has processed the image into a "mirror-following" image, the pose mapping relationship can be determined based on the robot's current torso pose and rotation relationship F. OQ Obtain the torso pose mapping relationship from the robot to the user, and the robot executes the torso pose mapping relationship F. RQ The corresponding mapping command can adjust the user's torso posture to match the user's mirror image.

[0146] The torso calculation unit 1221 transmits the calculated torso posture mapping relationship [θ, Ψ] to the optimization module 123.

[0147] In this embodiment, the upper arm calculation unit 1222 is communicatively connected to the optimization module 123, and is used to obtain the posture mapping relationship [α1, α2] of the upper arm posture. (Refer to...) Figure 5 Specifically:

[0148] First, based on the coordinates of the left shoulder feature point P1 and the left elbow feature point P5 in the torso dynamic coordinate system Q, the left upper arm vector is constructed. A similar method was used to construct the right upper arm vector.

[0149] The angle between the projection of the upper arm vector onto the plane formed by the X' and Y' axes of the torso dynamic coordinate system and the X' axis is defined as the shoulder roll angle α1. Based on the projection of the upper arm vector onto this plane and onto the X' axis, the shoulder roll angle α1 on the corresponding side of the robot is solved using the dot product formula.

[0150] The angle between the projection of the upper arm vector onto the plane formed by the Y' and Z' axes of the torso dynamic coordinate system and the Y' axis is defined as the shoulder pitch angle α2. Based on the projection of the upper arm vector onto this plane and onto the Y' axis, the shoulder pitch angle α2 on the corresponding side of the robot is solved using the dot product formula.

[0151] The left and right upper arm posture mapping relationships calculated by the upper arm calculation unit 1222 together constitute the upper arm posture mapping relationship [α1, α2], which is then transmitted to the optimization module 123.

[0152] In this embodiment, the forearm calculation unit 1223 is communicatively connected to the optimization module 123 and is used to obtain the posture mapping relationship [β1, β2] of the forearm posture. (Refer to...) Figure 5 The details are as follows:

[0153] Based on the coordinates of the left elbow feature point P5 and the left wrist feature point P7 in the torso dynamic coordinate system Q, a left forearm vector is constructed. A similar method was used to construct the right forearm vector.

[0154] The angle between the upper arm vector and the lower arm vector is defined as the elbow roll angle β1. The angle is solved using the dot product formula to obtain the elbow roll angle β1 on the corresponding side of the robot.

[0155] The normal vector of the "arm bending plane" subtended by the upper arm vector and the forearm vector. Projection of the torso's horizontal axis x' onto the upper arm plane The included angle between the two sides is defined as the elbow yaw angle. This parameter is a key parameter reflecting the internal and external rotation of the arm. The included angle is solved using the dot product formula, thereby obtaining the elbow yaw angle β2 on the corresponding side of the robot.

[0156] The left forearm posture mapping relationship and the right forearm posture mapping relationship calculated by the forearm calculation unit 1223 together constitute the forearm mapping relationship [β1, β2], which is then transmitted by the forearm calculation unit 1223 to the optimization module 123.

[0157] In this embodiment, the hand calculation unit 1224 is communicatively connected to the optimization module 123 and is used to obtain the hand posture mapping relationship [r, H]. (Refer to...) Figure 5 The details are as follows:

[0158] Based on the characteristic point P at the base of the left index finger 11 Construct the left palm vector by taking the coordinates of the feature point P9 at the base of the left little finger in the dynamic coordinate system Q of the torso. A similar method is used to construct the right palm vector. A local coordinate system S is constructed for the corresponding side, with the forearm vector direction as the Z'' axis and the inner arm direction as the Y'' axis.

[0159] The angle between the projection of the palm vector onto the X''Y'' plane of the local coordinate system and the Y'' axis is defined as the wrist yaw angle r. This angle is solved using the dot product formula to obtain the wrist yaw angle r on the corresponding side. The wrist yaw angle r is related to the user's wrist rotation angle.

[0160] Based on the coordinates of the index fingertip and thumbtip feature points in the torso dynamic coordinate system Q, the three-dimensional Euclidean distance D between the thumbtip and index fingertip is calculated, such as D. 左 =|P 13 -P 15 |; Based on the coordinates of the wrist feature point and the index finger base feature point in the torso dynamic coordinate system Q, solve for the three-dimensional Euclidean distance L from the wrist to the index finger base, such as L 左 =|L 13-L7|; The corresponding hand flexion H is defined as H=D / L; The corresponding hand flexion H can be obtained by calculation.

[0161] The left hand posture mapping relationship and the right hand posture mapping relationship calculated by the hand calculation unit 1224 together constitute the hand posture mapping relationship [r, H], which is then transmitted by the hand calculation unit 1224 to the optimization module 123.

[0162] The set of attitude mapping relationships calculated by the morphology calculation module 122 constitutes the first attitude mapping relationship F1=[θ,Ψ,α1,α2,β1,β2,r,H], including the mapping parameters θ,Ψ,α1,α2,β1,β2,r,H.

[0163] This invention completely solves the technical pain point of "stiff and rigid hands" in humanoid robot dance by introducing wrist rotation and hand opening / closing degrees of freedom in addition to the four degrees of freedom of the shoulder and elbow. The robot can accurately reproduce the fingertip rhythm and wrist flipping movements of a dancer, thus presenting a refined and expressive performance comparable to that of a human dancer. Thanks to cross-morphological lightweight vector computation, this invention fully adopts forward geometric calculation based on relative vector projection, thereby naturally avoiding the problems of division by zero and mathematical singularities outside the scope.

[0164] In this embodiment, the optimization module 123 includes a correction unit 1231 and a filtering unit 1232. The correction unit 1231 is used to correct the first attitude mapping relationship F1 and output the corrected attitude mapping relationship. The correction unit 1231 is communicatively connected to the filtering unit 1232, which is used to filter the attitude mapping relationship and output the filtered attitude mapping relationship. The correction unit 1231 has: a direction correction condition for correcting the direction of the attitude mapping relationship; a safety range correction condition for correcting the safety range of the attitude mapping relationship; and a straight arm correction condition for correcting the elbow yaw angle in the attitude mapping relationship.

[0165] The morphological calculation module 122 is connected to the correction unit 1231 and is used to transmit the calculated first attitude mapping relationship F1 to the correction unit 1231; the filtering unit 1232 is connected to the communication module 124 and is used to transmit the filtered attitude mapping relationship to the communication module 124.

[0166] In this embodiment, the correction unit 1231 and the filtering unit 1232 are connected in series. However, the optimization module 123 of the present invention is not limited to this connection method. The correction unit 1231 and the filtering unit 1232 can be selected to be set; the correction unit 1231 and the filtering unit 1232 can also be set in parallel so that they can be used selectively.

[0167] When the correction unit 1231 runs, it first utilizes the symmetry of the robot's left and right mechanical structures, taking α1 and β1 from the first attitude mapping relationship F1 obtained above as inputs, and corrects the orientation of α1 and β1 according to the orientation correction conditions. Then, it updates the first attitude mapping relationship F1 with the corrected α1 and β1 as outputs. The orientation mapping relationship after orientation correction is expressed as F... 11 Specifically, based on the current left and right sides of the upper limb, the directional difference between the human upper limb posture parameters and the robot's upper limb joint coordinate system is corrected. This directional correction is used to perform sign transformation and bias compensation on the direction-sensitive mapping parameters α1 and β1. The directional correction conditions are:

[0168] α 1输出 =R S (α 1输入 )

[0169] β 1输出 =M S (β 1输入 )

[0170] in:

[0171] For upper limb lateral parameters, left upper limb Right upper limb ;

[0172] α1 is the shoulder roll angle;

[0173] β1 For elbow roll angle;

[0174] R S This represents the shoulder roll correction function corresponding to the current upper limb side, when... At that time, R S Adjust α1 to the negative range, preferably to... ,when At that time, R S Adjust α1 to a positive value range, preferably to a positive value range. This achieves sign transformation. Furthermore, it eliminates the original shoulder roll angle. To mitigate the impact on the correction results, the original roll angle α0 of the corresponding shoulder is removed from the correction results to achieve offset compensation.

[0175] M S This represents the elbow roll correction function corresponding to the current upper limb side; when At that time, M S β1 Corrected to the negative range, preferably corrected to ;when At that time, M S β1 Corrected to a positive range, preferably corrected to... This allows for sign transformation. Furthermore, it eliminates the possibility of the original elbow roll angle β0. To mitigate the impact of this, the original roll angle β0 of the corresponding elbow is removed from the corrected result. This achieves bias compensation.

[0176] By correcting the direction as described above, the direction of the human body mirror image movement, the coordinate direction of the human body posture estimation, and the positive and negative directions of the robot's left and right upper limb joints can be kept consistent, avoiding the robot's upper limb movements from being reversed or uncoordinated due to the different direction definitions of the human body coordinate system and the robot's joint coordinate system.

[0177] The correction unit 1231 then, based on the movable range of the robot's limbs, modulates the first pose mapping relationship F1 obtained by the morphology calculation module or the pose mapping relationship F after orientation correction. 11 Perform a safety range adjustment. The conditions for safety range adjustment are as follows:

[0178] α1' = k1α1

[0179] α2' = k2α2

[0180] β1' = k3β1

[0181] β2' = k4β2

[0182] r' = k5r

[0183] H' = k6H

[0184] in:

[0185] k1 is the correction coefficient for the shoulder roll angle α1, which is determined based on the ratio between the maximum range of motion of the user's shoulder roll angle and the maximum range of motion of the robot's shoulder roll angle.

[0186] k2 is the correction coefficient for the shoulder pitch angle α2, which is determined based on the ratio between the maximum range of motion of the user's shoulder pitch angle and the maximum range of motion of the robot's shoulder pitch angle.

[0187] k3 is the correction coefficient for the elbow roll angle β1, which is determined based on the ratio between the maximum range of motion of the user's elbow roll angle and the maximum range of motion of the robot's elbow roll angle.

[0188] k4 is the correction coefficient for the elbow yaw angle β2, which is determined based on the ratio between the maximum range of the user's elbow yaw angle and the maximum range of the robot's elbow yaw angle.

[0189] k5 is the correction coefficient for the wrist yaw angle r, which is determined based on the ratio between the maximum range of the user's wrist yaw angle and the maximum range of the robot's wrist yaw angle.

[0190] k6 is the correction coefficient for the hand opening degree H, which is determined based on the ratio between the maximum range of motion of the user's hand opening and closing and the maximum range of motion of the robot's hand opening and closing.

[0191] The correction unit 1231 calculates α1', α2', β1', β2', r', and H' based on the aforementioned safety range correction conditions, thereby obtaining the second attitude mapping relationship F2 = [θ, Ψ, α1', α2', β1', β2', r', H'] for the current frame. The attitude mapping relationship after safety range correction is represented as the second attitude mapping relationship F2.

[0192] The server in this invention has orientation correction and safety range correction functions, so that even if the user makes extreme or irregular limb movements, the system can ensure the absolute stability and safety of the robot hardware.

[0193] Furthermore, the correction unit 1231 also performs straight-arm correction on the posture mapping relationship based on the degree of elbow joint opening (i.e., the degree of straight arm). The mapping relationship can be a first posture mapping relationship F1, a second posture mapping relationship F2, or a posture mapping relationship F after direction correction. 11 Taking the first attitude mapping relationship F1 as an example, the conditions for straight arm correction are as follows:

[0194]

[0195] Where W is the dynamic weight decay coefficient.

[0196] The correction unit 1231 calculates the dynamic weight attenuation coefficient W based on the above straight arm correction conditions and compares it with the threshold B, where B is a constant close to 1. When W = B, β1 in the first pose mapping relationship F1 of the current frame is assigned the value 0; otherwise, it remains unchanged. The pose mapping relationship after straight arm correction is represented as F... 21 .

[0197] When the elbow roll angle β1 is greater than 2.5 radians (i.e., approaching a straight arm state), the cross product of the upper arm vector and the lower arm vector approaches zero, leading to mathematical singularities in the normal plane calculation. This can easily cause the motors to violently reverse and lock up during robot following. This embodiment utilizes a dynamic weight attenuation coefficient for straight arm control, effectively avoiding the aforementioned situation.

[0198] Furthermore, in this embodiment, there are no restrictions on the order and options of direction correction, safety range correction, and straight arm correction; their order and options can be freely adjusted according to the actual situation.

[0199] In this embodiment, the filtering unit 1232 receives the attitude mapping relationship of each frame delivered by the correction unit 1231. This attitude mapping relationship can be represented as the attitude mapping relationship after direction correction, as F. 11 The second attitude mapping relationship F2 or the attitude mapping relationship F after straight arm correction. 21 The image is filtered to eliminate high-frequency jitter generated by the morphology calculation module 122 during the processing of multiple frames. (Refer to...) Figure 6 , Figure 7 The filtering unit includes a θ filter, a Ψ filter, an α1 filter, an α2 filter, a β1 filter, a β2 filter, an r filter, and an H filter. Each filter performs filtering on its corresponding parameter. These filters can be configured in series or in parallel: in series, if any parameter in the attitude mapping relationship meets the jitter elimination condition, full-parameter filtering is triggered; in parallel, filtering is only triggered on the corresponding parameter. One-Euro filters are used. Taking the filtering of the second attitude mapping relationship F2 as an example, the specific steps are as follows:

[0200]

[0201]

[0202] in:

[0203] f c This is the dynamic cutoff frequency;

[0204] f min The basic low-pass cutoff frequency is set to 0.8–1 Hz;

[0205] δ is the velocity response coefficient, set to 0.4–0.5;

[0206] Let x be the instantaneous rate of change of each mapping parameter x in the second attitude mapping relationship F2, where x includes θ, Ψ, α1', α2', β1', β2', r', and H'.

[0207] n is the number of frames (a positive integer), x n The mapping parameter x represents the second pose mapping relationship in the nth frame. n+1 The mapping parameters represent the second pose mapping relationship in the (n+1)th frame.

[0208] Filtering unit 1232 calculates the dynamic cutoff frequency f of the current frame based on the above jitter cancellation conditions. c And compare it with a threshold A, A based on f min The setting is to be close to f. min constants:

[0209] When f cWhen > A, the second pose mapping relationship of the current frame is output as the third pose mapping relationship F3 of the current frame;

[0210] When f c When A ≤ A, the second attitude mapping relationship of the previous frame is output as the third attitude mapping relationship F3 of the current frame. The filtered attitude mapping relationship is represented as the third attitude mapping relationship F3.

[0211] When dancers perform vigorous movements such as rapid arm swings Surge, cutoff frequency f c Rapidly increasing speed, the filter unit 1232 responds with almost zero delay; when the dancer remains still, the velocity approaches 0, f c The frequency is reduced back to 0.8Hz for strong low-pass filtering, which instantly blocks out the tiny noise from the camera and ensures that there is no visible mechanical vibration in the robot's limbs.

[0212] This invention deeply integrates a dynamically adaptive cutoff frequency filter into the robot's full-body joint spatial control model. This mechanism can automatically determine the dynamic and static states of a dancer's movements within microseconds, perfectly eliminating high-frequency mechanical oscillations caused by monocular visual noise, and greatly reducing the risk of robot motor overheating and burning out.

[0213] Furthermore, in this example, the filtering unit 1232 also has a smoothing condition. The filtering unit 1232 uses the smoothing condition to smooth the third pose mapping relationship F3 of each frame image. Specifically: based on the smoothing condition, the parameters of the third pose mapping relationship F3 of the current frame are calculated to obtain the fourth pose mapping relationship F4 = [ , , , , , , , The smoothed attitude mapping is represented as the fourth attitude mapping, F4. The smoothing condition is:

[0214]

[0215] in:

[0216] Smoothing factor ;

[0217] The time difference between adjacent frames. This is the time difference between frames.

[0218] Smoothing processes avoid abrupt changes in angle / pose by weighted fusion of the current frame and the previous frame, making the robot's limb movements continuous and smooth, eliminating inter-frame jitter and abrupt transitions, and making the dance more natural.

[0219] In this embodiment, the optimization module 123 is interconnected with the communication module 124. The optimization module 123 transmits the optimized attitude mapping relationship of each frame to the communication module 124. The attitude mapping relationship can be a first attitude mapping relationship, a second attitude mapping relationship, a third attitude mapping relationship, or a fourth attitude mapping relationship. When it is a first attitude mapping relationship or a second attitude mapping relationship, the correction unit 1231 in the optimization module 123 is interconnected with the communication module 124; when it is a third attitude mapping relationship or a fourth attitude mapping relationship, the filtering unit 1232 in the optimization module 123 is interconnected with the communication module 124.

[0220] The communication module 124 is wirelessly interconnected with the robot 200. The communication module 124 generates attitude mapping instructions from the attitude mapping relationship and sends them to the robot 200 via the TCP protocol.

[0221] After receiving the instruction data packet, the robot 200 does not write it directly to the motor. Instead, it performs a secondary verification on the data to check whether the rigidity of the robot's torso is in an awake state, in order to prevent the robot from falling over due to direct execution of actions without power. After verification, the data is then written to the robot motor for execution.

[0222] This invention also provides a posture mapping method for a dancing robot, referring to... Figure 8 .

[0223] In the initial state, robot 200 maintains its initial standing posture, and client 100 establishes a communication connection with robot 200.

[0224] S1: Image Acquisition

[0225] User 300 waves their arms in front of image acquisition module 11. Image acquisition module 11 uses a monocular RGB camera to capture images, which are the user's motion video stream. The image processing module 121 of client 100 reads the video stream at a rate of about 33 frames per second, obtains multiple frame images, and performs mirror flipping to obtain mirrored frame images. The frame time difference Δt between consecutive frame images is about 0.03 seconds. Mirror flipping can be implemented using the cv2.flip(frame, 1) function in OpenCV to conform to the user's mirror-following intuition.

[0226] S2: Construct a user digital twin skeleton 300' based on the acquired images.

[0227] This embodiment provides a method for constructing a digital twin skeleton, referring to... Figure 3 , Figure 9 The steps are as follows:

[0228] S21 image processing module 121 transmits the mirror frame image of the current frame to the neural network;

[0229] S22 Perform pose estimation on the mirrored frame image of the current frame through a neural network to obtain three-dimensional key points of the human body, wherein the number of the three-dimensional key points of the human body is greater than 50;

[0230] S23 Construct a three-dimensional network model of the user in the world coordinate system according to the connection relationship between the three-dimensional key points of the human body;

[0231] S24 Use a neural network to extract human body feature points from the three-dimensional network model, and mark and store them in the form of coordinates P i =(x i , y i , z i ), where i is a preset feature point number, and the number of the feature points is less than the number of the key points.

[0232] Each feature point corresponding to the mirrored frame image of the current frame and its coordinates in the world coordinate system O constitute the digital twin skeleton 300' of the frame image.

[0233] S3: Parse the robot pose mapping relationship according to the digital twin skeleton

[0234] This embodiment provides a method for acquiring a pose mapping relationship, with reference to Figure 10 , the steps are as follows:

[0235] S31 Establish the I-shaped reference of the user. With reference to Figure 3 , construct a shoulder vector, a hip vector and a spine vector according to the trunk feature points of the current frame to form the I-shaped reference of the user. Specifically:

[0236] Construct the shoulder vector according to the coordinates of the left and right shoulder feature points P1 and P2 in the world coordinate system ;

[0237] Construct the hip vector according to the coordinates of the left and right hip bone feature points P3 and P4 in the world coordinate system ;

[0238] Construct the spine vector according to the midpoint of the left and right shoulder feature points P1 and P2 and the midpoint of the left and right hip bone feature points P3 and P4 = ; where P m1 is the midpoint of P1 and P2, P m2 is the midpoint of P3 and P4.

[0239] The above shoulder vector , hip vector , spine vector collectively constitute the I-shaped reference of the user.

[0240] S32 establishes a dynamic coordinate system for the torso. Based on the "I" shaped reference, the origin and coordinate axes of the dynamic coordinate system for the torso are determined, and the coordinates of each feature point in this coordinate system are obtained. Specifically:

[0241] Shoulder vector Spinal vector The orthogonal intersection point is the origin Q of the torso dynamic coordinate system, and the normalized shoulder vector is... Spinal vector These are respectively used as the X' and Y' axes of the torso dynamic coordinate system; using the shoulder vector Spinal vector The cross product (outer product) is normalized to generate the Z' axis perpendicular to the front of the chest: .

[0242] This yields the torso dynamic coordinate system for user 300, and the coordinates P of each feature point in the torso dynamic coordinate system Q are obtained based on the mirrored frame image. i =(x' i , y' i , z' i ).

[0243] S33 obtains the robot's torso posture mapping relationship. Specifically, based on the projection of the torso dynamic coordinate system onto the world coordinate system, the torso rotation angle θ and torso tilt angle Ψ are solved, thereby obtaining the torso posture mapping relationship [θ,Ψ].

[0244] S34 obtains the robot's limb pose mapping relationship. Specifically:

[0245] S341 constructs the user's limb vectors. Specifically, based on the limb feature points of the current frame, it constructs the vectors for the upper arm, forearm, and hand.

[0246] S342 obtains the robot's limb posture mapping relationship. Specifically, based on the projection of each limb vector in the torso dynamic coordinate system Q, the shoulder roll angle α1, shoulder pitch angle α2, elbow roll angle β1, elbow yaw angle β2, and wrist yaw angle r are solved, thereby obtaining the limb posture mapping relationship [α1, α2, β1, β2, r].

[0247] S35 acquires the target hand opening and closing posture of the robot. Specifically, it calculates the user's hand opening degree H = D / L, thus obtaining the hand opening and closing posture H. D is the three-dimensional Euclidean distance D between the tip of the thumb and the tip of the index finger, and L is the three-dimensional Euclidean distance from the wrist to the base of the index finger.

[0248] The first pose mapping relationship F1 of the current frame is formed by calculating θ, Ψ, α1, α2, β1, β2, r, and H from the mirror frame image of the current frame. The robot performs pose mapping according to this relationship and can control itself to follow the user's actions captured in the current frame.

[0249] S4: Optimize attitude mapping relationship

[0250] This embodiment provides a method for correcting attitude mapping relationships, referring to... Figure 11 The steps are as follows, at least one of which is:

[0251] S41 performs orientation correction on the first posture mapping relationship F1 based on the symmetry of the robot's left and right structures; the orientation correction uses α1 and β1 in the first posture mapping relationship F1 obtained in S3 as α 1输入 β 1输入 α is calculated based on the direction correction condition. 1输出 β 1输出 and with α 1输出 β 1输出 The numerical values ​​are used to update α1 and β1 in the first attitude mapping relationship F1, and the updated first attitude mapping relationship F1 is output. 11 .

[0252] The direction correction conditions are as follows:

[0253] α 1输出 =R S (α 1输入 )

[0254] β 1输出 =M S (β 1输入 )

[0255] in:

[0256] For upper limb lateral parameters, left upper limb Right upper limb ;

[0257] α1 is the shoulder roll angle;

[0258] β1 For elbow roll angle;

[0259] R S This represents the shoulder roll correction function corresponding to the current upper limb side, when... At that time, R S Adjust α1 to the negative range, preferably to... ,when At that time, R S Adjust α1 to a positive value range, preferably to a positive value range. .

[0260] M S This represents the elbow roll correction function corresponding to the current upper limb side; when At that time, M S β1 Corrected to the negative range, preferably corrected to ;when At that time, M S β1 Corrected to a positive range, preferably corrected to... .

[0261] S42 maps the first pose relationship F after orientation correction based on the maximum range of motion of the robot's limbs. 11 After performing a safety range correction, the second attitude mapping relationship F2 is obtained; the conditions for the safety range correction are as follows:

[0262] α1' = k1α1

[0263] α2' = k2α2

[0264] β1' = k3β1

[0265] β2' = k4β2

[0266] r' = k5r

[0267] H' = k6H

[0268] in:

[0269] k1 is the correction coefficient for the shoulder roll angle α1, which is determined based on the ratio between the maximum range of motion of the user's shoulder roll angle and the maximum range of motion of the robot's shoulder roll angle.

[0270] k2 is the correction coefficient for the shoulder pitch angle α2, which is determined based on the ratio between the maximum range of motion of the user's shoulder pitch angle and the maximum range of motion of the robot's shoulder pitch angle.

[0271] k3 is the correction coefficient for the elbow roll angle β1, which is determined based on the ratio between the maximum range of motion of the user's elbow roll angle and the maximum range of motion of the robot's elbow roll angle.

[0272] k4 is the correction coefficient for the elbow yaw angle β2, which is determined based on the ratio between the maximum range of the user's elbow yaw angle and the maximum range of the robot's elbow yaw angle.

[0273] k5 is the correction coefficient for the wrist yaw angle r, which is determined based on the ratio between the maximum range of the user's wrist yaw angle and the maximum range of the robot's wrist yaw angle.

[0274] k5 is the correction coefficient for the hand opening degree H, which is determined based on the ratio between the maximum range of motion of the user's hand opening and closing and the maximum range of motion of the robot's hand opening and closing.

[0275] Calculate α1', α2', β1', β2', r', and H' based on the conditions for safety range correction, obtain the second pose mapping relationship F2 = [θ, Ψ, α1', α2', β1', β2', r', and H'] for the current frame, and output it.

[0276] Although the embodiments given above illustrate that the input parameter for the safety range correction of the present invention is the updated first attitude mapping relationship F in S41 11 The mapping parameters in the diagram are not limited to these aspects. The required mapping parameters, calculated according to the conditions of the safety range correction, can be the mapping parameters in the first attitude mapping relationship F1 obtained in S3.

[0277] S43 calculates the dynamic weight attenuation coefficient W based on the degree of straightening of the elbow joint, and performs straightening correction on the elbow roll angle β1' in the second posture mapping relationship F2 based on the weight attenuation coefficient W.

[0278]

[0279] Where W is the dynamic weight decay coefficient.

[0280] The straight arm correction condition is as follows: the weight decay coefficient W of the current frame is compared with the threshold B, where B is a constant close to 1. When W = B, β1' in the second pose mapping relationship F2 of the current frame is assigned to 0 and output; otherwise, it remains unchanged and is output.

[0281] This embodiment also provides a filtering control method based on attitude mapping relationship, the steps of which are as follows:

[0282] S44 calculates the instantaneous rate of change of each mapping parameter in the second pose mapping relationship F2 between adjacent frames. and dynamic cutoff frequency f c .

[0283]

[0284]

[0285] in:

[0286] f c This is the dynamic cutoff frequency;

[0287] f min The basic low-pass cutoff frequency is set to 0.8–1 Hz;

[0288] δ is the velocity response coefficient, set to 0.4–0.5;

[0289] Let x be the instantaneous rate of change of each parameter x in the second attitude mapping relationship F2, where x includes θ, Ψ, α1', α2', β1', β2', r', and H'.

[0290] n is the number of frames.

[0291] S45 compared to f c The value of the threshold A is compared with the value of the threshold A. Based on the comparison result, the second pose mapping relationship of the previous frame is retained, or the pose mapping relationship of the current frame is iterated to form the third pose mapping relationship F3. When f c When > A, the second pose mapping relationship of the current frame is output as the third pose mapping relationship F3 of the current frame; when f c When A ≤ A, the second pose mapping relationship of the previous frame is output as the third pose mapping relationship F3.

[0292] S46 calculates the parameters of the third pose mapping relationship F3 in the current frame based on the smoothing condition, and obtains the fourth pose mapping relationship F4=[ , , , , , , , The smoothing condition is:

[0293]

[0294] in:

[0295] Smoothing factor ;

[0296] The time difference between adjacent frames. This is the time difference between frames.

[0297] S5: Transmit attitude mapping relationship and execute attitude mapping instructions.

[0298] Steps S2 to S4 are executed sequentially on each frame of the acquired images in real time. The fourth pose mapping relationship F4 of each frame is packaged into a standard JSON format and sent to robot 200 by the communication module via TCP protocol.

[0299] Although the above embodiments illustrate that the fourth posture mapping relationship F4 is issued to the robot 200 by the present invention, its scope is not limited to these aspects. The posture mapping relationship issued to the robot 200 can also be the first posture mapping relationship, the second posture mapping relationship or the third posture mapping relationship, which can be freely selected according to actual optimization needs.

[0300] The robot server safely captures the network stream and parses the JSON payload using a try-except block. The robot performs posture mapping according to the above posture mapping relationship, so that its 14 motors run smoothly and stably to the target position of each frame at a standardized speed of 0.45, thereby completing the robot's follow of the user's dance movements.

[0301] The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to the specific details described above; within the scope of the technical concept of the present invention, various equivalent transformations (such as changing the quantity, shape, position, etc.) can be made to the technical solution of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A posture mapping method for a dancing robot, characterized in that, include: Image acquisition; A user digital twin skeleton is constructed based on the image, and the user digital twin skeleton includes human feature points; A user baseline is constructed based on the feature points, and a dynamic coordinate system of the torso is established based on the user baseline to obtain the robot torso posture mapping relationship. A user limb vector is constructed based on the feature points, and the robot limb posture mapping relationship is obtained based on the projection angle of the user limb vector in the dynamic coordinate system of the torso. The robot performs posture mapping based on the torso posture mapping relationship and the limb posture mapping relationship.

2. The attitude mapping method according to claim 1, characterized in that, The user benchmarks include: A shoulder vector constructed from feature points of the left and right shoulders; The hip vector constructed from the feature points of the left hip and the feature points of the right hip; The spine vector is constructed from the midpoint of the shoulder and the midpoint of the hip.

3. The attitude mapping method according to claim 2, characterized in that, Establishing the dynamic coordinate system of the torso includes: The origin of the dynamic coordinate system of the torso is the orthogonal intersection point of the shoulder vector and the spine vector. The normalized shoulder vector and spine vector are used as the horizontal and vertical axes of the dynamic coordinate system of the torso, respectively. By using the cross product of the shoulder vector and the spine vector and normalizing it, a normal axis perpendicular to the front of the chest is generated.

4. The attitude mapping method according to claim 1, characterized in that, The torso pose mapping relationship is [θ,Ψ], where: θ is the trunk rotation angle, reflecting the degree of trunk twisting; Ψ is the trunk tilt angle, reflecting the degree of trunk tilt.

5. The attitude mapping method according to claim 1, characterized in that, The limb vectors include the upper arm vector, forearm vector, and hand vector; the limb posture mapping relationship includes [α1, α2, β1, β2, r], where: α1 is the shoulder roll angle, which reflects the degree to which the arm is raised to the sides of the body; α2 is the shoulder pitch angle, which reflects the degree of arm swing back and forth; β1 is the elbow roll angle, which reflects the degree of arm flexion; β2 is the elbow yaw angle, reflecting the degree of internal and external rotation of the arm; r is the wrist yaw angle, which represents the lateral orientation of the palm.

6. The attitude mapping method according to claim 1, characterized in that, Also includes: The robot's hand engagement degree H is obtained based on limb feature points, and H is calculated using the following formula: H = D / L Where D is the three-dimensional Euclidean distance between the thumb tip feature point and the index finger tip feature point, and L is the three-dimensional Euclidean distance from the wrist feature point to the index finger root feature point.

7. The attitude mapping method according to claim 1, characterized in that, Also includes: The attitude mapping relationship is optimized to generate an optimized attitude mapping relationship; The optimization process includes at least one of the following: direction correction, safety range correction, straight arm correction, filtering, and smoothing.

8. A posture mapping system for a dancing robot, characterized in that, include: The image processing module is used to capture the user's motion video stream and obtain the user's human body feature points; The morphology calculation module is used to construct user limb vectors, user reference, and torso dynamic coordinate system based on the feature points, and to obtain posture mapping relationships. The user limb vectors include upper arm vectors, forearm vectors, and hand vectors. The morphology calculation module includes: a torso calculation unit, used to obtain torso posture mapping relationships based on the user reference; an upper arm calculation unit, used to obtain upper arm posture mapping relationships based on upper arm vectors; a forearm calculation unit, used to obtain forearm posture mapping relationships based on forearm vectors; and a hand calculation unit, used to obtain hand posture mapping relationships based on hand vectors. The optimization module is used to optimize the attitude mapping relationship and generate an optimized attitude mapping relationship. The communication module is used to send the optimized posture mapping relationship to the robot.

9. A robot system for following a dance, characterized in that, include: The client includes the attitude mapping system and image acquisition module as described in claim 8; The robot is used to receive and execute the posture mapping relationship sent by the client.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the attitude mapping method as described in any one of claims 1 to 7.

Images