Shadow Play Autonomous Arrangement Method and System Based on OpenPose Model and Robot Kinematics

Abstract

The shadow puppetry autonomous music arrangement method and system based on the OpenPose model and robot kinematics of the present invention uses the OpenPose algorithm to detect the skeletal points of the performers, outputs the coordinate points of the key nodes of the actions, then determines the action mapping relationship between the shadow puppetry robot and the performers, and outputs the end coordinates of the shadow puppetry robot according to the mapping relationship. Finally, the execution angles of each joint of the shadow puppetry robot are obtained through the inverse solution of robot kinematics, and each joint of the shadow puppetry robot is controlled to move according to the execution angles, achieving autonomous music arrangement that imitates human postures. This method can greatly reduce the time of manually inputting angles for music arrangement and improve the music arrangement efficiency.

Description

Technical Field

[0001] The present invention relates to a robot interaction method and system, specifically a method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics. Background Art

[0002] With the continuous development of human - machine interaction technology, there are more and more ways for shadow play robots to interact with people. People only change the input method of the traditional "command - execution" interaction mode through human - machine interaction, and use interaction technology to control shadow play robots for deeper applications, that is, the autonomous music arrangement function and the compilation of action series.

[0003] Currently, most of the shadow play robot interaction technologies developed by some companies and universities passively regard interaction as the execution of commands and lack deeper capabilities, such as the method for choreographing shadow play robot actions. Summary of the Invention

[0004] Aiming at the problems existing in the prior art, the present invention provides a method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics. This method can enable a shadow play robot to imitate human actions through human motion information and systematic processing, so as to achieve the ability of autonomous music arrangement and realize human - machine interaction.

[0005] The present invention is realized through the following technical solutions:

[0006] A method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics, comprising the following steps:

[0007] Step 1: The autonomous music arranger and the shadow play robot perform the same actions. According to the OpenPose model, identify the key nodes of the actions, and determine the mapping relationship between the key nodes of the autonomous music arranger's actions and the key nodes of the shadow play robot's actions;

[0008] Step 2: Use the OpenPose model to identify the key nodes of the track actions of the autonomous music arranger, and map the key node coordinates to the shadow play plane coordinate system through the mapping relationship;

[0009] Step 3: According to kinematic simulation, determine whether the key node coordinates mapped to the shadow play plane coordinate system are within the reachable motion space of the shadow play robot. When the key node coordinates are not within the reachable motion space of the shadow play robot, abandon the action. When the key node coordinates of the action are within the reachable motion space of the shadow play robot, execute Step 4.

[0010] Step 4: Perform inverse kinematics on the key node coordinates mapped to the shadow play plane coordinate system to obtain the execution angles of each joint of the shadow play robot when performing this action;

[0011] Step 5: Repeat Steps 2 - 4 to obtain the execution angles for the shadow puppetry robot to perform each action in the track, and control the shadow puppetry robot to perform shadow puppetry according to all the execution angles.

[0012] Preferably, the key nodes of the autonomous music arranger include the left and right hands and the head, and the key nodes of the shadow puppetry robot include the execution ends of the two arms and the execution end of the head.

[0013] Preferably, the method for determining the mapping relationship between the key nodes of the actions of the autonomous music arranger and the key nodes of the actions of the shadow puppetry robot in Step 1 includes the following steps:

[0014] S1: Construct the origin of the human body coordinate system and the shadow puppetry robot coordinate system using the same method.

[0015] S2: Take the performance curtain plane as the shadow puppetry coordinate system plane, and translate the origin of the shadow puppetry coordinate system to the performance curtain to establish a shadow puppetry plane coordinate system.

[0016] S3: The autonomous music arranger and the shadow puppetry robot perform the same predetermined actions, determine the coordinates of the key nodes of the autonomous music arranger in the human body coordinate system, and obtain the coordinates of the key nodes of the shadow puppetry robot in the shadow puppetry coordinate system.

[0017] S4: Determine the mapping relationship between the key nodes of the actions of the autonomous music arranger and the key nodes of the actions of the shadow puppetry robot according to the coordinates of the key nodes of the autonomous music arranger in the human body coordinate system and the coordinates of the key nodes of the shadow puppetry robot in the shadow puppetry coordinate system.

[0018] Preferably, the method for constructing the coordinate system origin in Step S1 is as follows:

[0019] Take the intersection of the connection line of the two hands and the perpendicular line of the head as the origin of the coordinate system.

[0020] Preferably, the method for determining the mapping relationship in Step S4 is as follows:

[0021] Determine the distance L1 from the key node to the origin according to the coordinates of the key node of the action of the autonomous music arranger in the human body coordinate system; determine the distance L2 from the key node to the origin according to the coordinates of the key node of the action of the shadow puppetry robot in the shadow puppetry coordinate system.

[0022] Determine the mapping relationship according to the ratio of the distance L1 and the distance L2.

[0023] Preferably, the method for judging the coordinates of the key nodes on the shadow puppetry plane coordinate system in the reachable motion space of the shadow puppetry robot in Step 3 is as follows:

[0024] Establish a mathematical model of the shadow puppetry robot. Use the coordinate system of the mathematical model and combine the D-H parameter method to describe the lengths between the angles of each joint of the shadow puppetry robot and the coordinates of the moving end. Use the forward kinematics and MatLab to simulate the reachable workspace of the shadow puppetry robot in three-dimensional space using the Monte Carlo method. Take one of the two-dimensional planes as the shadow puppetry performance plane as the reachable workspace;

[0025] Determine whether the key node coordinates on the shadow puppetry plane coordinate system are within the reachable workspace. If they are within the workspace, step 5 can be performed.

[0026] Preferably, the method for determining the execution angles of the joints in step 5 is as follows:

[0027] According to the mathematical model of the shadow puppetry robot described by the D-H parameters and combined with the kinematic equations, obtain the relationship equation between the coordinates of the joint end of the shadow puppetry robot and the angles of each joint. Substitute the key node coordinates in the shadow puppetry plane coordinate system into the relationship equation between the joint end position coordinates and the angles for calculation to obtain the set of angle solutions for each joint. Take the solution with the shortest movement time compared to the previous action in the set of angle solutions as the optimal solution to obtain the execution angles of the joints.

[0028] Preferably, the method for determining the optimal solution is as follows:

[0029] Obtain the maximum change angle Δθ of each set of angle solutions max , and obtain multiple Δθ max , take the set of angle solutions corresponding to the minimum Δθ max among them as the optimal solution.

[0030] Preferably, the expression of the kinematic equation is as follows:

[0031]

[0032] Among them, is the transformation matrix from the {0} coordinate system to the {4} coordinate system, is the transformation matrix from the {0} coordinate system to the {1} coordinate system, is the transformation matrix from the {1} coordinate system to the {2} coordinate system, is the transformation matrix from the {2} coordinate system to the {3} coordinate system, is the transformation matrix from the {3} coordinate system to the {4} coordinate system.

[0033] A system for an autonomous shadow puppetry composition method based on the OpenPose model and robot kinematics includes,

[0034] A mapping relationship module, which is used to identify the same actions and key nodes of the independent arranger and the shadow puppet robot according to the OpenPose model, and determine the mapping relationship between the key nodes of the actions of the independent arranger and the key nodes of the actions of the shadow puppet robot;

[0035] A key node module, which is used to obtain the key nodes of the track actions of the independent arranger, and map the key node coordinates to the shadow puppet plane coordinate system through the mapping relationship;

[0036] A key node judgment module, which is used to determine whether the key node coordinates are within the reachable motion space of the shadow puppet robot according to kinematic simulation. When the key node coordinates are not within the reachable motion space of the shadow puppet robot;

[0037] An execution angle module, which is used to perform inverse kinematics on the key node coordinates to obtain the execution angles of each joint of the shadow puppet robot when performing this action;

[0038] A track control module, which is used to obtain the execution angles of each action performed by the shadow puppet robot in the track, and control the shadow puppet robot to perform shadow puppet performances according to all the execution angles.

[0039] Compared with the prior art, the present invention has the following beneficial technical effects:

[0040] The shadow puppet independent arrangement method based on the OpenPose model and robot kinematics of the present invention constructs a shadow puppet robot independent arrangement method using the OpenPose algorithm and the forward kinematics and inverse kinematics of robot kinematics in this method. The OpenPose algorithm is used to detect the bone points of the performer, output the key node coordinate points of the action, then determine the action mapping relationship between the shadow puppet robot and the performer, and output the end coordinates of the shadow puppet robot according to the mapping relationship. Finally, the inverse kinematics of robot kinematics is used to obtain the execution angles of each joint of the shadow puppet robot, and the movement of each joint of the shadow puppet robot is controlled according to the execution angles to achieve independent arrangement of imitating human postures. This method can greatly reduce the time of independently inputting angles for arrangement and improve the arrangement efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the independent arrangement method of the present invention;

[0042] Figure 2 It is a flowchart of determining the mapping relationship of the present invention;

[0043] Figure 3 It is a flowchart of judging the reachable working space of the shadow puppet robot of the present invention;

[0044] Figure 4 It is a simulation diagram of the kinematic reachable working space of the present invention;

[0045] (a) is the reachable workspace of the robot in three-dimensional space;

[0046] (b) is the reachable workspace of the robot on a two-dimensional plane;

[0047] Figure 5 This is the OpenPose key point map of the present invention;

[0048] Figure 6 This is the coordinate system of the shadow puppetry robot mathematical model of the present invention;

[0049] Figure 7 This is the mapping diagram of human body postures and the shadow puppetry robot of the present invention;

[0050] (a) is the schematic diagram of the human body coordinate system;

[0051] (b) is the schematic diagram of the shadow puppetry coordinate system;

[0052] Figure 8 This is the schematic diagram of the human body coordinate system and the shadow puppetry robot coordinate system of the present invention;

[0053] (a) is the human body coordinate system diagram;

[0054] (b) is the shadow puppetry coordinate system diagram;

[0055] Figure 9 This is the effect diagram of the present invention;

[0056] Figure 9 (a) is the schematic diagram with both hands down, reaching Figure 9 (b) is the state diagram with both arms tilted, Figure 9 (c) is the state diagram with both arms horizontal.

[0057] Figure 10 This is the structure diagram of the system of the present invention.

[0058] In the figure: 1. Shadow puppetry robot; 2. Acquisition device; 3. Communication device; 4. Main control system. Detailed implementation manner

[0059] The present invention will be further described in detail below with reference to the accompanying drawings. The following is an explanation of the present invention rather than a limitation.

[0060] Refer to Figure 1 , the shadow puppetry autonomous music composition method based on the OpenPose model and robot kinematics includes the following steps:

[0061] Step 1. Establish and train an OpenPose model for identifying key nodes in human action pictures.

[0062] Specifically, the trained OpenPose model is used to identify the joint points of each frame of the human action image, and only three key joint points of the head, left hand, and right hand are retained.

[0063] Refer to Figure 5 As shown, each frame of the image includes the joint points of the human body. In Figure 5 , it includes joint points 0 to 17. In this embodiment, only three joint points of the head, left hand, and right hand need to be retained, and their corresponding serial numbers are 0, 4, and 7. The head, left hand, and right hand of the human body correspond to the three joint ends on the shadow puppet robot.

[0064] Step 2: The autonomous music composer and the shadow puppet robot perform the same actions. The OpenPose model identifies the key joint points of the actions and determines the mapping relationship between the key joint point coordinates of the autonomous music composer's actions and the key joint point coordinates of the shadow puppet robot's actions.

[0065] The method for establishing the mapping relationship between the key joint point coordinates of the autonomous music composer's actions and the key joint point coordinates of the shadow puppet robot's actions refers to Figure 2 , and specifically includes the following steps:

[0066] S2.1: Construct a human body coordinate system, and take the vertical intersection point of the connection line of the two hands and the head as the origin of the human body coordinate system.

[0067] Take pictures of the autonomous music composer through the camera. A network structure will be constructed in the picture, and the overall size is the size of the pixel points. The position of each grid is the pixel point. Take the vertical intersection point of the connection line of the two hands of the autonomous music composer and the head as the origin of the human body coordinate system. As Figure 8 shown in a, the horizontal coordinates between the key joint points and the origin are (u, v), indicating that the pixel point of the abscissa is u pixel points away from the origin, and the pixel point of the ordinate is v pixel points away from the origin. The included angle with the abscissa is a. For the convenience of calculation, if the pixel point falls into a certain grid, it is determined as the center position of the current grid.

[0068] S2.2: Construct a shadow puppet robot coordinate system, and take the intersection point of the horizontal line of the shadow puppet robot's arm and the vertical line of the head as the origin of the shadow puppet robot coordinate system.

[0069] Participate in Figure 9 b. Draw a vertical line through the head of the shadow puppet robot, and draw a horizontal line along the end of the arm. The intersection point of the vertical line and the horizontal line is the origin of the shadow puppet robot coordinate system.

[0070] When the human body appears in the detection range of the camera, the coordinates of the human body will be collected, and the coordinates of the left and right hands and the head of the human body can be detected in Figure 9As shown in (a) of [Figure ID], with both hands down, the shadow puppetry robot remains stationary. Based on (a) of the figure, the left and right hands move upward to reach the state shown in (b). After obtaining the change in the human body coordinates, the calculated joint angles of the robot change accordingly. The servo motors drive the robot joints to rotate to complete the action imitation. Finally, both hands are nearly horizontal, reaching the state shown in (c).

[0071] S2.3. Take the plane of the performance curtain as the plane of the shadow puppetry coordinate system, and translate the origin of the shadow puppetry coordinate system to the performance curtain to establish a shadow puppetry plane coordinate system.

[0072] As Figure 7 shown, during the following relationship mapping process, only three points of the joint end coordinates are required for the shadow puppetry robot, namely the three key nodes corresponding to the left and right hands and the head of the human body. Therefore, the key node coordinate points are reserved in advance among the output key nodes.

[0073] S2.4. The autonomous choreographer makes a predetermined action, and the OpenPose model identifies the key nodes of the action and determines the coordinates of the key nodes in the human body coordinate system.

[0074] For example, select the autonomous choreographer with feet together and hands open and parallel to the ground, and calculate the coordinates of the head, left hand, and right hand according to the human body coordinate system.

[0075] S2.5. The shadow puppetry robot makes the same predetermined action as in step S2.4, and obtains the coordinates of the key nodes of the shadow puppetry robot in the shadow puppetry coordinate system.

[0076] The shadow puppetry robot makes the same action, and obtains the coordinates of the left and right hands and the head in the shadow puppetry coordinate system through measurement.

[0077] S2.6. Determine the distance L1 from the key node to the origin according to the coordinates of the key nodes of the autonomous choreographer in the human body coordinate system in step S2.5, determine the distance L2 from the key node to the origin according to the coordinates of the key nodes of the shadow puppetry robot in the shadow puppetry coordinate system in step S2.4, and determine the mapping relationship according to the distance L1 and the distance L2. The mapping relationship is the ratio of the distance L1 and the distance, that is, the mapping coefficient.

[0078] For example, the distance from the coordinate of the left hand key point to the origin in the human body coordinate system is denoted as L1, and the distance from the end of the left arm joint of the shadow puppetry robot to the origin in the shadow puppetry robot coordinate system is L2. The ratio of the two is the mapping coefficient of the left hand and the end of the left arm joint. Similarly, the mapping coefficients of the right hand and the head can be obtained, and the included angle is the same in the two coordinate systems. In summary, the determination of the mapping parameters is completed. The parameters include the mapping coefficients of the left hand, right hand, and head, and the angles are the same.

[0079] Step 3: Use the OpenPose model to identify the key nodes of the track actions of the autonomous arranger, and map the coordinates of the key nodes to the shadow play plane coordinate system through the mapping relationship.

[0080] Step 4: According to kinematic simulation, determine whether the coordinates of the key nodes in Step 3 are within the reachable motion space of the shadow play robot. When the coordinates of the key nodes are not within the reachable motion space of the shadow play robot, abandon the action or adjust the action until the coordinates of the key nodes of the action are within the reachable motion space of the shadow play robot, and then execute Step 5.

[0081] Specifically, refer to Figure 3 First, establish a mathematical model of the shadow play robot. The coordinate system of the mathematical model is established as Figure 6 shown. Use the coordinate system of the mathematical model and combine the D-H parameter method to describe the angles of each joint of the shadow play robot and the lengths between the end coordinates of the motion. Use the forward kinematics and MatLab to simulate the reachable workspace of the shadow play robot in three-dimensional space using the Monte Carlo method as Figure 4 shown in the left figure. Take one of the two-dimensional planes as the shadow play performance plane as the final reachable workspace as Figure 4 shown in the right figure. The reachable workspace represents all the activity spaces where the shadow play robot can move on the performance plane. Judge by comparing the coordinates calculated in Step 3 with this workspace to obtain a judgment on whether a result can be obtained. If it is within the workspace, Step 5 can be carried out; if not, re-collect the pictures.

[0082] Step 5: Perform inverse kinematics on the coordinates of the key nodes in Step 4 to obtain the execution angles of each joint when the shadow play robot executes this action.

[0083] Specifically, use the mathematical model of the shadow play robot described by the established D-H parameters, and combine the algebraic solution of the kinematic equation to solve the inverse equation to obtain the relationship equation between the end coordinates of the joints of the shadow play robot and the angles of each joint. Substitute the coordinates of the key nodes in the shadow play plane coordinate system obtained in Step 4 into the relationship equation between the end position coordinates and angles of the joints for calculation to obtain the set of angle solutions. Take the solution with the shortest motion time from the set of angle solutions to the previous action as the final solution to obtain the execution angles of the joints.

[0084] In the process of calculating the algebraic solution, there are usually multiple solutions. The solution results of each angle of the robot are θ i , and the optimal solution needs to be found. In the scenario of shadow play performance, an action requires the angles of all joints, but the motion time of an action is determined by the joint with the largest change in angle among all joints. Therefore, it is necessary to find the largest change in angle Δθ max of each group of solutions, and then find multiple Δθ maxAnd take the Δθ among them max The smallest set of solutions is the final solution, ensuring the shortest action movement time. The initial angle is not in the arrangement sequence and can be given arbitrarily.

[0085] The above kinematic equation expressions are as follows:

[0086]

[0087] Among them, is the transformation matrix from the {0} coordinate system to the {4} coordinate system, which is obtained by is the transformation matrix from the {0} coordinate system to the {1} coordinate system, and so on, and is the transformation matrix of the coordinate system.

[0088] Step 6: Repeat steps 3 - 5 to obtain the execution angles for the shadow puppet robot to perform each action in the repertoire, and control the shadow puppet robot to perform shadow puppet shows based on all the execution angles.

[0089] Specifically, as Figure 9 shown, send the angle information to the main control system of the shadow puppet robot, and the main control system of the shadow puppet robot controls the movement of each joint of the shadow puppet robot according to the execution angle to realize the autonomous performance of shadow puppet actions by the shadow puppet robot.

[0090] Refer to Figure 10 , a system for a shadow puppet autonomous arrangement method based on the OpenPose model and robot kinematics, including a shadow puppet robot 1, a collection device 2; a communication device 3 and a main control system 4;

[0091] The collection device 2 is used to obtain the action images of the autonomous arranger and the shadow puppet robot;

[0092] The communication device 3 is used to transmit the image information obtained by the collection device 2 to the main control system 4;

[0093] The main control system 4 includes a mapping relationship module, a key node module, a key node judgment module, an execution angle module, and a repertoire control module;

[0094] The mapping relationship module is used to identify the same actions and key nodes of the autonomous arranger and the shadow puppet robot according to the OpenPose model, and determine the mapping relationship between the key nodes of the autonomous arranger's actions and the key nodes of the shadow puppet robot's actions;

[0095] The key node module is used to obtain the key nodes of the repertoire actions of the autonomous arranger, and map the key node coordinates to the shadow puppet plane coordinate system through the mapping relationship;

[0096] The key node judgment module is used to determine whether the coordinates of the key nodes are within the reachable motion space of the shadow puppet robot according to kinematic simulation. When the coordinates of the key nodes are not within the reachable motion space of the shadow puppet robot;

[0097] The execution angle module is used to perform inverse kinematic solution on the coordinates of the key nodes to obtain the execution angles of each joint of the shadow puppet robot when performing this action;

[0098] The repertoire control module is used to obtain the execution angles of the shadow puppet robot for each action in the repertoire, and control the shadow puppet robot to perform shadow puppet shows according to all the execution angles.

[0099] The shadow puppet autonomous arrangement method based on the OpenPose model and robot kinematics of the present invention uses the OpenPose algorithm and the forward kinematics and inverse kinematics of robot kinematics to construct the shadow puppet robot's autonomous arrangement method. The OpenPose algorithm is used to detect the bone points of the performer, and the key node coordinate points of the action are output, including the left hand, right hand and head. Then, the proportionality coefficient of the action mapping between the shadow puppet robot and the performer is determined. The performer stretches the hand as straight as possible, and determines the origin and the proportionality coefficient after recording the action. The three end positions corresponding to the shadow puppet robot are the head, the left hand and the right hand. And it is scaled according to the proportionality coefficient to output the end coordinates suitable for the shadow puppet robot. The angles of each joint of the shadow puppet robot are obtained through inverse kinematic solution of the robot kinematics, and the angle information is transmitted to the main control system of the shadow puppet robot. The main control system of the shadow puppet robot will control the joints of the shadow puppet robot to move to achieve autonomous arrangement of imitating human postures. This method can greatly reduce the time of manually inputting angles for arrangement, improve efficiency, and can imitate the actions of many opera actors. Taking the video of the opera actor as input, the corresponding shadow puppet performance opera can be obtained.

[0100] The above content is only to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics, characterized in that It includes the following steps: Step 1: The independent music arranger and the shadow puppet robot perform the same actions. According to the OpenPose model, identify the key nodes of the actions, and determine the mapping relationship between the key nodes of the independent music arranger's actions and the key nodes of the shadow puppet robot's actions; Step 2: Map the key node coordinates of the independent music arranger's actions to the shadow puppet plane coordinate system according to the mapping relationship; Step 3: According to kinematic simulation, determine whether the key node coordinates mapped to the shadow puppet plane coordinate system are within the reachable motion space of the shadow puppet robot. When the key node coordinates are not within the reachable motion space of the shadow puppet robot, abandon the action. When the key node coordinates of the action are within the reachable motion space of the shadow puppet robot, execute Step 4; Step 4: Perform inverse kinematics on the key node coordinates mapped to the shadow puppet plane coordinate system to obtain the execution angles of each joint of the shadow puppet robot when performing this action; Step 5: Repeat Steps 2 - 4 to obtain the execution angles of each action performed by the shadow puppet robot in the repertoire, and control the shadow puppet robot to perform shadow puppet shows according to all the execution angles.

2. The method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics according to claim 1, characterized in that The key nodes of the independent music arranger include the left and right hands and the head, and the key nodes of the shadow puppet robot include the execution ends of the two arms and the execution end of the head.

3. The shadow puppetry autonomous music composition method based on the OpenPose model and robot kinematics according to claim 1, characterized in that, The method for determining the mapping relationship between the key nodes of the independent music arranger's actions and the key nodes of the shadow puppet robot's actions in Step 1 includes the following steps: S1: Use the same method to construct the origin of the human body coordinate system and the shadow puppet robot coordinate system; S2: Take the performance curtain plane as the shadow puppet coordinate system plane, translate the origin of the shadow puppet coordinate system to the performance curtain, and establish the shadow puppet plane coordinate system; S3: The independent music arranger and the shadow puppet robot perform the same predetermined actions, determine the coordinates of the key nodes of the independent music arranger in the human body coordinate system, and obtain the coordinates of the key nodes of the shadow puppet robot in the shadow puppet coordinate system; S4: According to the coordinates of the key nodes of the independent music arranger in the human body coordinate system and the coordinates of the key nodes of the shadow puppet robot in the shadow puppet coordinate system, determine the mapping relationship between the key nodes of the independent music arranger's actions and the key nodes of the shadow puppet robot's actions.

4. The shadow puppetry autonomous music arrangement method based on the OpenPose model and robot kinematics according to claim 3, characterized in that The method for constructing the coordinate system origin in Step S1 is as follows: Take the intersection of the connection line of both hands and the perpendicular line of the head as the origin of the coordinate system.

5. The shadow puppetry autonomous music composition method based on the OpenPose model and robot kinematics according to claim 3, characterized in that The method for determining the mapping relationship in Step S4 is as follows: Determine the distance L1 from the key node to the origin according to the coordinates of the key nodes of the independent music arranger's actions in the human body coordinate system; determine the distance L2 from the key node to the origin according to the coordinates of the key nodes of the shadow puppet robot's actions in the shadow puppet coordinate system; Determine the mapping relationship according to the ratio of the distance L1 and the distance L2.

6. The shadow puppetry autonomous music arrangement method based on the OpenPose model and robot kinematics according to claim 1, characterized in that, The method for judging whether the key node coordinates on the shadow puppet plane coordinate system are within the reachable motion space of the shadow puppet robot in Step 3 is as follows: Establish a mathematical model of the shadow puppet robot, use the coordinate system of the mathematical model and combine the D - H parameter method to describe the angles of each joint of the shadow puppet robot and the lengths between the coordinates of the motion ends. Use forward kinematics and MatLab to simulate using the Monte Carlo method to obtain the reachable working space of the shadow puppet robot in three - dimensional space, and take one of the two - dimensional planes as the shadow puppet performance plane as the reachable working space; Determine whether the key node coordinates in the shadow play plane coordinate system are within the reachable workspace. If they are in the workspace, proceed to step 5.

7. The shadow puppetry autonomous music composition method based on the OpenPose model and robot kinematics according to claim 1, characterized in that The method for determining the execution angles of the joints in step 4 is as follows: Based on the mathematical model of the shadow play robot described by the D-H parameters and combined with the kinematic equation, obtain the relationship equation between the joint end coordinates and the angles of each joint of the shadow play robot. Substitute the key node coordinates in the shadow play plane coordinate system into the relationship equation between the joint end position coordinates and the angles for calculation to obtain the set of angle solutions for each joint. Select the solution with the shortest movement time compared to the previous action in the set of angle solutions as the optimal solution to obtain the execution angles of the joints.

8. The method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics according to claim 7, wherein The method for determining the optimal solution is as follows: Obtain the maximum change angle Δθ of the solution sets at each angle max , and obtain multiple Δθ max , and select the angle solution set corresponding to the minimum value of Δθ among them max as the optimal solution.

9. The method for autonomous shadow play music arrangement based on the OpenPose model and robot kinematics according to claim 7, characterized in that, The expression of the kinematic equation is as follows: Among them, is the transformation matrix from the {0} coordinate system to the {4} coordinate system, is the transformation matrix from the {0} coordinate system to the {1} coordinate system, is the transformation matrix from the {1} coordinate system to the {2} coordinate system, is the transformation matrix from the {2} coordinate system to the {3} coordinate system, is the transformation matrix from the {3} coordinate system to the {4} coordinate system.

10. A system for the method of autonomous shadow play music arrangement based on the OpenPose model and robot kinematics according to any one of claims 1-9, characterized in that, including, A mapping relationship module, which is used for the autonomous music composer and the shadow play robot to perform the same actions, identify the key nodes of the actions according to the OpenPose model, and determine the mapping relationship between the key nodes of the actions of the autonomous music composer and the key nodes of the actions of the shadow play robot; A key node module, which is used to map the key node coordinates of the actions of the autonomous music composer to the shadow play plane coordinate system according to the mapping relationship; A key node judgment module, which is used to determine whether the key node coordinates are within the reachable motion space of the shadow play robot according to the kinematic simulation. When the key node coordinates are not within the reachable motion space of the shadow play robot; An execution angle module, which is used to perform the inverse kinematics of the key node coordinates to obtain the execution angles of each joint when the shadow play robot performs this action; A track control module, which is used to obtain the execution angles of each action performed by the shadow play robot in the track, and control the shadow play robot to perform shadow play according to all the execution angles.

Images