Motion control method and device, robot and storage medium

Abstract

The present disclosure relates to a motion control method, device, robot and storage medium, the method is applied to a robot, comprising: obtaining a music file to be played, wherein the music file comprises a plurality of playing times, and at least one played instrument for each playing time; determining a head position corresponding to each playing time according to at least one played instrument for each playing time; and controlling a playing part and a head of the robot to perform a playing action according to at least one played instrument and the head position for each playing time, so that the corresponding played instrument sounds at each playing time and the head is at the corresponding head position.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, specifically to a motion control method, device, robot, and storage medium. Background Technology

[0002] In recent years, robotics technology has been continuously developing, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Current robots can not only perform repetitive, periodic actions such as sweeping and walking, but they can also play musical instruments; for example, a robot can perform a drum set performance. However, when robots perform drum set performances, there are still issues such as simple and monotonous movements, low anthropomorphism, and insufficient dynamic characteristics, meaning the performance effect needs further improvement. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides a motion control method, device, robot, and storage medium to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, a motion control method is provided for a robot, the method comprising: Obtain a music file to be played, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time; The head position for each performance time is determined based on at least one instrument played at each performance time. Based on at least one instrument being played and the head position at each performance time, the robot's playing part and head are controlled to perform playing actions, so as to make the corresponding instrument being played sound at each performance time and to position the head in the corresponding head position.

[0005] In one embodiment, each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The step of determining the head position for each performance time based on at least one instrument being played includes: If at least one of the directions marked by the instrument being played is different at any performance time, and the head position at the previous performance time is in the first direction of the original head position, the head position at that performance time is determined within the second direction of the original head position. In one embodiment, determining the head position for the performance time in the second direction from the original head position includes: The head position at the performance time is determined by any one of at least one preset position in the second direction of the original head position.

[0006] In one embodiment, each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The step of determining the head position for each performance time based on at least one instrument being played includes: If at least one of the instruments being played is marked in the same direction at any given performance time, the head position for that performance time is determined based on the position of the at least one instrument being played.

[0007] In one embodiment, determining the head position for the current performance time within the direction relative to the head position of the previous performance time includes: The head position for the current performance time is determined from any one of at least one preset position in the direction opposite to the head position of the previous performance time.

[0008] In one embodiment, determining the head position corresponding to the performance time based on the position of the at least one played instrument includes: If there is a playing instrument during the performance time, the head position corresponding to the playing instrument is determined as the head position during the performance time. When multiple instruments are played during a performance, the midpoint of the head position corresponding to each of the multiple instruments is determined as the head position during that performance.

[0009] In one embodiment, the music file further includes at least one playing part of each playing instrument at each performance time; The method further includes: If any of the instruments played at any time in the music file exceeds the range of motion of the corresponding playing part, the playing part of the instrument will be updated to another playing part.

[0010] In one embodiment, controlling the robot's playing part and head to perform playing actions based on at least one played instrument and head position at each playing time, so as to make the corresponding played instrument sound and position the head in the corresponding head position at each playing time, includes: For each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time; For each instrument played at each performance time, the performance trajectory of the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory. For each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

[0011] In one embodiment, generating the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and controlling the playing part corresponding to the instrument being played to execute the performance trajectory, includes: The duration of each stage of the performance action is determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, a playing trajectory for each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory for each stage in sequence.

[0012] In one embodiment, determining the duration of each stage of the performance action based on the attributes of the instrument being played and the performance duration corresponding to the performance time includes: When the instrument being played is a drum, the duration of the lifting phase and the duration of the striking phase are determined based on the performance duration corresponding to the performance time.

[0013] In one embodiment, generating the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and controlling the playing part corresponding to the instrument being played to execute the performance trajectory, includes: The action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a playing trajectory for the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory within the duration of the action.

[0014] In one embodiment, determining the action duration and waiting duration based on the action time range of the playing part corresponding to the played instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit.

[0015] In one embodiment, determining the action duration and waiting duration based on the action time range of the playing part corresponding to the played instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is within the action time range of the corresponding playing part of the instrument, the action duration is determined to be the performance duration of at least one instrument corresponding to the performance time, and the waiting time is determined to be 0.

[0016] In one embodiment, determining the action duration and waiting duration based on the action time range of the playing part corresponding to the played instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

[0017] According to a second aspect of the present disclosure, a motion control device is provided for use in a robot, the device comprising: The acquisition module is used to acquire the music file to be played, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time; The determination module is used to determine the head position for each performance time based on at least one instrument being played at each performance time. The motion module is used to control the robot's playing part and head to perform playing actions based on at least one instrument being played and the head position at each playing time, so as to make the corresponding instrument being played sound and to position the head in the corresponding head position at each playing time.

[0018] In one embodiment, each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The determining module is specifically used for: If at least one of the directions marked by the instrument being played is different at any given performance time, and the head position at the previous performance time is in a first direction of the original head position, the head position at that performance time is determined in a second direction of the original head position.

[0019] In one embodiment, the determining module is used to determine the head position during the performance time within a second direction from the original head position, specifically for: The head position at the performance time is determined by any one of at least one preset position in the second direction of the original head position.

[0020] In one embodiment, each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The determining module is specifically used for: If at least one of the instruments being played is marked in the same direction at any given performance time, the head position for that performance time is determined based on the position of the at least one instrument being played.

[0021] In one embodiment, when the determining module determines the head position corresponding to the performance time based on the position of the at least one played instrument, it is specifically used for: If there is a playing instrument during the performance time, the head position corresponding to the playing instrument is determined as the head position during the performance time. When multiple instruments are played during a performance, the midpoint of the head position corresponding to each of the multiple instruments is determined as the head position during that performance.

[0022] In one embodiment, the music file further includes at least one playing part of each playing instrument at each performance time; The device further includes an update module for: If any of the instruments played at any time in the music file exceeds the range of motion of the corresponding playing part, the playing part of the instrument will be updated to another playing part.

[0023] In one embodiment, the action module is specifically used for: For each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time; For each instrument played at each performance time, the performance trajectory of the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory. For each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

[0024] In one embodiment, the action module is used to generate a performance trajectory for the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The duration of each stage of the performance action is determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, a playing trajectory for each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory for each stage in sequence.

[0025] In one embodiment, when the action module is used to determine the duration of each stage of the performance action based on the attributes of the instrument being played and the performance duration corresponding to the performance time, it is specifically used for: When the instrument being played is a drum, the duration of the lifting phase and the duration of the striking phase are determined based on the performance duration corresponding to the performance time.

[0026] In one embodiment, the action module is used to generate a performance trajectory for the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a playing trajectory for the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory within the duration of the action.

[0027] In one embodiment, when the action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time, it is specifically used for: If the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit.

[0028] In one embodiment, when the action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time, it is specifically used for: If the performance duration of at least one instrument corresponding to the performance time is within the action time range of the corresponding playing part of the instrument, the action duration is determined to be the performance duration of at least one instrument corresponding to the performance time, and the waiting time is determined to be 0.

[0029] In one embodiment, when the action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time, it is specifically used for: If the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

[0030] According to a third aspect of the present disclosure, a robot is provided, the robot including a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the motion control method described in the first aspect when executing the computer instructions.

[0031] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0032] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: The motion control method provided in this disclosure acquires a music file to be played, which yields multiple performance times and at least one instrument to be played at each performance time. Based on this, the head position for each performance time is determined. Finally, based on the at least one instrument and the head position at each performance time, the robot's playing part and head are controlled to perform performance actions, causing the corresponding instrument to sound and the head to be in the corresponding position at each performance time. The performance time is the duration of instrument sound. Because the head position for each performance time is determined, the robot's head moves rhythmically during the performance, increasing the diversity of movements and improving anthropomorphism, overcoming problems such as insufficient dynamic characteristics and poor performance effects. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0034] Figure 1 This is a flowchart illustrating a motion control method according to an exemplary embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a drum kit shown in an exemplary embodiment of this disclosure; Figure 3 This is a flowchart illustrating the determination of head position as shown in an exemplary embodiment of this disclosure; Figure 4 This is a flowchart illustrating adaptive preprocessing of music files according to an exemplary embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the execution of a performance action as shown in an exemplary embodiment of this disclosure; Figure 6 This is a flowchart illustrating the execution of a performance action as shown in another exemplary embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of a motion control device shown in an exemplary embodiment of the present disclosure; Figure 8 This is a structural block diagram of a robot illustrated in an exemplary embodiment of this disclosure. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0037] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0038] In recent years, robotics technology has been continuously developing, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Current robots can not only perform repetitive, periodic actions such as sweeping and walking, but also play musical instruments; for example, a robot can perform a drum set performance. However, when robots perform drum set performances, there are still issues such as simple and monotonous movements, low anthropomorphism, and insufficient dynamic characteristics, meaning the performance effect needs further improvement.

[0039] In related technologies, robots used to play musical instruments are often specialized robots that can only perform single, repetitive actions to play the instrument. These robots are highly mechanized, with simple, repetitive movements and low anthropomorphism. Furthermore, they cannot perform other actions; they are only capable of playing instruments in pre-defined scenarios, resulting in limited functionality and poor versatility. For example, related technologies use multiple (usually three or more) industrial robotic arms to play a drum kit, with each arm responsible for two drums. This results in low anthropomorphism and the ability to only perform the function of playing a drum kit.

[0040] Based on this, in a first aspect, at least one embodiment of this disclosure provides a motion control method, please refer to the appendix. Figure 1It illustrates the process of the method, including steps S101 to S103.

[0041] This method can be applied to robots, such as legged robots. The robot may have a torso and limbs. In one possible embodiment, please refer to the appendix. Figure 2 This method can be applied to scenarios where robots play drums; where a drum set includes multiple drums, namely... Figure 2 The robot is a general-purpose robot, not a dedicated robot for playing the drum kit. The robot's lower limbs have 10 degrees of freedom, and its upper limbs also have 10 degrees of freedom, enabling it to perform multiple functions such as walking, jumping, picking up and placing objects, and dancing.

[0042] In step S101, a music file to be played is obtained, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time.

[0043] The music file can be a digital file, such as a MIDI (Musical Instrumental Digital Interface) file. MIDI (Musical Instrumental Digital Interface) is a real-time data communication protocol between electronic musical instruments, synthesizers, and other music-related hardware. The encoding of this protocol has been extended to become a file format that can record music information, namely a MIDI file.

[0044] A performance time can be the instant at which at least one instrument produces sound; that is, at least one instrument needs to be played during the performance time. Multiple performance times within a music file can form a time sequence, and the at least one instrument played at each performance time can serve as a note for that performance time. Additionally, the music file can also include the corresponding playing part for each of the at least one instrument played at each performance time, further specifying which playing part should be used for each instrument.

[0045] In a scenario where a robot plays the drums, at least one instrument in each performance time can be a portion of the drums on the drum kit, and the corresponding playing part for each instrument can be one of the robot's four limbs. In this scenario, the music file can be simply understood as a time sequence of different drumheads being struck, such as: at 1 second, drum 1 is struck by the right hand, drum 4 by the left hand, drum 8 by the left foot, and drum 9 by the right foot; at 1.1 seconds, drum 2 is struck by the right hand, and drum 3 by the left hand; at 1.35 seconds, drum 6 is struck by the right hand, and so on.

[0046] In step S102, the head position for each performance time is determined according to at least one instrument played at each performance time.

[0047] It's understandable that when a person performs, their head will move accordingly to enhance the performance. Therefore, to improve the anthropomorphism and diversity of movements, humanoid robots can also generate head movement sequences based on music files, that is, determine the head position at each performance moment. The movements of humanoid robots can be defined as two situations: one is a state of immersion, where the robot's head sways up and down or left and right; the other is a following state, where the robot's head makes movements that follow the parts being played.

[0048] The range of motion of the head can be symmetrical about its original position, meaning the range of motion can be divided into a first direction and a second direction. The original head position refers to the robot's initial assembly position, and the first and second directions are relative directions, such as left and right, or forward and backward. For example, the head position can be represented by head-related joint angles (e.g., the joints between the head and neck, or the joints between the neck and torso). When the head is in its original position, the head-related joint angle is 0; when the head is in the first direction (e.g., left, forward), the head-related joint angle is less than 0; and when the head is in the second direction (e.g., right, backward), the head-related joint angle is greater than 0.

[0049] Correspondingly, each instrument being played is marked with a first or second direction, meaning the direction of each instrument is predefined. For example... Figure 2 The drum kit shown has cymbals 1, cymbals 2, drum 4, drum 5, and pedal 8 marked as left, and cymbals 3, drum 6, drum 7, and pedal 9 marked as right.

[0050] Based on the above directional description, for example, at any given performance time, at least one of the directions marked by the played instruments are different directions (e.g., the played instrument at that performance time is...). Figure 2 In the case of drums 4 and 6 in the previous performance, and the head position in the previous performance is in the first direction of the original head position, the head position for this performance is determined within the second direction of the original head position. For example, any one of at least one preset position within the second direction of the original head position can be (randomly) determined as the head position for this performance. In this example, the robot is in an ecstatic state. The directions of the robot's head can be predefined as left and right, and an ecstatic position can be preset on the left and right sides respectively. Then, when the head position needs to be determined on the left, the ecstatic position on the left can be set as the head position, and when the head position needs to be determined on the right, the ecstatic position on the right can be set as the head position.

[0051] For example, at any given performance time, at least one of the instruments being played is marked in the same direction (e.g., the instrument being played at that performance time is...). Figure 2 In the case of drums 4 and 5, the head position for the performance time is determined based on the position of the at least one instrument being played. For example, if there is one instrument being played during the performance time, the head position corresponding to that instrument is determined as the head position for that performance time; if there are multiple instruments being played during the performance time, the midpoint of the head positions corresponding to the multiple instruments is determined as the head position for that performance time. In this example, the robot is in a following state, where the head position corresponding to each instrument can be preset.

[0052] Please refer to the appendix. Figure 3 The example illustrates the process of determining the head position for each performance time based on the instrument being played.

[0053] After starting, read the notes for the i-th performance time in the music file, such as: at the 3-second mark, drum 7 is struck by the left hand, pedal 8 by the left foot, and pedal 9 by the right foot. Then, determine if the instruments on the left and right sides are struck simultaneously. If both drums are struck simultaneously, the frame is considered to be in an ecstasy state. Read the head position from the previous frame. If the head position in the previous frame was on the left or in the middle, determine the head position as the ecstasy position on the right (e.g., -0.3 rad). If the head position in the previous frame was on the right, then... The head position is determined as the left-hand ecstasy position (e.g., 0.3 rad). If the left and right drums are not struck simultaneously, the frame is considered a follow-up frame. If one drum is struck in the frame, the head position is determined as the head position corresponding to the struck drum. If two or more drums are struck in the frame, the head position is determined as the midpoint (average angle) of the head positions corresponding to the struck drums. The head position corresponding to each drum can be preset, such as 0.5 rad for drum number 3. Finally, this process is repeated for all performance times in the music file, determining the head position for each frame to obtain the head movement sequence. This sequence includes the time and head position for each frame.

[0054] In step S103, based on at least one instrument being played and the head position at each performance time, the robot's playing part and head are controlled to perform playing actions, so that the corresponding instrument being played sounds at each performance time and the head is in the corresponding head position.

[0055] Since robots are not necessarily dedicated to playing musical instruments, and the performance time in music files is uneven, using a robot may not be suitable for a particular music file. In other words, directly playing an instrument according to the music file places high demands on the robot's motion control, and may even be impossible. Therefore, the music file can be pre-processed adaptively to enable the robot to adapt to it: If, during any performance time in the music file, any instrument exceeds the range of motion of its corresponding playing part, the playing part is updated to another playing part (whose range of motion covers the instrument and is currently idle). This means replacing actions that a humanoid robot cannot perform in the music file. For example, in a scenario where a robot is playing the drums, the robot can check which drumheads its hands cannot strike due to insufficient workspace and switch hands accordingly.

[0056] Please refer to the appendix. Figure 4 The example illustrates the process of adaptive preprocessing of music files, using a scenario of a robot playing the drums as an example.

[0057] After starting, the robot reads the notes for the i-th performance time in the music file, such as "In the 3rd second, drum 7 is struck by the left hand, drum 8 by the left foot, and drum 9 by the right foot." Then, it checks whether the drum to be struck by the left hand can be reached. If the left hand cannot strike the expected drum, i.e., drum 7, it checks whether the right hand is free. If the right hand is free, it lets the right hand strike, and the notes for that performance time are updated to "In the 3rd second, drum 7 is struck by the right hand, drum 8 by the left foot, and drum 9 by the right foot." If the right hand is not free (i.e., it has a striking task), it determines that the robot cannot complete the performance of the music file. Finally, the robot reads all the performance times of the music file in this way.

[0058] By adaptively preprocessing music files, any music file can be adapted for robot performance, thereby increasing the method's applicability to various music files and making it a universal method.

[0059] In one possible embodiment, it can be done as follows Figure 5 The method shown controls the robot's playing parts and head to perform playing actions, including steps S501 to S503.

[0060] In step S501, for each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time.

[0061] Taking a robot playing the drums as an example, the key to the humanoid robot's performance is ensuring accurate drumbeats, including simultaneous use of both hands, feet, and head to keep the rhythm, and maintaining fluid, natural movements during the performance. This step can convert the time stamps within the music file, facilitating subsequent control of the performance actions; for example, it can convert the first music file into a second music file.

[0062] The first music file is as follows: In the first second, drum 1 is struck by the right hand, drum 4 by the left hand, drum 8 by the left foot, and drum 9 by the right foot. The head turns 0.3 rad. At 1.1 seconds, drum 2 is struck by the right hand, and the head rotates to -0.3 rad; At 1.35 seconds, drum 6 is struck by the right hand, and the head rotates to -0.5 rad. At 1.7 seconds, drum 1 is struck by the right hand, drum 8 is struck by the left foot, drum 9 is struck by the right foot, and the head rotates by 0.3 rad. At 1.9 seconds, drum 2 is struck by the right hand, and the head rotates to -0.3 rad. At 2.2 seconds, drum 6 is struck by the right hand, and the head rotates 0.3 rad. At 2.3 seconds, drum 1 is struck by the right hand, drum 4 by the left hand, drum 8 by the left foot, and drum 9 by the right foot. The head turns to -0.3 rad. At 2.5 seconds, drum 2 is struck by the right hand and drum 3 is struck by the left hand, and the head turns 0.3 rad. At 2.75 seconds, drum 6 is struck by the right hand, and the head rotates to -0.3 rad.

[0063] The second music file is as follows: 1 second later, drum 1 is struck by the right hand; 1 second later, drum 4 is struck by the left hand; 1 second later, drum 8 is struck by the left foot; 1 second later, drum 9 is struck by the right foot; 1 second later, the head turns 0.3 rad. 0.1s later, drum 2 is struck by the right hand; 0.7s later, drum 8 is struck by the left foot; 0.7s later, drum 9 is struck by the right foot; 0.1s later, the head turns to -0.3rad. After 0.25 seconds, drum 6 is struck by the right hand; after 0.6 seconds, drum 4 is struck by the left hand, drum 8 is struck by the left foot, drum 9 is struck by the right foot; after 0.25 seconds, the head turns to -0.5 rad. After 0.35 seconds, drum 1 is struck by the right hand; after 0.2 seconds, drum 3 is struck by the left hand; after 0.35 seconds, the head turns 0.3 rad. 0.2 seconds later, drum 2 is struck by the right hand; 0.2 seconds later, the head turns to -0.3 rad. 0.3 seconds later, drum 6 is struck by the right hand; 0.3 seconds later, the head turns 0.3 rad. 0.1 seconds later, drum 1 is struck by the right hand; 0.1 seconds later, the head turns to -0.3 rad. 0.2 seconds later, drum 2 is struck by the right hand; 0.2 seconds later, the head turns 0.3 rad. 0.25 seconds later, drum 6 was struck by the right hand, and 0.25 seconds later, the head turned to -0.3 rad.

[0064] In step S502, for each instrument played at each performance time, a performance trajectory of the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory.

[0065] In one example, this step can be performed as follows: First, the duration of each stage of the performance action can be determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. The attributes of the instrument can include the duration proportion of each stage of the performance action; therefore, the duration of each stage can be determined based on the performance duration corresponding to the performance time and the duration proportion of each stage. For example, if the instrument being played is a drum, the duration of the lifting stage and the duration of the striking stage can be determined based on the performance duration corresponding to the performance time. Optionally, if the duration proportion of the striking stage is s, then the duration of the striking stage is the performance duration. s, the duration of the lifting phase is the performance duration. (1-s).

[0066] Next, based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, the playing trajectory of each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory of each stage in sequence.

[0067] Understandably, the performance duration corresponding to the performance time affects the trajectory generation. When the performance duration matches the speed of the playing part's movement well, the planned trajectory and performance movements are more flexible and vivid. When the performance duration and the speed of the playing part's movement are not well matched, the planned trajectory and performance movements are more mechanical and inflexible, with a lower degree of anthropomorphism. Therefore, in another example, when performing this step, we can analyze whether the performance duration and movement speed match, and adjust them if they do not match, that is, perform this step in the following way: First, the action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time.

[0068] For example, if the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit.

[0069] For example, if the performance duration of at least one of the instruments being played corresponding to the performance time is within the action time range of the corresponding playing part of the instrument being played, the action duration is determined to be the performance duration of at least one of the instruments being played corresponding to the performance time, and the waiting time is determined to be 0.

[0070] For example, if the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

[0071] Next, based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a performance trajectory for the performance action is generated, and the playing part corresponding to the instrument being played is controlled to execute the performance trajectory within the duration of the action. That is, the performance action is executed only during the duration of the action, while remaining stationary during the waiting period, thus matching and adapting the duration of the performance action to the speed of the playing part. The waiting period can be before, after, or within the working period.

[0072] It is understood that this step can be performed according to either of the two examples above, or in combination of the two examples. For example, first determine the action duration and the waiting duration, then determine the duration of each stage based on the action duration, then generate the trajectory for each stage based on the duration of each stage, and finally control the playing part to execute the trajectory of each stage, and arrange the waiting time between each stage. For example, in the scenario of a robot playing a drum kit, the waiting time can be arranged between the lifting stage and the striking stage, that is, first execute the trajectory of the lifting stage, then wait for the static period, and then execute the trajectory of the striking stage.

[0073] In step S503, for each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

[0074] In steps S502 and S503, polynomial interpolation can be used to generate the trajectory.

[0075] Understandably, when executing steps S502 and S503, after completing a performance action for one performance time, it can be determined whether the current time exceeds the total duration of the music file. If it does not exceed the total duration, the performance action for the next performance time can be executed. If it exceeds the total duration, the performance can be completed and the robot can be controlled to return to a stationary posture.

[0076] Please refer to the appendix. Figure 6 Using a robot playing the drums as an example, it exemplifies the process of controlling the playing parts and head to perform playing actions. The specific details of each step involved are already described in [reference needed]. Figure 5 The embodiments shown are described in detail and will not be repeated here.

[0077] The motion control method provided in this disclosure acquires a music file to be played, which yields multiple performance times and at least one instrument to be played at each performance time. Based on this, the head position for each performance time is determined. Finally, based on the at least one instrument and the head position at each performance time, the robot's playing part and head are controlled to perform performance actions, causing the corresponding instrument to sound and the head to be in the corresponding position at each performance time. The performance time is the duration of instrument sound. Because the head position for each performance time is determined, the robot's head moves rhythmically during the performance, increasing the diversity of movements and improving anthropomorphism, overcoming problems such as insufficient dynamic characteristics and poor performance effects.

[0078] Moreover, the motion control method provided in this disclosure is applicable to humanoid robot drum playing, has strong versatility, and does not require the design of a special robot or hardware for drum playing; it achieves coordinated control of the humanoid robot's head, hands, and feet, and the playing effect is better than the playing effect of special robots or simple dual arms with few degrees of freedom in related technologies; the automated generation and execution of the trajectories of hands, feet, head, etc. have a high degree of automation and intelligence.

[0079] According to a second aspect of the present disclosure, a motion control device is provided for use in a robot; please refer to the attached drawing. Figure 7 The device includes: The acquisition module 701 is used to acquire a music file to be played, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time. The determining module 702 is used to determine the head position for each performance time based on at least one instrument being played at each performance time. The motion module 703 is used to control the robot's playing part and head to perform playing actions based on at least one instrument being played and the head position at each playing time, so as to make the corresponding instrument being played sound and to position the head in the corresponding head position at each playing time.

[0080] In some embodiments of this disclosure, the determining module is specifically used for: If at least one of the instruments being played is in a different direction at any given performance time, the head position for that performance time is determined in the direction opposite to the direction to which the head position of the previous performance time belongs. If at least one of the instruments being played is in the same direction at any given time, the head position for that time is determined based on the position of the at least one instrument being played.

[0081] In some embodiments of this disclosure, when the determining module is used to determine the head position for the current performance time in a direction relative to the direction of the head position in the previous performance time, it is specifically used for: The head position for the current performance time is determined from any one of at least one preset position in the direction opposite to the head position of the previous performance time.

[0082] In some embodiments of this disclosure, when the determining module is used to determine the head position corresponding to the performance time based on the position of the at least one played instrument, it is specifically used for: If there is a playing instrument during the performance time, the head position corresponding to the playing instrument is determined as the head position during the performance time. When multiple instruments are played during a performance, the midpoint of the head position corresponding to each of the multiple instruments is determined as the head position during that performance.

[0083] In some embodiments of this disclosure, the music file also includes at least one playing instrument at each performance time, and the playing part corresponding to each playing instrument. The device further includes an update module for: If any of the instruments played at any time in the music file exceeds the range of motion of the corresponding playing part, the playing part of the instrument will be updated to another playing part.

[0084] In some embodiments of this disclosure, the action module is specifically used for: For each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time; For each instrument played at each performance time, the performance trajectory of the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory. For each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

[0085] In some embodiments of this disclosure, the action module is used to generate a performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The duration of each stage of the performance action is determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, a playing trajectory for each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory for each stage in sequence.

[0086] In some embodiments of this disclosure, when the action module is used to determine the duration of each stage of the performance action based on the attributes of the instrument being played and the performance duration corresponding to the performance time, it is specifically used for: When the instrument being played is a drum, the duration of the lifting phase and the duration of the striking phase are determined based on the performance duration corresponding to the performance time.

[0087] In some embodiments of this disclosure, the action module is used to generate a performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a playing trajectory for the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory within the duration of the action.

[0088] In some embodiments of this disclosure, when the action module is used to determine the action duration and waiting duration based on the action time range of the playing part corresponding to the played instrument and the performance duration corresponding to the performance time, it is specifically used for: If the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit. If the performance duration of at least one instrument being played corresponding to the performance time is within the action time range of the corresponding playing part of the instrument being played, the action duration is determined to be the performance duration of at least one instrument being played corresponding to the performance time, and the waiting time is determined to be 0. If the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

[0089] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method in the first aspect, and will not be elaborated upon here.

[0090] Thirdly, at least one embodiment of this disclosure provides a robot, please refer to the appendix. Figure 8 The diagram illustrates the structure of the robot, which includes a memory and a processor. The memory stores computer instructions that can run on the processor, and the processor controls motion based on the method described in any of the first aspects when executing the computer instructions.

[0091] Fourthly, at least one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects.

[0092] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0093] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A motion control method, characterized in that, Applied to robots, the method includes: Obtain a music file to be played, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time; The head position for each performance time is determined based on at least one instrument played at each performance time. Based on at least one instrument being played and the head position at each performance time, the robot's playing part and head are controlled to perform playing actions, so as to make the corresponding instrument being played sound at each performance time and to make the head position the corresponding head position. Each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The step of determining the head position for each performance time based on at least one instrument being played includes: If at least one of the directions marked by the instrument being played is different at any given performance time, and the head position at the previous performance time is in the first direction of the original head position, the head position at that performance time is determined in the second direction of the original head position.

2. The motion control method according to claim 1, characterized in that, Determining the head position for the performance time in the second direction from the original head position includes: The head position at the performance time is determined by any one of at least one preset position in the second direction of the original head position.

3. The motion control method according to claim 1, characterized in that, Each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The step of determining the head position for each performance time based on at least one instrument being played includes: If at least one of the instruments being played is marked in the same direction at any given performance time, the head position for that performance time is determined based on the position of the at least one instrument being played.

4. The motion control method according to claim 3, characterized in that, Determining the head position corresponding to the performance time based on the position of the at least one instrument being played includes: If there is a playing instrument during the performance time, the head position corresponding to the playing instrument is determined as the head position during the performance time. When multiple instruments are played during a performance, the midpoint of the head position corresponding to each of the multiple instruments is determined as the head position during that performance.

5. The motion control method according to claim 1, characterized in that, The music file also includes the playing part of at least one instrument played at each performance time; The method further includes: If any of the instruments played at any time in the music file exceeds the range of motion of the corresponding playing part, the playing part of the instrument will be updated to another playing part.

6. The motion control method according to claim 5, characterized in that, The step of controlling the robot's playing part and head to perform playing actions based on at least one instrument being played and the head position at each playing time, so as to make the corresponding instrument play sound at each playing time and to position the head in the corresponding head position, includes: For each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time; For each instrument played at each performance time, a performance trajectory for the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory. For each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

7. The motion control method according to claim 6, characterized in that, The step of generating the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and controlling the playing part corresponding to the instrument being played to execute the performance trajectory, includes: The duration of each stage of the performance action is determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, a playing trajectory for each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory for each stage in sequence.

8. The motion control method according to claim 7, characterized in that, The step of determining the duration of each stage of the performance action based on the attributes of the instrument being played and the performance duration corresponding to the performance time includes: When the instrument being played is a drum, the duration of the lifting phase and the duration of the striking phase are determined based on the performance duration corresponding to the performance time.

9. The motion control method according to claim 6, characterized in that, The step of generating the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and controlling the playing part corresponding to the instrument being played to execute the performance trajectory, includes: The action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a playing trajectory for the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory within the duration of the action.

10. The motion control method according to claim 9, characterized in that, The step of determining the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit.

11. The motion control method according to claim 9, characterized in that, The step of determining the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is within the action time range of the corresponding playing part of the instrument, the action duration is determined to be the performance duration of at least one instrument corresponding to the performance time, and the waiting time is determined to be 0.

12. The motion control method according to claim 9, characterized in that, The step of determining the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument and the performance duration corresponding to the performance time includes: If the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

13. A motion control device, characterized in that, The device, applied to robots, includes: The acquisition module is used to acquire the music file to be played, wherein the music file includes multiple performance times and at least one instrument to be played in each performance time; The determination module is used to determine the head position for each performance time based on at least one instrument being played at each performance time. The motion module is used to control the robot's playing part and head to perform playing actions based on at least one instrument being played and the head position at each playing time, so as to make the corresponding instrument being played sound and to position the head in the corresponding head position at each playing time. Each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The determining module is specifically used for: If at least one of the directions marked by the instrument being played is different at any given performance time, and the head position at the previous performance time is in a first direction of the original head position, the head position at that performance time is determined in a second direction of the original head position.

14. The motion control device according to claim 13, characterized in that, When determining the head position at the performance time within the second direction of the original head position, the determining module is specifically used for: The head position at the performance time is determined by any one of at least one preset position in the second direction of the original head position.

15. The motion control device according to claim 13, characterized in that, Each instrument being played is marked with a first direction or a second direction, wherein the first direction and the second direction are opposite directions; The determining module is specifically used for: If at least one of the instruments being played is marked in the same direction at any given performance time, the head position for that performance time is determined based on the position of the at least one instrument being played.

16. The motion control device according to claim 15, characterized in that, When the determining module determines the head position corresponding to the performance time based on the position of the at least one played instrument, it is specifically used for: If there is a playing instrument during the performance time, the head position corresponding to the playing instrument is determined as the head position during the performance time. When multiple instruments are played during a performance, the midpoint of the head position corresponding to each of the multiple instruments is determined as the head position during that performance.

17. The motion control device according to claim 13, characterized in that, The music file also includes the playing part of at least one instrument played at each performance time; The device further includes an update module for: If any of the instruments played at any time in the music file exceeds the range of motion of the corresponding playing part, the playing part of the instrument will be updated to another playing part.

18. The motion control device according to claim 17, characterized in that, The action module is specifically used for: For each performance time, the time interval between the performance time and the previous performance time is determined as the performance duration corresponding to the performance time; For each instrument played at each performance time, a performance trajectory for the performance action is generated based on the current position of the end of the playing part corresponding to the instrument played, the position of the instrument played, and the performance duration corresponding to the performance time, and the playing part corresponding to the instrument played is controlled to execute the performance trajectory. For each performance time, a head trajectory for the performance action is generated based on the current head position, the head position during the performance time, and the performance duration corresponding to the performance time, and the head is controlled to execute the head trajectory.

19. The motion control device according to claim 18, characterized in that, The action module is used to generate the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The duration of each stage of the performance action is determined based on the attributes of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of each stage of the playing action, a playing trajectory for each stage of the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory for each stage in sequence.

20. The motion control device according to claim 19, characterized in that, The action module, when determining the duration of each stage of the performance action based on the attributes of the instrument being played and the performance duration corresponding to the performance time, is specifically used for: When the instrument being played is a drum, the duration of the lifting phase and the duration of the striking phase are determined based on the performance duration corresponding to the performance time.

21. The motion control device according to claim 18, characterized in that, The action module is used to generate the performance trajectory of the performance action based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the performance duration corresponding to the performance time, and to control the playing part corresponding to the instrument being played to execute the performance trajectory, specifically for: The action duration and waiting duration are determined based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Based on the current position of the end of the playing part corresponding to the instrument being played, the position of the instrument being played, and the duration of the action, a playing trajectory for the playing action is generated, and the playing part corresponding to the instrument being played is controlled to execute the playing trajectory within the duration of the action.

22. The motion control device according to claim 21, characterized in that, The action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Specifically, it is used for: If the performance duration of at least one instrument corresponding to the performance time is greater than the upper limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the upper limit, and the waiting time is determined to be the difference between the performance duration of at least one instrument corresponding to the performance time and the upper limit.

23. The motion control device according to claim 22, characterized in that, The action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Specifically, it is used for: If the performance duration of at least one instrument corresponding to the performance time is within the action time range of the corresponding playing part of the instrument, the action duration is determined to be the performance duration of at least one instrument corresponding to the performance time, and the waiting time is determined to be 0.

24. The motion control device according to claim 22, characterized in that, The action module is used to determine the action duration and waiting duration based on the action time range of the corresponding playing part of the instrument being played and the performance duration corresponding to the performance time. Specifically, it is used for: If the performance duration of at least one instrument corresponding to the performance time is less than the lower limit of the action time range of the corresponding playing part of the instrument, the action time is determined to be the lower limit, and the waiting time is determined to be 0.

25. A robot, characterized in that, The robot includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the motion control method of any one of claims 1 to 12 when executing the computer instructions.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 12.

Images