Guitar playing simulation animation generation method and device and medium
Through in -depth strengthening of learning and reverse sports technology, controlling the string and strings of the machine's hand to solve the problem of insufficient natural and accuracy of guitar performance animation in computer animation, and achieve high -quality and low -cost animation generation.
Patent Information
- Application Number
- CN202411378093.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-24
AI Technical Summary
In the field of computer animation, traditional methods are difficult to generate high-quality guitar performance animation, especially when simulating fine hand movements, there are problems of insufficient naturalness and precision.
Through deeply strengthening the study of neural networks, combined with reverse sports calculation of the joint angle, control the pressing and dialing action of the machine hand, thereby generating real and natural guitar playing animations.
The highly realistic guitar played animation, reduced the time and labor cost of animation, and enhanced the adaptability and diversity of animation.
Smart Images
Figure CN119359876A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of computer animation, and in particular to a method, device and medium for generating guitar performance simulation animation. Background Art
[0002] At present, in the field of computer animation, there are certain research results in simulating real physical effects, character animation and complex scene rendering, but there are still many challenges in creating multi-fingered animated characters with high flexibility and precision. When it comes to the animation production of artistic performances such as simulating fine hand movements such as guitar playing, there is still a big gap between the animation produced by traditional methods and the real human performance movements.
[0003] Traditionally, computer animators rely on manual keyframe animation or motion capture technology to create realistic hand movements. However, keyframe animation requires animators to have superb skills and a lot of time investment, and it is difficult to achieve the naturalness and accuracy comparable to human finger movements. Although motion capture technology can capture the movements of real actors, it is expensive, data acquisition is limited, and for non-standard or highly specialized movements, such as special fingering in guitar playing, the captured data may not be directly applied to animation. In order to overcome these limitations, in recent years, the application of deep learning, reinforcement learning and other technologies to the field of computer animation has begun to be explored to optimize the motion planning and control strategies of animated characters. However, research on simulating fine hand movements such as guitar playing in computer animation is still in its infancy. Therefore, how to automatically generate high-quality hand animation while reducing manual intervention and dependence has become a technical problem that needs to be solved. Summary of the invention
[0004] The purpose of the present invention is to provide a method, device and medium for generating guitar playing simulation animation in order to overcome the defects of the above-mentioned prior art. Guitar playing animation is generated by fingering generation, string pressing control and string plucking action control. The neural network is trained by using deep reinforcement learning to control the left hand to press the strings. The angles of the joints are calculated by inverse kinematics to generate the action of the machine's right hand plucking the strings, thereby obtaining a realistic and natural guitar playing animation.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] According to one aspect of the present invention, there is provided a method for generating a guitar performance simulation animation, comprising the following steps:
[0007] S1. Generate fingerings for pressing and plucking strings that are in line with human movement habits based on the six-line score;
[0008] S2, based on the string-pressing fingering, controlling the left hand of the animation machine to press the required frets through reinforcement learning training;
[0009] S3. Based on the plucking fingering, plan the trajectory of the right hand plucking the string, and then calculate the joint angles through inverse kinematics to control the right hand plucking the string.
[0010] Furthermore, the specific steps of S1 include: creating a standard six-line staff class, including note names, durations, and corresponding playing fingers; searching for currently available fingers through a brute force search algorithm to avoid finger conflicts; if a single fret needs to be pressed, calculating the distance between the finger and the target fret at each moment, and selecting the finger closest to the target fret; if multiple frets need to be pressed at the same time, commanding the low frets to be pressed by the right finger and the high frets to be pressed by the left finger through a sorting algorithm.
[0011] Furthermore, the specific steps of S2 include: creating a guitar playing simulation reinforcement learning environment based on a 3D physics engine; rewarding and shaping the robot hand so that the robot hand presses the target fret; and training using a reinforcement learning algorithm.
[0012] The establishment of the reinforcement learning environment includes: importing the guitar model into the 3D physics engine; installing a virtual sensor on each fret of the guitar; if the fret is needed but not pressed at the current moment, the sensor displays red, and if the fret is pressed at the current moment, the sensor displays green.
[0013] The reward shaping includes rewarding the distance between the responsible finger and the target grade, the smaller the distance, the greater the reward; rewarding the finger for continuously pressing the corresponding target grade, and punishing the finger for pressing the wrong grade; punishing the torque of the hand to maintain the stability of the hand.
[0014] Furthermore, the S3 string-plucking fingering method includes: the thumb is responsible for plucking the 6th, 5th, and 4th strings, the index finger is responsible for plucking the 3rd string, the middle finger is responsible for plucking the 2nd string, and the ring finger is responsible for plucking the 1st string; the trajectory of the thumb plucking the strings is approximated as a straight line; the trajectory of the index finger, middle finger, and ring finger plucking the strings is approximated as an ellipse; the angle corresponding to the finger joints at each moment is calculated by inverse kinematics to control the string plucking. The trajectory of the thumb plucking the strings starts at the upper left of the target string and ends at the lower right of the position of the target string, and the strings are plucked in a straight line trajectory. The trajectory of the index finger, middle finger, and ring finger plucking the strings uses the target string as the string-plucking point, and the lower front position of the string-plucking point as the center of the string-plucking ellipse trajectory. A corresponding ellipse is generated according to the center position, so that the string-plucking point is located on the ellipse, and the spatial position of the ellipse is changed through a matrix to adapt to the inclination angle of the hand.
[0015] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.
[0016] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.
[0017] Compared with the prior art, the present invention has the following beneficial effects:
[0018] (1) Improve the realism of animation: Deep reinforcement learning is used to train neural networks and inverse kinematics to control the pressing and plucking of strings by the robot arm. The robot arm is precisely controlled to generate highly realistic and natural guitar playing animations.
[0019] (2) Reduced time and labor costs for animation production: Compared with traditional manual keyframe animation and motion capture technology, the present invention greatly reduces the cost and time of animation production. No expensive equipment and motion capture personnel are required. High-quality guitar playing animation can be generated through simple input and automated processing.
[0020] (3) Enhance the adaptability of guitar playing simulation animation: By adjusting parameters and algorithms, six-line tabs of different difficulty and style can be easily modified to meet the diverse needs of guitar playing animation. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A flowchart of a method for generating a guitar playing simulation animation;
[0022] Figure 2 is the reward mapping function image;
[0023] Figure 3 The performance statistics of indicators for different song training sets are shown below;
[0024] Figure 4 This is the rendering of the guitar playing effect. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0026] like Figure 1 As shown, a method for generating a guitar performance simulation animation comprises the following steps:
[0027] S1. Generate fingerings for pressing and plucking strings that are in line with human movement habits based on the six-line score;
[0028] S2, based on the string-pressing fingering, controlling the left hand of the animation machine to press the required frets through reinforcement learning training;
[0029] S3. Based on the plucking fingering, plan the trajectory of the right hand plucking the string, and then calculate the joint angles through inverse kinematics to control the right hand plucking the string.
[0030] The specific steps of S1 include: creating a standard six-line staff class, including note names, durations, and corresponding playing fingers; searching for currently available fingers through a brute force search algorithm to avoid finger conflicts; if a single fret needs to be pressed, calculating the distance between the finger and the target fret at each moment, and selecting the finger closest to the target fret; if multiple frets need to be pressed at the same time, using a sorting algorithm to command the low frets to be pressed by the right finger and the high frets to be pressed by the left finger.
[0031] In this embodiment, each note in the six-line stave is converted into a class in the computer, which includes the start time, duration and target fret position of each note. The notes are sorted according to the start time of the notes to ensure that the finger assignment follows the time order to avoid conflicts. The last used time variable is set for each finger of each hand (such as the index finger, middle finger, ring finger and little finger of the left hand, numbered 1 to 4 respectively) and initialized to 0. For each sorted note, the start time of the current note is compared with the last used time of each finger. If the last used time of a finger is later than the start time of the current note, the finger is not available.
[0032] Select the finger closest to the target fret position from the available fingers. Specifically, if the current left hand position is P, the fret positions corresponding to each finger are P, P+1, P+2, and P+3. By calculating the absolute value difference between each finger position and the target fret position Pt, select the finger with the smallest difference for assignment. Assign the selected finger to the current note, and update its last used time to the start time of the current note plus the duration of the note.
[0033] When multiple notes share the same fret position, adjust finger allocation according to human playing habits: use the right-hand fingers (such as the ring finger and pinky finger) for lower frets, and use the left-hand fingers (such as the index finger and middle finger) for higher frets. Sort the currently available fingers and give priority to the fingers that meet the above habits.
[0034] The thumb of the right hand is responsible for plucking the sixth, fifth and fourth strings; the index finger, middle finger and ring finger are responsible for plucking the third, second and first strings respectively. The right hand plucking fingering is generated according to the string where each note on the six-line staff is located.
[0035] The specific steps of S2 include: creating a guitar playing simulation reinforcement learning environment based on a 3D physics engine; rewarding the robot hand, including distance reward, fret reward and energy reward, so that the robot hand can press the target fret; and using the DroQ reinforcement learning algorithm for training. The establishment of the reinforcement learning environment includes importing the guitar model into the 3D physics engine; installing virtual sensors on each fret of the guitar; if the fret is needed but not pressed at the current moment, the sensor displays red, and if the fret is pressed at the current moment, the sensor displays green. Distance Reward is a reward for the distance between the responsible finger and the target fret, and the smaller the distance, the greater the reward; Fret Reward is a reward for the finger to continue to press the corresponding target fret, and a penalty is imposed for pressing the wrong fret; Energy Reward is a penalty for the opponent's torque to maintain hand stability. The DroQ algorithm is an advanced algorithm that can handle continuous action space. Compared with the commonly used reinforcement learning algorithm, it introduces a hierarchical decision-making process and can more effectively handle state space exploration in complex environments. At the same time, it has a high data sampling rate to reduce sample efficiency issues and performs outstandingly in scenarios with sparse rewards.
[0036] In this embodiment, in the virtual simulation environment, a pressure sensor is installed for each fret on each string of the guitar to accurately detect the pressing of the finger on the scale during the simulation process. When the pressure of the finger on the fret exceeds a set threshold, the fret is considered to be successfully pressed.
[0037] Define the observation space and action space of the environment, where the observation space is composed of the position of the finger, whether the finger is available, the finger that needs to be pressed at the current moment, the angle of the finger joint, the position of the forearm, and the action and reward of the previous frame. The action space is the target angle of the hand joint and the position of the hand. At the same time, shape the reward function to control the left hand to press the target finger quickly, accurately and smoothly, which is composed of Distance Reward, Fret Reward and Energy Reward. Distanc e Reward is used to motivate the finger to move toward the target grade, and the expression is:
[0038] r x =m(|f x -k x |, 0.5W fret , 0.1),
[0039] r y =m(|f y -k y |, 0.001, 0.1),
[0040] =m(|f z -k z |, 0.001, 0.1),
[0041] Among them, m is the reward mapping function, such as Figure 2 As shown, W fret is the width of the current target grade, (f x , f y , f z ) is the position of the fingertip, (k x , k y ,k z ) The center of the taste, (r x , r y ,r z ) is the reward along each coordinate axis.
[0042] Fret Reward, which rewards the finger for continuously pressing the target position and punishes the finger for pressing a non-target position, is expressed as:
[0043]
[0044] Among them, fs is the normalized pressing force on each grade, N is the number of grades that need to be pressed by the finger at the current moment, 1 {false positive} When any prohibited position is pressed by a finger, its value is 1, otherwise it is 0.
[0045] Energy Reward penalizes the opponent's torque.
[0046] r energy =-c||τ joints | T |v joints |,
[0047] Where c is a coefficient with a value of 0.0025, and when there are no notes to play, the value is 0.005.
[0048] The total reward expression is:
[0049] r=0.5r fret +0.5r distance +r energy .
[0050] Finally, the DroQ reinforcement learning algorithm is used for training, and the action strategy is continuously adjusted according to reward feedback until the robot fingers can achieve accurate and stable string-pressing actions in a virtual environment.
[0051] S3 string plucking fingering includes: the thumb is responsible for plucking the 6th, 5th, and 4th strings, the index finger is responsible for plucking the 3rd string, the middle finger is responsible for plucking the 2nd string, and the ring finger is responsible for plucking the 1st string; the trajectory of the thumb plucking the strings is approximated as a straight line; the trajectory of the index finger, middle finger, and ring finger plucking the strings is approximated as an ellipse; the angle corresponding to the finger joint at each moment is calculated by inverse kinematics to control the plucking. The trajectory of the thumb plucking the strings starts at the upper left of the target string and ends at the lower right of the position of the target string, and the strings are plucked in a straight line trajectory. The trajectory of the index finger, middle finger, and ring finger plucking the strings uses the target string as the plucking point, and the lower front position of the plucking point as the center of the plucking ellipse trajectory. The corresponding ellipse is generated according to the center position, so that the plucking point is located on the ellipse, and the spatial position of the ellipse is changed through a matrix to adapt to the inclination angle of the hand. The inverse kinematics calculation calculates the angle corresponding to the finger joint according to the position of the finger end, and can be solved using gradient descent or Newton iteration.
[0052] In order to further verify the feasibility and effect of the guitar performance simulation animation generation technology in the embodiment, the six-line score from the Internet is used as the training data set, and the F1 index is used to evaluate the integrity and accuracy of the system playing the song. The results show that for most of the simple to medium difficulty songs, the system can successfully play most of the notes, so that the audience can recognize the songs played. The training set index performance is as follows: Figure 3 As shown, the final visual effect of the animation is as follows Figure 4 shown.
[0053] The guitar playing simulation animation method, device and medium disclosed in the present invention can be applied in the VR field, and in the VR scene, it is shown as follows: there is a digital person NPC in the VR scene, and the user can let the NPC play the guitar with corresponding actions by inputting a guitar six-line score.
[0054] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0055] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0056] A plurality of components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disk, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the method of the present invention in any other appropriate manner (e.g., by means of firmware).
[0057] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0058] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0059] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0060] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for generating a guitar performance simulation animation, characterized in that: The following steps are involved: S1. Generate fingerings for pressing and plucking strings that are in line with human movement habits based on the six-line score; S2, based on the string-pressing fingering, controlling the left hand of the animation machine to press the required frets through reinforcement learning training; S3. Based on the plucking fingering, plan the trajectory of the right hand plucking the string, and then calculate the joint angles through inverse kinematics to control the right hand plucking the string.
2. A method for generating a guitar performance simulation animation according to claim 1, characterized in that: The specific steps of S1 include: Create a standard six-line staff class, including note names, durations, and corresponding playing fingers; search for currently available fingers through a brute force search algorithm to avoid finger conflicts; if you need to press a single fret, calculate the distance between the finger and the target fret at each moment, and select the finger closest to the target fret; if you need to press multiple frets at the same time, use a sorting algorithm to order the low frets to be pressed by the right finger, and the high frets to be pressed by the left finger.
3. A method for generating a guitar performance simulation animation according to claim 1, characterized in that: The specific steps of S2 include: Create a guitar playing simulation reinforcement learning environment based on a 3D physics engine; reward and shape the robot hand so that it presses the target fret; use reinforcement learning algorithm for training.
4. A method for generating a guitar performance simulation animation according to claim 3, characterized in that: The establishment of the reinforcement learning environment includes: importing the guitar model into the 3D physics engine; installing a virtual sensor on each fret of the guitar; if the fret is needed but not pressed at the current moment, the sensor displays red, and if the fret is pressed at the current moment, the sensor displays green.
5. A method for generating a guitar performance simulation animation according to claim 3, characterized in that: The reward shaping includes rewarding the distance between the responsible finger and the target grade, the smaller the distance, the greater the reward; rewarding the finger for continuously pressing the corresponding target grade, and punishing the finger for pressing the wrong grade; punishing the torque of the hand to maintain the stability of the hand.
6. The method for generating a guitar performance simulation animation according to claim 1, wherein: The S3 plucking fingering method includes the thumb plucking the 6th, 5th and 4th strings, the index finger plucking the 3rd string, the middle finger plucking the 2nd string, and the ring finger plucking the 1st string. The trajectory of the thumb plucking the strings is approximated as a straight line; the trajectory of the index finger, middle finger and ring finger plucking the strings is approximated as an ellipse; the corresponding angle of the finger joints at each moment is calculated through inverse kinematics to control the plucking of the strings.
7. A method for generating a guitar performance simulation animation according to claim 6, characterized in that: The trajectory of the thumb plucking the string starts at the upper left of the target string and ends at the lower right of the position of the target string, and the string is plucked along a straight trajectory.
8. The method for generating a guitar performance simulation animation according to claim 6, wherein: The trajectory of the index finger, middle finger and ring finger plucking the string uses the target string as the plucking point, and the position below and in front of the plucking point as the center of the plucking elliptical trajectory. A corresponding ellipse is generated based on the position of the center of the circle, so that the plucking point is located on the ellipse, and the spatial position of the ellipse is changed through a matrix to adapt to the tilt angle of the hand.
9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.