Hand motion generation method and device for musical instrument playing and medium

Abstract

The invention discloses a hand motion generation method and device for musical instrument playing and a medium. The method comprises the steps of obtaining a target audio clip; inputting the target audio clip into the basic model, and extracting the target audio clip by using an audio detail extractor to obtain audio features; performing hand position prediction on the audio features by using a hand position predictor to obtain a hand prediction position sequence; and inputting the target audio clip and the hand prediction position sequence into a gesture generator for multi-step inverse diffusion processing to obtain de-noised gesture features, and obtaining a hand motion diagram output by the basic model based on the hand prediction position sequence and the de-noised gesture features. According to the invention, the accuracy of generating the hand motion diagram can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and computer vision technology, and in particular to a method, device and medium for generating hand movements for playing a musical instrument. Background Art

[0002] Artificial intelligence has significant potential in skill learning, especially in the field of musical instrument learning. For piano, players need to have a deep understanding of the relationship between music scores and body movements, and only through repeated practice can they master the piano fingering and hand movement techniques. In order to provide more intuitive piano learning guidance, the AI ​​piano guidance system came into being, which can generate 3D hand movement images corresponding to piano playing based on music, helping people learn piano better.

[0003] However, current piano instruction methods focus too much on the local fingering constraints of consecutive notes and ignore long-distance fingering relationships, resulting in relatively low accuracy of the resulting gestures, which do not quite meet the gesture requirements of actual performances. Summary of the invention

[0004] The present application provides a method, device and storage medium for generating hand movements for playing a musical instrument, which can improve the accuracy of generating hand movement graphs.

[0005] In a first aspect, the present application provides a method for generating hand movements for playing a musical instrument, wherein a basic model includes an audio detail extractor, a hand position predictor, and a gesture generator; the method includes:

[0006] Get the target audio segment;

[0007] Input the target audio clip into the basic model, and use the audio detail extractor to extract the target audio clip to obtain audio features;

[0008] The hand position predictor is used to predict the hand position based on the audio features to obtain a hand prediction position sequence;

[0009] The target audio clip and the predicted hand position sequence are input into the gesture generator for multi-step inverse diffusion processing to obtain the hand motion graph output by the basic model.

[0010] A further technical solution is that after obtaining the audio features, the method further includes:

[0011] Encoding the audio features using an audio detail extractor to obtain encoded audio features;

[0012] The hand position predictor is used to predict the hand position based on the audio features to obtain a hand prediction position sequence, including:

[0013] A hand position predictor is used to predict the hand position of the encoded audio features to obtain a hand prediction position sequence;

[0014] The target audio clip and the predicted hand position sequence are input into the gesture generator to obtain the hand motion graph output by the basic model, including:

[0015] The encoded audio features and the predicted hand position sequence are fed into the gesture generator to obtain the hand motion graph output by the base model.

[0016] A further technical solution is that the hand position predictor includes a hand position decoder and a linear mapping layer, and the hand position predictor is used to predict the hand position of the encoded audio features to obtain a hand prediction position sequence, including:

[0017] Inputting the encoded audio features into a hand position decoder, and using the hand position decoder to decode the encoded audio features to obtain latent features;

[0018] A linear mapping layer is used to map the latent features to output positions, resulting in a sequence of predicted hand positions.

[0019] A further technical solution is that the gesture generator includes a time embedding layer and a gesture decoder, and the encoded audio features and the hand predicted position sequence are input into the gesture generator to obtain the hand motion graph output by the basic model, including:

[0020] Inputting the encoded audio features into a gesture decoder to obtain gesture features output by the gesture decoder;

[0021] Inputting a preset diffusion time step number into a time embedding layer to obtain time step embedding information output by the time embedding layer;

[0022] Input the time step embedding information, gesture features and hand predicted position sequence into the trained denoising model to obtain denoised gesture features;

[0023] Based on the predicted hand position sequence and denoised gesture features, the hand motion graph output by the basic model is obtained.

[0024] A further technical solution is that the preset noise sample is obtained in the following manner:

[0025] Get the original samples to be trained;

[0026] The forward diffusion process is used to construct a Markov chain. Within a preset time step, noise is gradually added to the original samples to be trained to obtain preset noise samples.

[0027] The further technical solution is that the network training method of the basic model includes:

[0028] Feed the encoded audio features into the hand position predictor;

[0029] The encoded audio features are decoded using a gesture decoder to obtain latent features;

[0030] Use a linear mapping layer to map the latent features to output positions to obtain a sequence of predicted hand positions;

[0031] Based on the hand prediction position sequence and the hand actual position, determine the position loss value and the speed loss value;

[0032] The position predictor is trained using the position loss and velocity loss values.

[0033] A further technical solution is to determine the position loss value and the speed loss value based on the hand predicted position sequence and the hand real position, including:

[0034] The Manhattan loss function is used to calculate the Manhattan distance between the hand prediction position sequence and the actual hand position to obtain the position loss value;

[0035] The Euclidean loss function is used to calculate the Euclidean distance between the predicted hand position sequence and the actual hand position to obtain the speed loss value.

[0036] A further technical solution is that the network training method of the basic model also includes:

[0037] The trained hand position predictor is used to guide the gesture generator to perform velocity prediction training.

[0038] In a second aspect, the present application provides a device for generating hand movements for playing a musical instrument, characterized in that the device for generating hand movements for playing a musical instrument comprises a unit for executing any of the above methods.

[0039] In a third aspect, the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the steps of any of the above methods.

[0040] The beneficial effects of the present application are as follows: different from the prior art, the present application can deeply analyze the subtle features such as rhythm, pitch, intensity, etc. in the target audio clip through an audio detail extractor, so as to accurately capture the complex nonlinear relationship between music and hand movements, reflect the subtle changes in the music in the fine-tuning of hand movements, improve the realism of hand movement generation, and use the hand predicted position sequence estimated by the position predictor to guide the gesture generator, so that the gesture generator generates a coherent and smooth hand movement path.

[0041] In addition, since the gesture generator adopts multi-step inverse diffusion processing technology, combined with the target audio clips, predicted hand positions and specific noise samples, it gradually refines and optimizes the hand movement graph, thereby improving the accuracy of hand movement graph generation, making the final generated hand movement graph more delicate and natural, and able to show continuous and expressive gesture changes, which is more in line with the gesture requirements of real performances. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0043] Figure 1 It is a flowchart of the first embodiment of the method for generating hand movements for playing a musical instrument provided by the present application;

[0044] Figure 2 It is a processing flow diagram of the basic model provided by this application;

[0045] Figure 3 is a schematic diagram of the processing flow of the gesture generator provided by this application;

[0046] Figure 4 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] Artificial intelligence has significant potential in skill learning, especially in the field of musical instrument learning. For piano, players need to have a deep understanding of the relationship between music scores and body movements, and only through repeated practice can they master the piano fingering and hand movement techniques. In order to provide more intuitive piano learning guidance, the AI ​​piano guidance system came into being, which can generate 3D hand movement images corresponding to piano playing based on music, helping people learn piano better.

[0050] Since the hand movements for the same note are different in different melodies, it is very important to capture the nonlinear relationship between music and the complex hand movements during piano playing in the generation of piano hand movements.

[0051] However, current piano instruction applications are mainly divided into two types: classifier-based prediction and rule-based. Among them, the classifier-based prediction method is good at generating static gestures and positions based on piano scores. Since it cannot capture the diversity and coherence of piano performance, it is difficult to construct a continuous and smooth hand movement trajectory; while the rule-based method is too focused on the local fingering constraints of consecutive notes, and ignores the long-distance fingering relationship, resulting in a relatively low accuracy of the obtained gestures, which does not meet the gesture requirements of real performances.

[0052] Therefore, in order to solve the above technical problems, the present application provides a method for generating hand movements for playing a musical instrument, which can improve the accuracy of generating a hand movement graph. Please refer to the following embodiments for details.

[0053] See also Figure 1 and Figure 2 , Figure 1 is a flow chart of the first embodiment of the method for generating hand movements for playing a musical instrument provided by the present application, Figure 2 It is a processing flow diagram of the basic model provided by this application.

[0054] The basic model includes an audio detail extractor, a hand position predictor, and a gesture generator; and the method includes the following steps:

[0055] Step 110: Obtain the target audio segment.

[0056] Step 120: Input the target audio segment into the basic model, and use the audio detail extractor to extract the target audio segment to obtain audio features.

[0057] Step 130: Use the hand position predictor to predict the hand position of the audio features to obtain a hand prediction position sequence.

[0058] Step 140: Input the target audio clip and the hand predicted position sequence into the gesture generator for multi-step inverse diffusion processing to obtain the hand movement graph output by the basic model.

[0059] This embodiment uses an audio detail extractor to deeply analyze subtle features such as rhythm, pitch, intensity, etc. in the target audio clip to accurately capture the complex nonlinear relationship between music and hand movements, and reflect subtle changes in music in the fine-tuning of hand movements, thereby improving the realism of hand movement generation. The predicted hand position sequence estimated by the position predictor is used to guide the gesture generator, so that the gesture generator generates a coherent and smooth hand movement path.

[0060] In addition, since the gesture generator adopts multi-step inverse diffusion processing technology, combined with the target audio clips, predicted hand positions and specific noise samples, it gradually refines and optimizes the hand movement graph, thereby improving the accuracy of hand movement graph generation, making the final generated hand movement graph more delicate and natural, and able to show continuous and expressive gesture changes, which is more in line with the gesture requirements of real performances.

[0061] In some embodiments, the obtained audio features may be encoded and then input into a hand position predictor for hand position prediction, see the second embodiment. The second embodiment includes the following steps:

[0062] Step 210: Obtain the target audio segment.

[0063] The target audio segment may be a piano audio segment A having N frames.

[0064] Step 220: Input the target audio segment into the basic model, and use the audio detail extractor to extract the target audio segment to obtain audio features.

[0065] See also Figure 2 The basic model may include an input layer 110, an audio detail extractor 120, a hand position predictor 130, a gesture generator 140, an output layer 150.

[0066] The target audio segment may be first input into the input layer 110 in the basic model, and then the audio detail extractor 120 may be used to extract the target audio segment to obtain audio features.

[0067] In some implementations, the audio detail extractor 120 may include an audio encoder 121 , and the audio encoder 121 may be used to encode the target audio segment.

[0068] Among them, the pre-trained audio detail extractor Φ can be used a Extract audio features f from piano audio clip A a ,for example Where C is Φ a The feature dimension in .

[0069] Step 230: Encode the audio features using an audio detail extractor to obtain encoded audio features.

[0070] Step 240: Use a hand position predictor to predict the hand position of the encoded audio features to obtain a hand prediction position sequence.

[0071] The hand prediction position sequence refers to the continuous position changes of the performer's hands over time during the performance of the piano or other musical instruments. That is, the hand prediction position sequence is a time series that covers the expected position arrangement of the hands on the instrument during the performance.

[0072] Specifically, the predicted hand position sequence can include finger placement, span changes, and various adjustments for playing different notes or chords. The hand position sequence emphasizes the dynamic process and overall layout, and is an important part of playing skills and musical expression.

[0073] In addition, the hand position predictor can distinguish between the left and right hands according to the different data sets, so that the final hand prediction position sequence contains the 3D coordinates of the left and right hands.

[0074] See also Figure 2 In some embodiments, the hand position predictor 130 includes a hand position decoder 131 and a linear mapping layer 132, and step 240 may include the following process:

[0075] 1) Inputting the encoded audio features into the hand position decoder, and using the hand position decoder to decode the encoded audio features to obtain latent features;

[0076] Among them, the feature embedding module can be selected as the hand position decoder to map the audio features to latent features with more comprehensive temporal information. For details, please refer to the following formula 1:

[0077]

[0078] Among them, Φ p is the hand position decoder, f a is the encoded audio feature, f p as potential characteristics.

[0079] In some embodiments, the feature embedding module can choose to use a Transformer model or SSM (State-Space Model). Among them, the Transformer uses self-attention mechanisms to effectively capture long-distance dependencies and contextual information, and then learn temporal relationships. In addition, an expression based on the S6 (Selective Structured State Space Model) model can also be used to replace the self-attention mechanism, and the previously input information can be effectively obtained through a structure similar to the classic RNN (Recurrent Neural Network).

[0080] 2) Use a linear mapping layer to map the latent features to output positions to obtain a sequence of predicted hand positions.

[0081] Step 250: Input the encoded audio features and the hand prediction position sequence into the gesture generator for multi-step inverse diffusion processing to obtain the hand movement graph output by the basic model.

[0082] Among them, the gesture generator can use a diffusion probabilistic model to capture the complex distribution of hand movements in a large-scale piano motion dataset.

[0083] Exemplarily, a preset noise sample can be used as the initial input of the inverse diffusion process, and the injected noise can be gradually reduced through a series of inverse diffusion steps. At the same time, the encoded audio features and hand predicted position sequence are used as conditional inputs to guide the generation process of the gesture generator to ensure that the generated hand motion map maintains consistency and coherence with the target audio features and hand predicted position sequence.

[0084] Among them, a model parameterized by a deep neural network, such as a denoising model, can be trained to obtain the inverse process of another Markov chain and learn the mapping p(Θ t-1 |Θ t ) The clean samples, i.e. the original samples to be trained, are gradually denoised within T steps.

[0085] Among them, the denoising model can include a 4-layer U-Net, each layer has 256, 512, 1024, and 2048 dimensions respectively.

[0086] In some embodiments, the gesture generator 140 includes a time embedding layer 141 and a gesture decoder 142. Step 250 may include the following sub-steps, see Figure 3 :

[0087] Step 251: Input the encoded audio features into a gesture decoder to obtain gesture features output by the gesture decoder.

[0088] For details, please refer to the following formula 2:

[0089] f g =Φ g (f a ), Formula 2.

[0090] Among them, f a can be the encoded audio feature, Φ g is the gesture decoder, f g The gesture feature.

[0091] By inputting the encoded audio features as conditions into the denoising model, the noise in the piano audio can be reduced.

[0092] Step 252: input the preset diffusion time step number into the time embedding layer to obtain the time step embedding information output by the time embedding layer.

[0093] Since the finger movements at different positions under the same pitch are different, the present application uses the hand position prediction sequence P generated by the hand position predictor 130 to guide the denoising model G. That is, in the denoising process, the time step embedding information of the preset diffusion time step number t can be connected in series.

[0094] Step 253: Input the time step embedding information, gesture features and hand predicted position sequence into the trained denoising model 143.

[0095] The preset noise sample is obtained in the following way:

[0096] 1) Obtain the original samples to be trained;

[0097] 2) The forward diffusion process is used to construct a Markov chain. Within a preset time step, noise is gradually added to the original samples to be trained to obtain preset noise samples.

[0098] Specifically, a Markov chain can be constructed through a forward diffusion process, where the clean sample θ of the inward motion sequence in a preset time step T steps is 0 That is, the original samples to be trained are gradually added with noise to generate a series of noise samples, namely the preset noise samples Θ 1 , ..., Θ T .

[0099] For example, the following formula 3 and formula 4 can be used:

[0100]

[0101]

[0102] in, is a Gaussian distribution, and β represents the variance scheduler of the added noise.

[0103] Step 254: Use the gesture features and the hand predicted position sequence to limit the denoising direction of the trained denoising model, so that the trained denoising model denoises the gesture features to obtain denoised gesture features.

[0104] For details, please refer to the following formula 5:

[0105]

[0106] Among them, G is the denoising model, P is the hand prediction position sequence, Θ t is the preset noise sample, The hand motion in N frames is represented by the denoised gesture feature, which contains the Euler angles of each joint of both hands.

[0107] t is the preset diffusion time step, and the denoising model connects the time step embedding information of the preset diffusion time step t in series.

[0108] Step 255: Based on the hand predicted position sequence and denoised gesture features, obtain the hand motion graph output by the basic model.

[0109] Among them, the hand movement graph includes the predicted hand position and gesture.

[0110] In addition, the network training method of the basic model provided in this application mainly includes the following two points:

[0111] (I) Use the position loss value and speed loss value to train the position predictor. For details, please refer to the following steps:

[0112] S1: Feed the encoded audio features into the hand position predictor.

[0113] S2: Use the gesture decoder to decode the encoded audio features to obtain latent features.

[0114] S3: Use a linear mapping layer to map the latent features to output positions to obtain a sequence of predicted hand positions.

[0115] S4: Based on the hand predicted position sequence and the hand actual position, determine the position loss value and the speed loss value.

[0116] Among them, S4 may include the following methods:

[0117] 1) Use the Manhattan loss function to calculate the Manhattan distance between the hand prediction position sequence and the actual hand position to obtain the position loss value;

[0118] Specifically, the position loss value can be calculated using the following formula 6:

[0119]

[0120] where is the position loss value, is the predicted hand position, P is the true hand position, and ||...|| 1 represents the L1 norm, or the Manhattan distance between two vectors.

[0121] 2) Calculate the Euclidean distance between the predicted hand position sequence and the true hand position using the Euclidean loss function to obtain the velocity loss value.

[0122] Specifically, the velocity loss value can be calculated using the following formula 7: 3)

[0124]

[0125] where is the velocity loss value, n represents the nth frame in the hand movement sequence, is the predicted hand velocity at the nth frame, is the predicted hand velocity at the (n - 1)th frame, P n is the true hand velocity at the nth frame, P n-1 is the true hand velocity at the (n - 1)th frame, and ||...|| 2 represents the L2 norm, or the Euclidean distance between two vectors in Euclidean space.

[0126] Among them, the time smoothness required for generating smooth motion can be deduced from the velocity loss value.

[0127] S5: Train the position predictor using the position loss value and the velocity loss value.

[0128] (2) Use the trained hand position predictor to guide the gesture generator for velocity prediction training.

[0129] That is, fix the parameters of the hand position predictor and use the hand position predicted by the hand position predictor to guide the gesture generator for velocity prediction training.

[0130] It should be noted that during the network training of the basic model, the parameters of the audio detail extractor can be fixed first.

[0131] In addition, since the pre-trained Wav2Vec2.0 and HuBERT models proposed by Facebook AI can use a large amount of unlabeled data for unsupervised pre-training to learn high-dimensional speech representations, this application can use HuBERT to extend the semi-supervised learning method using pseudo labels to self-supervised learning. Specifically, the basic models mentioned above can use wav2vec2-base-960h and hubert-base-ls960. For example, wav2vec2-large-960h-lv60-self and hubert-large-ls960-ft can be used as automatic feature extractors for large models, i.e., audio detail extractors.

[0132] Since there is a highly nonlinear relationship between audio signals and hand movements, the hand movement generation method for musical instrument performance provided in this application can use artificial intelligence to simulate and generate 3D images of hand movements in music performance. Specifically, it can be applied to various musical instrument songs such as guitar, violin, flute, and wind instruments. The above embodiment is mainly explained by taking the piano as an example, that is, given a piano piece, it is possible to locate the positions of both hands and use the basic model to generate a long list of corresponding gestures for the performance.

[0133] Corresponding to the method for generating hand movements for playing a musical instrument in the above embodiment, the present application also provides a device for generating hand movements for playing a musical instrument, and the device for generating hand movements for playing a musical instrument includes an acquisition unit, an extraction unit, a prediction unit and a gesture generation unit.

[0134] An acquisition unit, used for acquiring a target audio segment;

[0135] An extraction unit, used for inputting a target audio segment into a basic model, extracting the target audio segment using an audio detail extractor, and obtaining audio features;

[0136] A prediction unit, configured to predict the hand position of the audio feature using a hand position predictor to obtain a hand prediction position sequence;

[0137] The gesture generation unit is used to input the target audio clip and the hand prediction position sequence into the gesture generator for multi-step inverse diffusion processing to obtain the hand movement graph output by the basic model.

[0138] It can be understood that the above-mentioned units are also used to implement the technical solution of any embodiment in the present application.

[0139] See also Figure 4 , Figure 4 The structure of an embodiment of a computer-readable storage medium provided by the present application is shown in FIG. 7 . The computer-readable storage medium 70 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the following method steps:

[0140] Get the target audio segment;

[0141] Input the target audio clip into the basic model, and use the audio detail extractor to extract the target audio clip to obtain audio features;

[0142] The hand position predictor is used to predict the hand position based on the audio features to obtain a hand prediction position sequence;

[0143] The target audio clip and the predicted hand position sequence are input into the gesture generator for multi-step inverse diffusion processing to obtain the hand motion graph output by the basic model.

[0144] It can be understood that when the computer program 71 is executed by the processor, it is also used to implement the technical solution of any embodiment of the present application.

[0145] In the several layers of implementation provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as two layers of units or components can be combined or integrated into another layer of the system, or some features can be ignored or not executed.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one layer or distributed on two layers of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0147] In addition, each functional unit in each layer implementation mode of the present application can be integrated into a layer of processing units, or each layer unit can exist physically separately, or two or more layers of units can be integrated into a layer of units. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated units in the above other implementations are implemented in the form of software functional units and sold or used as independent products, they can be stored in a layer of computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a layer of storage medium, including several instructions to enable a computer device (which can be a layer of personal computers, servers, or network devices, etc.) or a processor (processor) to execute all or part of the steps of the implementation methods of each layer of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0149] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating hand movements for playing a musical instrument, characterized in that: The basic model includes an audio detail extractor, a hand position predictor, and a gesture generator; the method includes: Get the target audio segment; Inputting the target audio segment into the basic model, and extracting the target audio segment using the audio detail extractor to obtain audio features; Using the hand position predictor to predict the hand position of the audio feature to obtain a hand prediction position sequence; The target audio segment and the hand predicted position sequence are input into the gesture generator for multi-step inverse diffusion processing to obtain a hand motion graph output by the basic model.

2. The method for generating hand movements for playing a musical instrument according to claim 1, characterized in that: After obtaining the audio features, the method further includes: Encoding the audio features using the audio detail extractor to obtain encoded audio features; The step of using the hand position predictor to predict the hand position of the audio feature to obtain a hand prediction position sequence includes: Using the hand position predictor to predict the hand position of the encoded audio features to obtain a hand prediction position sequence; The step of inputting the target audio segment, the predicted hand position sequence and the preset noise sample into the gesture generator to obtain the hand motion graph output by the basic model comprises: The encoded audio features and the hand predicted position sequence are input into the gesture generator to obtain the hand motion graph output by the basic model.

3. The method for generating hand movements for playing a musical instrument according to claim 2, characterized in that: The hand position predictor includes a hand position decoder and a linear mapping layer, and the hand position predictor is used to predict the hand position of the encoded audio features to obtain a hand prediction position sequence, including: Inputting the encoded audio features into the hand position decoder, and using the hand position decoder to decode the encoded audio features to obtain latent features; The latent features are mapped to output positions using the linear mapping layer to obtain a sequence of predicted hand positions.

4. The method for generating hand movements for playing a musical instrument according to claim 2, characterized in that: The gesture generator includes a time embedding layer and a gesture decoder, and the encoded audio features and the hand predicted position sequence are input into the gesture generator to obtain the hand motion graph output by the basic model, including: Inputting the encoded audio features into the gesture decoder to obtain gesture features output by the gesture decoder; Inputting a preset diffusion time step number into the time embedding layer to obtain time step embedding information output by the time embedding layer; Inputting the time step embedding information, the gesture features and the hand predicted position sequence into a trained denoising model to obtain denoised gesture features output by the gesture generator; Based on the hand predicted position sequence and the denoised gesture features, a hand motion graph output by the basic model is obtained.

5. The method for generating hand movements for playing a musical instrument according to claim 4, characterized in that: The trained denoising model is trained in the following way: Get the original samples to be trained; A Markov chain is constructed by using a forward diffusion process, and noise is gradually added to the original sample to be trained within a preset time step to obtain a preset noise sample; The preset noise samples are input into a denoising model for training to obtain a trained denoising model.

6. The method for generating hand movements for playing a musical instrument according to claim 4, characterized in that: The network training method of the basic model includes: inputting the encoded audio features into the hand position predictor; Decoding the encoded audio features using the gesture decoder to obtain latent features; Mapping the latent features to output positions using the linear mapping layer to obtain a hand prediction position sequence; Determining a position loss value and a speed loss value based on the hand predicted position sequence and the actual hand position; The position predictor is trained using the position loss value and the speed loss value.

7. The method for generating hand movements for playing a musical instrument according to claim 6, characterized in that: The determining of the position loss value and the speed loss value based on the hand predicted position sequence and the hand actual position includes: The Manhattan loss function is used to calculate the Manhattan distance between the hand prediction position sequence and the hand actual position to obtain a position loss value; The Euclidean loss function is used to calculate the Euclidean distance between the hand prediction position sequence and the actual hand position to obtain a speed loss value.

8. The method for generating hand movements for playing a musical instrument according to claim 6, characterized in that: The network training method of the basic model also includes: The trained hand position predictor is used to guide the gesture generator to perform speed prediction training.

9. A device for generating hand movements for playing a musical instrument, characterized in that: The device for generating musical instrument playing hand movements includes a unit for executing the method as claimed in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program data, which, when executed by a processor, is used to implement the method for generating hand movements for musical instrument performance as described in any one of claims 1-8.

Images