Music playing system and method based on gesture recognition

Abstract

The present invention discloses a music performance system and method based on gesture recognition, the system comprising: a camera module for acquiring a performance video, a recognition module for recognizing gestures, an analysis and calculation module for converting the recognition result into a music performance attribute, a playback module for converting the music performance attribute into a music instruction and sound in real time, and a storage module for storing the performance video and the music instruction. The present invention can achieve the effect of accurately and smoothly playing music in the air without the performer wearing any equipment or performing key interaction, and can improve the accuracy and real-time performance of the performance in the air, thereby realizing easy-to-operate, high-precision music performance.

Description

Technical Field

[0001] The present invention relates to the field of video gesture recognition and music performance, and in particular to a music performance system and method for gesture recognition based on skeleton key point sequences. Background Art

[0002] Music performance is usually performed in the way of playing physical instruments, while using gestures to perform music means only through hand movements and gesture changes to achieve the purpose of playing music as accurately and smoothly as real instruments. Compared with the former, using gestures to perform music has great potential. On the one hand, it can save the cost of purchasing musical instruments, and on the other hand, it can replace the complex playing techniques and movements of real instruments with customized simple gestures, making music performance simpler and even richer.

[0003] At present, the related technologies of the music performance system based on gesture recognition include: invention patents with application publication numbers CN105389013 and CN107248407. The former requires the performer to wear special gloves or rings, and the internal accelerometer and gyroscope in the device obtain hand movement information as the basis for distinguishing different gestures, and finally achieve the purpose of controlling the audio signal. This technology can assist in controlling the sound effects of an existing music audio playback through gestures, but does not achieve the purpose of playing music directly through gestures. The latter preprocesses the collected image into a grayscale image, performs gesture segmentation and extracts contour features, and then uses the distance matching algorithm to obtain the gesture category, and finally selects the audio in the file library for playback according to the gesture category. This technology first selects the playback parameters such as pitch, timbre, and eight beats through the screen buttons, and then uses the gesture recognition results as the trigger event to call the corresponding pre-stored audio in the file library and play it through the speaker, so as to achieve the purpose of using gestures to play music. The disadvantage of this technology is that the hand contour features cannot accurately describe the position or movement relationship of each finger, which makes it difficult for the classifier to support more categories. In addition, this technology only uses gesture recognition results as trigger events for different note names, which means that it is impossible to control note attributes such as duration and loudness through gestures, nor is it possible to simultaneously input notes and modify timbre through gestures. In summary, this technology does not take into account the complexity of music itself and the coherence of the performance itself in music performance, and cannot achieve accurate and smooth performance like real instruments. Summary of the invention

[0004] In order to address the deficiencies in the above-mentioned prior art, the present invention proposes a music playing system and method based on gesture recognition, so as to recognize the gestures of the performer in the video. The performer does not need to wear any equipment or perform key interactions, so as to achieve the effect of playing music accurately and smoothly like a real instrument, and can improve the accuracy and real-time performance of remote performance, thereby realizing easy-to-operate and high-precision music performance.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The music playing system based on gesture action recognition of the present invention is characterized in that it comprises: a camera module, a recognition module, an analysis and calculation module, a playing module and a storage module;

[0007] The camera module acquires the performance video and sends it to the recognition module, and saves the performance video to the storage module as a video file;

[0008] The recognition module processes the performance video, obtains a recognition result including a gesture action category, and sends the result to the analysis and calculation module;

[0009] The analysis and calculation module processes the recognition result to obtain the music performance attribute and sends it to the playing module;

[0010] The playing module converts the music performance attributes into music instructions and sounds in real time, and stores the music instructions in the storage module as music information files.

[0011] The music performance system based on gesture action recognition described in the present invention is also characterized in that the recognition module includes a static gesture recognizer and a dynamic gesture recognizer; the static gesture recognizer processes each frame image in the performance video obtained by the camera module separately to obtain a static gesture recognition result; the dynamic gesture recognizer processes a group of continuous image frames in the performance video obtained by the camera module to obtain a dynamic gesture recognition result.

[0012] The analysis and calculation module sets the music performance attributes according to the recognition results of the recognition module. The music performance attributes consist of note attributes and system attributes, and are all controlled by gestures. The note attributes include: note name, loudness, octave group, sharp and flat signs, timbre, vibrato intensity, delay, frequency balance, and hold status. The system attributes include: performance recognition status, recording status, track number, and track status.

[0013] The playing module converts the music performance attributes into music instructions in real time, and uses the timbre library to convert the music instructions into actual musical sounds for playing through the loudspeaker.

[0014] The storage module saves the performance video acquired by the camera module and the music instructions generated in real time by the playback module respectively according to the parameter value of "recording status" in the system properties, so as to realize the real record of the improvisation process.

[0015] The music playing method based on gesture recognition of the present invention is characterized in that:

[0016] Step 1: Collect gesture videos of controlling music and set corresponding gesture category labels to build a gesture recognition dataset:

[0017] The gesture categories include static gestures and dynamic gestures;

[0018] The static gestures include four types: fist, palm, number, and thumb. Among them, any i-th static gesture contains m i Seed category, i∈[1,4]; and the category label of the static gesture is annotated on each frame;

[0019] The types of dynamic gestures include 6 types: open palm, shake palm, make letters, stroke palm, press finger down, and lift finger. Among them, any j-th dynamic gesture contains n j Seed category, j∈[1,6]; and the category labels of dynamic gestures are only marked on the start frame and the end frame;

[0020] Step 2: Build a gesture recognition network, including a static gesture recognition network and a dynamic gesture recognition network:

[0021] The static gesture recognition network includes: s A fully connected layer using the relu activation function and a softmax activation function;

[0022] The dynamic gesture recognition network includes: a sequence encoding module Encoder, an attention module Attn and a classification module Class;

[0023] The sequence encoding module Encoder comprises: g A stacked unidirectional GRU module;

[0024] The attention module Attn includes: a correlation calculation module and a fusion module;

[0025] The classification module Class includes: m A fully connected layer using the relu activation function and a softmax activation function;

[0026] Step 3: Select a video clip from the gesture recognition dataset, where the video clip contains a set of L consecutive frames of images, calculate the loss function of the video clip and train the network:

[0027] Step 3.1: Training of static gesture recognition network:

[0028] The openpose algorithm is used to extract H hand key point vectors x corresponding to any frame image in the video clip, and input them into the static gesture recognition network, and then pass through N s The fully connected layer is mapped to a length of l s The latent vector is activated by the softmax function to obtain the probability value of each gesture category of the frame image;

[0029] According to the probability value of each gesture category and its true label, the cross entropy loss function is calculated to train the static gesture recognition network until the cross entropy loss converges, and a trained static gesture recognition network is obtained;

[0030] Step 3.2: Training of dynamic gesture recognition network:

[0031] The openpose algorithm is used to extract H hand key point vectors corresponding to each frame image in the video clip, thereby obtaining a hand key point sequence X=[x1; ...; x t ;…;x L ]; where x t Represents the H hand key point vectors at the tth time step;

[0032] The hand key point sequence X is input into the dynamic gesture recognition network and passes through N g After a unidirectional GRU module, the output is the encoded information of L time steps h = [h1; ...; h t ;…;h L ]; where h t represents the encoding vector of the tth time step;

[0033] The attention module calculates the attention score based on the encoded information h Where W is the learnable weight matrix during training; T represents transpose;

[0034] The fusion module calculates the fusion information c=αh according to the attention score α T ;

[0035] The fusion information c is input into the classification module Class and passes through N m The mapping of the fully connected layer is l mThe latent vector is activated by the softmax function to obtain the probability value of the video clip in each gesture category;

[0036] According to the probability value of each gesture category of the video clip and its true label, a cross entropy loss function is calculated to train the dynamic gesture recognition network until the cross entropy loss converges, thereby obtaining a trained dynamic gesture recognition network;

[0037] Step 4: Use the trained gesture recognition network to recognize the performance movements in real time:

[0038] Use the openpose algorithm to identify the t-th frame image F in the music performance video t Extract K human key point vectors B t , H left-hand key point vectors LH t and H right-hand keypoint vectors RH t ; At the same time, extract the left hand key point vector LH from the t-1th frame, t-2th frame, ..., t-L+1th frame image in the music performance video to be recognized t-1 ~LH t-L+1 and the right hand keypoint vector RH t-1 ~RH t-L+1 ;

[0039] The t-th frame image F t The H left-hand key point vectors LH t Input into the static gesture recognition network to obtain the left hand static gesture recognition result LS t ;

[0040] The left hand key point vector LH t-L+1 ~LH t and the right hand keypoint vector RH t-L+1 ~RH t The left hand dynamic gesture recognition network is respectively sent to the left hand dynamic gesture recognition result LM. t And the recognition result of right hand dynamic gesture RM t ;

[0041] Step 5: Set the note attributes and system attributes according to the gesture recognition result, and convert them into MIDI instructions and output audio accordingly:

[0042] The note attributes include: note name, loudness, octave group, sharps and flats, timbre, vibrato intensity, delay, frequency balance, and hold status; wherein the note attributes of delay and frequency balance are set according to the left hand static gesture recognition result; the note attributes of timbre and vibrato intensity are set according to the left hand dynamic gesture recognition result; the note attributes of note name, loudness, octave group, sharps and flats, and hold status are set according to the right hand dynamic gesture recognition result;

[0043] The system attributes include: performance recognition state, recording state, track number, and track state; wherein the system attributes of "enable", track number, and track state of the performance recognition state are set according to the left-hand static gesture recognition result; the system attributes of "disable" and recording state of the performance recognition state are set according to the left-hand dynamic gesture recognition result;

[0044] Step 5.1, set the note properties:

[0045] If the left hand dynamic gesture recognition result LM t If it is a subcategory of "gesture letters" in dynamic gestures, set the "timbre" in the note attributes to the actual timbre corresponding to the subcategory;

[0046] If the left hand static gesture recognition result LS t If it is the "palm" in the static gesture, then set the "delay" in the note attribute to the value according to the human key point vector B t , the left hand key point vector LH t Calculate the actual delay R, set the "frequency balance" in the note attribute to the human key point vector B t , the left hand key point vector LH t The calculated actual equilibrium value D;

[0047] If the left hand dynamic gesture recognition result LM t For the dynamic gesture of "shaking palm", the "tremolo intensity" in the note attribute is set to the value according to the human key point vector B t , the left hand key point vector LH t The calculated actual force value F;

[0048] If the right hand dynamic gesture recognition result RM t is a subcategory of "finger down" or "finger up" in the dynamic gesture, then the "note name" in the note attribute is set to the actual note name corresponding to the subcategory, and the "hold state" in the note attribute is set to the actual hold state corresponding to the subcategory; the "octave group" in the note attribute is set to the actual hold state corresponding to the subcategory according to the human key point vector B t , the right hand key point vector RH tThe actual group O calculated, set the "flat and sharp group" in the note attribute to the human body key point vector B t , the right hand key point vector RH t The actual group SF calculated, the "loudness" in the note attribute is set according to the human key point vector B t , the right hand key point vector RH t The calculated actual loudness value A;

[0049] Step 5.2, set system properties:

[0050] If the left hand static gesture recognition result LS t If it is a "fist" in a static gesture, check whether the cumulative number of frames exceeds t seconds. If so, set the "performance recognition status" in the system properties to "enable", indicating that the note attribute is allowed to be converted into MIDI instructions;

[0051] If the left hand dynamic gesture recognition result LM t If the dynamic gesture is "Open Palm", check whether the "Performance Recognition Status" in the system properties is "Enabled". If so, set the "Recording Status" in the system properties to "Start", which means starting to record MIDI instructions and start recording the performance video;

[0052] If the left hand static gesture recognition result LS t is a subcategory of "number" in the static gesture, then check whether the "performance recognition status" in the system properties is "enabled", and if so, set the "track number" in the system properties to the actual track number corresponding to the subcategory; all MIDI instructions recorded in the track corresponding to the actual track number are used as segment loop instructions;

[0053] If the left hand static gesture recognition result LS t is a subcategory of the "thumbs-up" in the static gesture, then check whether the "performance recognition status" in the system properties is "enabled", and if so, set the "track status" in the system properties to the track status of "loop" or "clear" corresponding to the subcategory;

[0054] If the left hand dynamic gesture recognition result LM t For "Swiping Hands" in dynamic gestures, set "Performance Recognition Status" in System Properties to "Disable" and set "Recording Status" in System Properties to "Stop";

[0055] Step 5.3: Convert the note attributes and the system attributes into corresponding MIDI instructions according to the MIDI 2.0 protocol for audio output.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention can realize operations such as note input, timbre modification, segment looping, and audio storage through gestures, so as to achieve the purpose of playing music in real time. Compared with previous technologies, the performer does not need to wear any equipment or perform key interactions. It ensures that the process of playing music in the air is accurate and smooth, and achieves a performance effect close to that of real musical instruments, while making music performance more convenient.

[0058] 2. The present invention uses a monocular camera to capture video and extracts a sequence of key points of the hand as a classification basis for gesture recognition. Compared with previous technologies, it improves the recognition performance of finger movements in gestures, making the final music performance more accurate.

[0059] 3. The present invention divides hand gestures into static gestures and dynamic gestures, and uses deep learning networks to identify them respectively, and gives a conversion relationship from the recognition results to the music performance sounds, so that the hand gestures of the performers in the video can be identified in real time and converted into music instructions and sounds through analysis and calculation. Compared with previous technologies, it considers controlling multiple music performance attributes at the same time, making the final music performance more precise and smooth. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a work flow chart of the music performance system of the present invention;

[0061] Figure 2a is a schematic diagram of extracting key points of a body in a music performance system of the present invention;

[0062] Figure 2b Schematic diagram of extracting key points of hands in the music playing system of the present invention;

[0063] Figure 3 It is a schematic diagram of the left-hand timbre control action of the music performance system of the present invention;

[0064] Figure 4 1 is a schematic diagram of the left-hand instrument switching action of the music performance system of the present invention;

[0065] Figure 5 It is a schematic diagram of the right hand note name selection gesture of the music performance system of the present invention;

[0066] Figure 6 It is a schematic diagram of right-hand pitch area division of the music performance system of the present invention;

[0067] Figure 7 Schematic diagram of left-hand track control gestures of the music performance system of the present invention;

[0068] Figure 8 is a flow chart of a gesture action recognition method of the present invention;

[0069] Fig. 9 is a schematic diagram of a static gesture recognition network of the present invention;

[0070] Fig.10 Schematic diagram of the dynamic gesture recognition network of the present invention. DETAILED DESCRIPTION

[0071] like Figure 1 As shown, a music playing system based on gesture action recognition includes: a camera module, a recognition module, an analysis and calculation module, a playback module and a storage module;

[0072] The camera module uses a monocular camera to obtain the performance video and sends it to the recognition module, and saves the performance video to the storage module as a video file;

[0073] The recognition module processes the performance video, obtains the recognition result including the gesture action category and sends it to the analysis and calculation module;

[0074] The analysis and calculation module processes the recognition results, obtains the music performance attributes and sends them to the playback module;

[0075] The playing module converts the music performance attributes into music instructions and sounds in real time, and saves the music instructions in the storage module as music information files.

[0076] In a specific implementation, the recognition module includes a static gesture recognizer and a dynamic gesture recognizer.

[0077] like Fig. 9 As shown in Figure 1, the recognition network of the static gesture recognizer consists of three fully connected layers using the relu activation function and a softmax activation function. The static gesture recognizer processes each frame of the performance video obtained by the camera module separately, as shown in Figure 1. Figure 2b As shown, the openpose algorithm is first used to extract 21 key points of the hand in the frame image and form a vector, which is then input into the recognition network to finally obtain the static gesture recognition result; there are 9 categories of static gesture recognition results, namely: left hand fist, left hand palm, left hand numbers 1 to 4, left hand thumb up, left hand thumb down, and others;

[0078] like Fig.10As shown, the recognition network of the dynamic gesture recognizer consists of 3 stacked unidirectional GRU modules, an attention module, 2 fully connected layers using the relu activation function, and a softmax activation function. Among them, the attention module uses the attention mechanism commonly used in the field of deep learning to calculate the fusion information. The dynamic gesture recognizer processes a set of continuous image frames in the performance video obtained by the camera module. Specifically, at a video sampling rate of 24Hz, the openpose algorithm is used to extract 21 key points of the hand in each frame of the current frame and a total of 48 consecutive frames from this frame forward, and a matrix of dimension 42×48 is formed according to the time step, which is then input into the recognition network to finally obtain the dynamic gesture recognition result;

[0079] The analysis and calculation module sets the music performance attributes based on the recognition results of the recognition module. Figure 2a , Figure 2b As shown, the openpose algorithm is first used to extract the key points of the human body and the key points of the left and right hands in each frame of the performance video to assist in setting the music performance attributes.

[0080] Music performance attributes consist of note attributes and system attributes, and both are controlled by gesture actions.

[0081] Note attributes include: note name, loudness, octave group, sharp and flat signs, timbre, vibrato intensity, delay, frequency balance, and hold status;

[0082] Among them, the note attributes of delay and frequency balance are set according to the left hand static gesture recognition results; Figure 3 As shown in the figure, the left hand flipping upwards represents the use of longer note delays, and the left hand flipping downwards represents the use of shorter note delays. The counterclockwise rolling of the left hand represents the enhancement of low-frequency components in frequency equalization, and the clockwise rolling of the left hand represents the enhancement of high-frequency components in frequency equalization; delay and frequency equalization are calculated based on the coordinate information of the key points of the hand and the key points of the human body;

[0083] The note attributes of timbre and vibrato intensity are set according to the left hand dynamic gesture recognition results; Figure 4 As shown in the figure, the left hand quickly draws the letter "V" in the air to represent the use of violin sound, and the left hand quickly draws the letter "C" in the air to represent the use of piano sound. Figure 3 As shown, the left hand's horizontal continuous swing from side to side represents the note vibrato, and the vibrato intensity is calculated based on the relative horizontal speed from the key point of the hand to the key point of the body;

[0084] The note name, loudness, octave group, sharp and flat signs, and note attributes of the hold state are set according to the right hand dynamic gesture recognition results; Figure 5As shown, regarding the selection of different note names, pressing down with the fingers of the right hand from the thumb to the little finger represents the single note input of the note names do, re, mi, fa, and so respectively; pressing down with the thumb and index finger combined represents the single note input of the note name la; pressing down with the thumb, index finger, and middle finger combined represents the single note input of the note name si. The loudness of the note is determined by the maximum relative displacement of the finger key point relative to the wrist key point during the pressing action, among which the finger pressing action of the single notes la and si is based on the key point of the index finger. The hold status is divided into "on" and "off", corresponding to the finger pressing action and lifting action respectively. The method of obtaining octave groups and sharp and flat signs is as shown in Figure 6 As shown, in the area on the right side of the performer's body, according to the relative horizontal distance from the key point of the hand to the key point of the human neck, it is divided into three octave groups from near to far: low, middle and high; according to the relative height difference between the key point of the hand and the key point of the human neck, it is divided from top to bottom into the temporary sharp sign area, the restored pitch area and the temporary flat sign area;

[0085] System properties include: performance recognition status, recording status, track number, and track status;

[0086] The performance recognition status is divided into "Enable" and "Disable", where "Disable" means no gesture recognition; the recording status is divided into "Start" and "Stop", where "Start" means starting to record MIDI instructions and start recording performance videos;

[0087] The method for obtaining the track number and track loop status is as follows Figure 7 As shown, the left hand holding numbers 1 to 4 represents the selection of the corresponding audio tracks 1 to 4; the audio track loop status is divided into "loop" and "clear", and the left hand raising the thumb upward represents looping the content of the current audio track; the left hand raising the thumb downward represents clearing the content of the current audio track.

[0088] The system properties of "Enabled" of the performance recognition status, track number, and track status are set according to the left-hand static gesture recognition result; when the left-hand static gesture recognition result is the static gesture "fist", check whether the cumulative number of frames of the gesture exceeds 30 frames within 2 seconds. If so, set the system property "Performance Recognition Status" to "Enabled".

[0089] The "disabled" performance recognition state and the system properties of the recording state are set according to the left-hand dynamic gesture recognition result; when the left-hand dynamic gesture recognition result is the dynamic gesture "open palm", check whether the system property "performance recognition state" is "enabled", if so, set the system property "recording state" to "start". When the left-hand dynamic gesture recognition result is the dynamic gesture "swiping palm", set the system property "performance recognition state" to "disabled" and the system property "recording state" to "stop";

[0090] The playback module converts the music performance attributes into music instructions in real time, and uses the timbre library to convert the music instructions into actual music played through the speaker. Specifically, the note attributes and system attributes are converted into corresponding MIDI instructions according to the MIDI 2.0 protocol; the musescore3 open source timbre library is used to convert MIDI instructions into audio output.

[0091] The storage module saves the performance video obtained by the camera module and the music instructions generated in real time by the playback module according to the parameter value of the "recording status" in the system properties, so as to realize the real record of the improvisation process. Specifically, when the system property "recording status" is "start", the performance video is saved as a .mp4 file at a sampling rate of 24Hz, and the MIDI instruction sequence is saved as a .mid file.

[0092] In this embodiment, Figure 8 As shown, a music playing method based on gesture action recognition comprises:

[0093] Step 1: Collect gesture action videos for controlling music and set corresponding gesture category labels to establish a gesture action recognition dataset; wherein the gesture categories include static gestures and dynamic gestures;

[0094] There are four types of static gestures: fist, palm, number, and thumb. Among them, any i-th static gesture contains m i Seed category, i∈[1,4]; and the category label of the static gesture is marked on each frame; in this embodiment, a total of 9 specific categories are shown as follows: left fist, left palm, left numbers 1 to 4, left thumb up, left thumb down, and others;

[0095] There are six types of dynamic gestures: open palm, shake palm, make letters, stroke palm, press finger down, and lift finger up. j Seed category, j∈[1,6]; and the category label of the dynamic gesture is only marked on the start frame and the end frame; in this embodiment, a total of 20 specific categories are shown as follows: left hand open palm, left hand horizontal shaking palm, left hand palm drawing the letter "V", left hand palm drawing the letter "C", left hand swiping palm outward, right hand 7 single-tone finger pressing and finger lifting, others;

[0096] The video collection is dominated by dynamic gestures and is completed by 8 volunteers under different lighting and background conditions. Each action video needs to be ranked according to the speed of the action, the distance to the camera, and the amplitude of the action, such as Figure 6As shown in Figure 1, the gestures are fully captured in 9 active areas on the left or right side of the body. The video is resampled at a sampling rate of 24 Hz, and the gesture categories are manually annotated in the sampled video.

[0097] Step 2: Build a gesture recognition network, including a static gesture recognition network and a dynamic gesture recognition network:

[0098] The static gesture recognition network includes: N s = 3 fully connected layers with relu activation function and one softmax activation function;

[0099] The dynamic gesture recognition network includes: sequence encoding module Encoder, attention module Attn and classification module Class;

[0100] The sequence encoding module Encoder includes: g = 3 stacked unidirectional GRU modules;

[0101] The attention module Attn includes: a correlation calculation module and a fusion module;

[0102] The classification module Class includes: N m = 2 fully connected layers with relu activation function and one softmax activation function;

[0103] Step 3: Select a video clip from the gesture recognition dataset. The video clip contains a set of continuous L = 48 frames of images. Calculate the loss function of the video clip and train the network:

[0104] Step 3.1: Training of static gesture recognition network:

[0105] like Figure 2b As shown in FIG. 1 , the openpose algorithm is used to extract H = 21 hand key point vectors x corresponding to any frame image in the video clip and input them into the static gesture recognition network, such as Fig. 9 As shown, after N s = 3 fully connected layers mapped to a length of l s The hidden vector of s =9 is the number of static gesture categories, and after activation by the softmax function, the probability value of the frame image in each gesture category is obtained;

[0106] According to the probability value of each gesture category and its true label, the cross entropy loss function is calculated to train the static gesture recognition network until the cross entropy loss converges, and a trained static gesture recognition network is obtained;

[0107] Step 3.2: Training of dynamic gesture recognition network:

[0108] like Figure 2b As shown in FIG. 1 , the openpose algorithm is used to extract H hand key point vectors corresponding to each frame image in the video clip, thereby obtaining a hand key point sequence X = [x1; ...; x t ;…;x 48 ]; where x t Represents the hand key point vector at the tth time step; the dimension of X is 42×48;

[0109] like Fig.10 As shown in the figure, the hand key point sequence X is input into the dynamic gesture recognition network and passes through N g = 3 unidirectional GRU modules and then output L = 48 time steps of coded information h = [h1; ...; h t ;…;h 48 ]; where h t represents the encoding vector of the tth time step; the dimension of h is Q×48, where Q is the number of output feature channels of the last layer of GRU;

[0110] The attention module calculates the attention score based on the encoded information h Where L = 48, α is of dimension 1×L, W is the learnable weight matrix during training, and T represents transpose;

[0111] The fusion module calculates the fusion information c = αh according to the attention score α T ;

[0112] The fusion information c is input into the classification module Class and passes through N m = 2 fully connected layers, resulting in a length of l m The hidden vector of m =20 is the number of dynamic gesture categories, and the probability value of the video clip in each gesture category is obtained after activation by the softmax function;

[0113] According to the probability value of each gesture category in the video clip and its true label, the cross entropy loss function is calculated to train the dynamic gesture recognition network until the cross entropy loss converges, and a trained dynamic gesture recognition network is obtained;

[0114] Step 4: Use the trained gesture recognition network to recognize the performance movements in real time:

[0115] The video to be recognized is resampled at a sampling rate of 24 Hz, such as Figure 2a , Figure 2b As shown, the openpose algorithm is used to identify the t-th frame image F in the music performance video t Extract K = 18 human key point vectors Bt , H = 21 left-hand key point vectors LH t and H = 21 right-hand keypoint vectors RH t ; At the same time, extract the left hand key point vector LH from the t-1 frame, t-2 frame, ..., t-47 frame images in the music performance video to be recognized t-1 ~LH t-47 and the right hand keypoint vector RH t-1 ~RH t-47 ;

[0116] The t-th frame image F t H = 21 left-hand key point vectors LH t Input into the static gesture recognition network to get the left hand static gesture recognition result LS t ;

[0117] The left hand key point vector LH t-47 ~LH t and the right hand keypoint vector RH t-47 ~RH t The left hand dynamic gesture recognition results LM are respectively sent to the dynamic gesture recognition network to obtain the corresponding left hand dynamic gesture recognition results LM t And the recognition result of right hand dynamic gesture RM t ;

[0118] Step 5: Set the note attributes and system attributes according to the gesture recognition results, and convert them into MIDI instructions to output audio:

[0119] Note attributes include: note name, loudness, octave group, sharp and flat signs, timbre, vibrato intensity, delay, frequency balance, and hold status;

[0120] Among them, the note attributes of delay and frequency balance are set according to the left hand static gesture recognition results; Figure 3 As shown, the left hand flips up to represent the use of longer note delays, and the left hand flips down to represent the use of shorter note delays. The counterclockwise roll of the left hand represents the enhancement of low-frequency components in the frequency equalization, and the clockwise roll of the left hand represents the enhancement of high-frequency components in the frequency equalization;

[0121] The note attributes of timbre and vibrato intensity are set according to the left hand dynamic gesture recognition results; Figure 4 As shown in the figure, the left hand quickly draws the letter "V" in the air to represent the use of violin sound, and the left hand quickly draws the letter "C" in the air to represent the use of piano sound. Figure 3 As shown, the left hand's horizontal continuous swing from side to side represents the note vibrato, and the vibrato intensity is calculated based on the relative horizontal speed from the key point of the hand to the key point of the body;

[0122] The note name, loudness, octave group, sharp and flat signs, and note attributes of the hold state are set according to the right hand dynamic gesture recognition results; Figure 5 As shown, regarding the selection of different note names, pressing down with the fingers of the right hand from the thumb to the little finger represents the single note input of the note names do, re, mi, fa, and so respectively; pressing down with the thumb and index finger combined represents the single note input of the note name la; pressing down with the thumb, index finger, and middle finger combined represents the single note input of the note name si. The loudness of the note is determined by the maximum relative displacement of the finger key point relative to the wrist key point during the pressing action, among which the finger pressing action of the single notes la and si is based on the key point of the index finger. The hold status is divided into "on" and "off", corresponding to the finger pressing action and lifting action respectively. The method of obtaining octave groups and sharp and flat signs is as shown in Figure 6 As shown, in the area on the right side of the performer's body, according to the relative horizontal distance from the key point of the hand to the key point of the human neck, it is divided into three octave groups from near to far: low, middle and high; according to the relative height difference between the key point of the hand and the key point of the human neck, it is divided from top to bottom into the temporary sharp sign area, the restored pitch area and the temporary flat sign area;

[0123] System properties include: performance recognition status, recording status, track number, and track status;

[0124] The performance recognition status is divided into "Enable" and "Disable", where "Disable" means no gesture recognition; the recording status is divided into "Start" and "Stop", where "Start" means starting to record MIDI instructions and start recording performance videos;

[0125] The method for obtaining the track number and track loop status is as follows Figure 7 As shown, the left hand holding numbers 1 to 4 represents the selection of the corresponding audio tracks 1 to 4; the audio track loop status is divided into "loop" and "clear", and the left hand raising the thumb upward represents looping the content of the current audio track; the left hand raising the thumb downward represents clearing the content of the current audio track.

[0126] The "Enabled" of the performance recognition status, the track number, and the system properties of the track status are set according to the left-hand static gesture recognition results;

[0127] The "disabled" state of performance recognition and the system properties of recording state are set according to the left-hand dynamic gesture recognition results;

[0128] Step 5.1, set the note properties:

[0129] If the left hand dynamic gesture recognition result LM t If it is a subcategory "V" or "C" of "gesture letter" in the dynamic gesture, the note attribute "timbre" is set to the actual timbre violin or piano timbre corresponding to the subcategory;

[0130] If the left hand static gesture recognition result LS t If the static gesture is "Palm", set the "Delay" in the note attribute to the key point vector B of the human body. t , left hand key point vector LH t Calculate the actual delay R, set the "Frequency Balance" in the note properties to the human key point vector B t , left hand key point vector LH t The calculated actual equilibrium value D;

[0131] If the left hand dynamic gesture recognition result LM t For the "shaking palm" in the dynamic gesture, set the "vibrato intensity" in the note attribute to the human body key point vector B t , left hand key point vector LH t The calculated actual force value F;

[0132] If the right hand dynamic gesture recognition result RM t For example, if it is a subcategory "do", "re", "mi", "fa", "so", "la", "si" of "finger down" or "finger up" in the dynamic gesture, then the "note name" in the note attribute is set to the actual note name "do", "re", "mi", "fa", "so", "la", "si" corresponding to the subcategory, and the "hold state" in the note attribute is set to the actual hold state corresponding to the subcategory, that is, "on" or "off"; the "octave group" in the note attribute is set to the actual hold state corresponding to the subcategory according to the human key point vector B. t , right hand key point vector RH t The actual group calculated is O. Set the "flat and sharp group" in the note properties to the human body key point vector B t , right hand key point vector RH t Calculate the actual group SF and set the "loudness" in the note attribute to the human key point vector B t , right hand key point vector RH t The calculated actual loudness value A;

[0133] Step 5.2, set system properties:

[0134] If the left hand static gesture recognition result LS t If it is a "fist" in a static gesture, check whether the accumulated frame number exceeds t = 2 seconds. If so, set the "Performance Recognition Status" in the system properties to "Enable", indicating that the conversion of note attributes into MIDI instructions is allowed;

[0135] If the left hand dynamic gesture recognition result LM tIf the dynamic gesture is "Open Palm", check whether the "Performance Recognition Status" in the system properties is "Enabled". If so, set the "Recording Status" in the system properties to "Start", which means starting to record MIDI instructions and start recording the performance video;

[0136] If the left hand static gesture recognition result LS t If it is a subcategory "1-4" of "number" in static gestures, check whether "performance recognition status" in system properties is "enabled". If so, set "track number" in system properties to the actual track number 1-4 corresponding to the subcategory; all MIDI instructions recorded in the track corresponding to the actual track number are used as segment loop instructions;

[0137] If the left hand static gesture recognition result LS t If it is a subcategory of the static gesture "Thumbs Up" or "Thumbs Down", check whether the "Performance Recognition Status" in the system properties is "Enabled". If so, set the "Track Status" in the system properties to the track status corresponding to the subcategory, i.e., "Loop" or "Clear".

[0138] If the left hand dynamic gesture recognition result LMt is "swiping palm" in the dynamic gesture, set the "performance recognition status" in the system properties to "disable" and set the "recording status" in the system properties to "stop";

[0139] Step 5.3: Convert the note attributes and system attributes into corresponding MIDI instructions according to the MIDI 2.0 protocol; use the musescore3 open source timbre library to convert the corresponding MIDI instructions into audio output.

Claims

1. A music playing method based on gesture recognition, characterized in that: Step 1: Collect gesture videos of controlling music and set corresponding gesture category labels to build a gesture recognition dataset: The gesture categories include static gestures and dynamic gestures; The static gestures include four types: fist, palm, number, and thumb. i Static gestures include Seed category, ; And the category label of static gestures is marked on each frame; The types of dynamic gestures include 6 types: opening the palm, shaking the palm, making letters, swiping the palm, pressing the finger down, and raising the finger; any of the j Dynamic gestures include Seed category, ; And the category labels of dynamic gestures are only marked on the start frame and the end frame; Step 2: Build a gesture recognition network, including a static gesture recognition network and a dynamic gesture recognition network: The static gesture recognition network includes: N s A fully connected layer using the relu activation function and a softmax activation function; The dynamic gesture recognition network includes: a sequence encoding module Encoder, an attention module Attn and a classification module Class; The sequence encoding module Encoder comprises: N g A stacked unidirectional GRU module; The attention module Attn includes: a correlation calculation module and a fusion module; The classification module Class includes: N m A fully connected layer using the relu activation function and a softmax activation function; Step 3: Select a video clip from the gesture recognition dataset, where the video clip contains a set of continuous L Frame image, calculate the loss function of the video clip and train the network: Step 3.1: Training of static gesture recognition network: Use the openpose algorithm to extract the image corresponding to any frame in the video clip. H Hand keypoint vectors x , and input into the static gesture recognition network, and then pass through N s The fully connected layer is mapped to a length of l s The latent vector is activated by the softmax function to obtain the probability value of each gesture category of the frame image; According to the probability value of each gesture category and its true label, the cross entropy loss function is calculated to train the static gesture recognition network until the cross entropy loss converges, and a trained static gesture recognition network is obtained; Step 3.2: Training of dynamic gesture recognition network: Use the openpose algorithm to extract the corresponding H The hand key point vectors are obtained in the video clip. L The hand keypoint sequence of time steps X =[ x 1; …; x t ;…; x L ] ;in, x t Indicates t time step H hand keypoint vectors; The hand key point sequence X Input into the dynamic gesture recognition network, and pass through N g The output of the unidirectional GRU module L The encoding information of time steps h =[ h 1;…; h t ;…; h L ];in, Indicates t The encoding vector of time steps; The attention module is based on the encoded information h Calculating attention scores ,in, W is the learnable weight matrix during training, T represents transpose; The fusion module is based on the attention score Computational fusion information c = αh T ; The fusion information c Input into the classification module Class, and pass through N m The mapping of the fully connected layer is l m The latent vector is activated by the softmax function to obtain the probability value of the video clip in each gesture category; According to the probability value of each gesture category of the video clip and its true label, a cross entropy loss function is calculated to train the dynamic gesture recognition network until the cross entropy loss converges, thereby obtaining a trained dynamic gesture recognition network; Step 4: Use the trained gesture recognition network to recognize the performance movements in real time: Use the openpose algorithm to identify the first t Frame Image F t Extract K Individual key point vector B t , H Left-hand keypoint vector LH t and H Right hand keypoint vector RH t ; At the same time, the first t -1 frame, t -2 frames, ..., t - L +1 frame image extracts the left hand key point vector LH t-1 ~ LH t-L+1 and the right hand key point vector RH t-1 ~ RH t-L+1 ; The first t Frame Image F t of H Left-hand keypoint vector LH t Input into the static gesture recognition network to obtain the recognition result of the left hand static gesture LS t ; The left-hand keypoint vector LH t-L+1 ~ LH t and the right hand key point vector RH t-L+1 ~ RH t The left hand dynamic gesture recognition results are respectively sent to the dynamic gesture recognition network to obtain the left hand dynamic gesture recognition results. LM t And the recognition results of right hand dynamic gestures RM t ; Step 5: Set the note attributes and system attributes according to the gesture recognition result, and convert them into MIDI instructions and output audio accordingly: The note attributes include: note name, loudness, octave group, sharps and flats, timbre, vibrato intensity, delay, frequency balance, and hold status; wherein the note attributes of delay and frequency balance are set according to the left hand static gesture recognition result; the note attributes of timbre and vibrato intensity are set according to the left hand dynamic gesture recognition result; the note attributes of note name, loudness, octave group, sharps and flats, and hold status are set according to the right hand dynamic gesture recognition result; The system properties include: performance recognition state, recording state, track number, and track state; wherein the system properties of "enable", track number, and track state of the performance recognition state are set according to the left-hand static gesture recognition result; the system properties of "disable" and recording state of the performance recognition state are set according to the left-hand dynamic gesture recognition result; Step 5.1, set the note properties: If the left hand dynamic gesture recognition result LM t If it is a subcategory of "gesture letters" in dynamic gestures, set the "timbre" in the note attributes to the actual timbre corresponding to the subcategory; If the left hand static gesture recognition result LS t If it is the "palm" in the static gesture, then set the "delay" in the note attribute to the value according to the human key point vector B t , the left hand key point vector LH t Calculate the actual delay R, set the "frequency balance" in the note attribute to the human key point vector B t , the left hand key point vector LH t Calculated actual equilibrium value D ; If the left hand dynamic gesture recognition result LM t For the "shaking palm" in the dynamic gesture, the "tremolo intensity" in the note attribute is set to the value according to the human key point vector B t , the left hand key point vector LH t Calculated actual force value F ; If the right hand dynamic gesture recognition result RM t If it is a subcategory of "Finger Down" or "Finger Up" in the dynamic gesture, the "Note Name" in the note attribute is set to the actual note name corresponding to the subcategory, and the "Hold State" in the note attribute is set to the actual hold state corresponding to the subcategory; the "Octave Group" in the note attribute is set to the actual hold state corresponding to the subcategory according to the human body key point vector. B t , the right hand key point vector RH t The actual group calculated O , set the "flat and sharp group" in the note attribute to the human body key point vector B t , the right hand key point vector RH t The actual group calculated SF , set the "loudness" in the note attribute to the human key point vector B t , the right hand key point vector RH t Calculated actual loudness value A ; Step 5.2, set system properties: If the left hand static gesture recognition result LS t If it is a "fist" in a static gesture, check whether the cumulative number of frames exceeds t seconds. If it exceeds, the "Performance Recognition Status" in the system properties is set to "Enabled", indicating that the note attributes are allowed to be converted into MIDI instructions; If the left hand dynamic gesture recognition result LM t If the dynamic gesture is "Open Palm", check whether the "Performance Recognition Status" in the system properties is "Enabled". If so, set the "Recording Status" in the system properties to "Start", which means starting to record MIDI instructions and start recording the performance video; If the left hand static gesture recognition result LS t is a subcategory of "number" in the static gesture, then check whether the "performance recognition status" in the system properties is "enabled", if so, set the "track number" in the system properties to the actual track number corresponding to the subcategory; all MIDI instructions recorded in the track corresponding to the actual track number are used as segment loop instructions; If the left hand static gesture recognition result LS t is a subcategory of the "Thumbs Up" in the static gesture, then check whether the "Performance Recognition Status" in the system properties is "Enabled". If so, set the "Track Status" in the system properties to the track status of "Loop" or "Clear" corresponding to the subcategory; If the left hand dynamic gesture recognition result LM t For "Swipe Hand" in dynamic gestures, set "Performance Recognition Status" in System Properties to "Disable" and set "Recording Status" in System Properties to "Stop"; Step 5.3: Convert the note attributes and the system attributes into corresponding MIDI instructions according to the MIDI 2.0 protocol for audio output.

2. A music performance system based on gesture recognition, characterized in that: The method is applied to the music playing method as claimed in claim 1, and comprises: a camera module, a recognition module, an analysis and calculation module, a playback module and a storage module; The camera module acquires the performance video and sends it to the recognition module, and saves the performance video to the storage module as a video file; The recognition module processes the performance video, obtains a recognition result including a gesture action category, and sends the result to the analysis and calculation module; The analysis and calculation module processes the recognition result to obtain the music performance attribute and sends it to the playing module; The playing module converts the music performance attributes into music instructions and sounds in real time, and stores the music instructions in the storage module as music information files.

3. A music performance system based on gesture recognition according to claim 2, characterized in that: The recognition module includes a static gesture recognizer and a dynamic gesture recognizer; the static gesture recognizer processes each frame of the performance video acquired by the camera module separately to obtain a static gesture recognition result; The dynamic gesture recognizer processes a group of continuous image frames in the performance video acquired by the camera module to obtain a dynamic gesture recognition result.

4. The music playing system based on gesture recognition according to claim 2, characterized in that: The analysis and calculation module sets the music performance attributes according to the recognition results of the recognition module. The music performance attributes are composed of note attributes and system attributes, and are all controlled by gestures. The note attributes include: note name, loudness, octave group, sharp and flat signs, timbre, vibrato intensity, delay, frequency balance, and hold status; the system attributes include: performance recognition status, recording status, track number, and track status.

5. The music playing system based on gesture recognition according to claim 2, characterized in that: The playing module converts the music performance attributes into music instructions in real time, and uses the timbre library to convert the music instructions into actual musical sounds for playing through the loudspeaker.

6. The music playing system based on gesture recognition according to claim 2, characterized in that: The storage module saves the performance video obtained by the camera module and the music instructions generated in real time by the playback module according to the parameter value of the "recording status" in the system properties, so as to realize the real record of the improvisation process.

Images