Deep learning-based method for identifying metalle playing and generating music score

Abstract

The invention discloses a method for recognizing and generating a music score of dutall playing based on deep learning. The method is a method for recognizing a dutall playing state based on a target detection network, a key point detection network and a voice recognition network. According to the method, the current grade state is calculated according to the hand space position relation between the grade point of the dutel and the pressing grade, so that the hands of the dutel and the pressing grade of the current post of the video are detected through the target detection network, and then key point detection is performed on the hands of the dutel and the pressing grade through the key point detection network. The current grade state is determined according to the spatial position relation between the Dutall grade key point and the hand key point; one playing state is composed of a grade state and a string sweeping state, and the string sweeping states are summarized into sounds of two states (upward string sweeping and downward string sweeping), so that the current string sweeping state can be determined through a voice recognition network; combining the grade state and the string scanning state to generate a music score of a dutel playing music in the input video; on the basis of the framework, a system for generating music scores and assisting teaching videos can be made, and a standard Dutall playing database can also be established through the system, so that a user can learn contents which the user wants to learn from the database; a system for indicating errors of a user by comparing a playing mode of a video uploaded by the user with a playing mode in a standard database can also be made, and a system for assisting in collating a Dutall music score can also be made.

Description

Technical Field

[0001] The present invention is based on deep learning technology and studies a method for dutar performance recognition and music score generation based on deep learning. It is a method for identifying dutar performance status based on target detection network YOLOV5, key point detection network HRNet and speech recognition network conformer. A performance status consists of a fret state and a strumming state. Therefore, the subprocess library is first used to extract the audio of the input video, and then the audio is input to the audio processing part to determine the current strumming state and strumming time through the speech recognition network and the timestamp algorithm. Then the video is input to the image processing part to calculate the current fret state by the spatial position relationship between the fret point of the dutar and the hand pressing the fret. Therefore, the dutar and the hand pressing the fret of the current video are detected by the target detection network, and then the key point detection of the dutar and the hand pressing the fret is performed respectively by the HRNet network, and then the current fret state is determined by the spatial position relationship between the dutar fret key point and the hand key point. . The music score of the dutar performance music in the input video is generated by combining the fret state and the strumming state. The system based on this framework can be used to generate music scores and assist in teaching videos. A standard dutar playing database can also be established through the system. Users can learn what they want to learn from this database. The system can point out the user's mistakes by comparing the playing methods in the videos uploaded by the user with the playing methods in the standard database. The system can also assist in organizing dutar music scores. Background Art

[0002] Dutar is a musical instrument originated from Persian, meaning "two strings". It is mainly popular among the Uyghur and Uzbek peoples in Xinjiang, my country. It enjoys the title of "the mother of Uyghur musical instruments" in Uyghur music. As early as the 6th to 8th centuries AD, the ancestors of the Uyghurs, the Huihu people, created the stick-shaped straight-necked musical instrument with ethnic characteristics, the predecessor of the tar. Around the 14th century AD, the traditional dutar began to take shape and gradually spread to the present day. Its slender shape can still be seen on the music murals in the Kizil, Kumtura and Senmusaimu caves in Kuche.

[0003] As an important traditional musical instrument, the learning method and development form of dutar have changed significantly over time, and modern technology and educational concepts have been gradually integrated. The main learning methods and development forms of dutar at present include traditional master-apprentice teaching, professional education in music schools, online teaching and digital learning resources. However, manual teaching is expensive, so people who want to learn dutar learn by themselves, but self-study is difficult, which leads to scholars giving up learning halfway. Although someone has learned a new song, he does not know whether he has learned it correctly or where he is wrong. In this case, it is difficult to judge by himself without professional guidance. If someone sees a video of playing dutar shared by others on the Internet (short video platform, etc.), if he wants to learn, he will take notes step by step or see clearly which fret is pressed and whether to strum up or down, which is very difficult for people without a certain professional foundation. In the arrangement of dutar scores, they are all manually arranged, and manual arrangement is time-consuming. Therefore, the present invention starts from these problems, and the present invention proposes a dutar performance recognition method based on a target detection network, a key point detection network and a speech recognition network. Summary of the invention

[0004] The purpose of the present invention is to provide a method for dutar performance recognition and music score generation based on deep learning, which is beneficial to the learning of dutar scholars and the arrangement of music scores. Figure 1 As shown: (1) Extracting the audio part from the input video: A video of playing dutar is input to this framework. This framework first separates the video and audio, and then inputs the video and audio into the image processing part and the audio processing part respectively.

[0005] (2) Image processing part: In the image processing part, each image of the input video is first detected by the target detection network YOLOv5 to detect the pressing hand, the strumming hand and the dutar. The pressing hand and the dutar are respectively input into the corresponding key point detection network HRNet to detect each finger of the pressing hand and each taste key point of the dutar. The taste state (taste and corresponding time) corresponding to each moment is obtained through the coordinates of each key point of the pressing hand, the coordinates of the dutar taste key point and the video time.

[0006] (3) Audio processing part: In the audio processing part, the extracted audio is input into the speech recognition network Conformer, and then the strumming state corresponding to each moment (strumming and corresponding time) is obtained through the timestamp algorithm.

[0007] (4) Music score algorithm: The final music score, corresponding music score and corresponding time are obtained by using the fret state obtained by the image processing part and the strum state obtained by the audio processing part. The music score corresponding to the input dutar video can be obtained by using this information.

[0008] In the above technical solution, the method for recognizing dutar performance and generating music scores based on deep learning provided by the present invention has the following beneficial effects: (1) This invention can realize the music score generation and auxiliary teaching video generation system through this framework. Users can input the dutar playing video they want to learn, and the system provides the user with the music score and auxiliary teaching video to improve the learning efficiency of the learner. In addition, this system can be used to assist in the compilation of dutar music scores.

[0009] (2) This invention can build a dutar music score database through this framework, and users can learn by searching for the content they are interested in from the database, which provides users with better learning conditions.

[0010] (3) This invention uses this framework to make a system that points out the wrong playing status in the input dutar playing video. That is, the user inputs the video of his own playing to this system, and the system compares the playing status in the video provided by the user with the standard database and points out the problem of the user.

[0011] (4) The invention. The framework can be extended to other instruments with similar playing styles, such as the tambourine, rewap, and dombra. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The overall framework of dutar playing recognition proposed by the present invention Figure 2 Target detection data and key point detection data annotation Figure 3 Music notation format Figure 4 Music notation format Figure 5 Grade key point diagram before adding regression equation Figure 6 Grade key point diagram after adding regression equation Figure 7 The upper and lower column spacing and the upper and lower range of grade before adding the upper and lower column spacing algorithm Figure 8 The upper and lower range of grade after adding the upper and lower column spacing algorithm Fig. 9 The front and back spacing and the upper and lower range of the grade before adding the front and back spacing algorithm Fig.10 Add the upper and lower range of the grade before the front and back spacing algorithm Fig.11 The x-axis coordinate range of each grade of Dutar Fig.12 Results of duttar, up_hand and down_hand detection after YOLOv5 network training Fig.13 Detection results of the key points of the hand after HRNet network training Fig.14 Key point detection for dutar grade Detection results after HRNet network training Fig.15 The detection results of each image after inputting the dutar performance video into the method of deep learning-based dutar performance recognition and score generation (# indicates sharp, up indicates high pitch, the number indicates the fret, the following x indicates downward strumming, and s indicates upward strumming) DETAILED DESCRIPTION

[0013] According to the above description, the following is a specific implementation process, but the scope of protection of the present invention is not limited to this Implementation process.

[0014] (1) Related work: Dutar players usually put the resonance box of the dutar on their thighs, with one hand on the neck, mainly responsible for pressing the strings on the corresponding frets to change the timbre, and the other hand on the panel, mainly responsible for strumming the strings to play. Because some people's playing method is opposite to that mentioned above, whether the left hand or the right hand is used to strum the strings is called the hand that sweeps the strings, and whether the left hand or the right hand is used to press the frets is called the hand that presses the frets. In the present invention, the current fret state is obtained by the spatial relationship between the fret key points on the dutar and the key point coordinates of the fingers of the hand. The key point detection network HRNet network used in the present invention can only detect the key points of the same type of object at a time, but the method needs to detect the key point detection of the hand pressing the frets and the dutar, which belong to two types of objects and need to detect different key points. Therefore, in order to solve this problem, it is necessary to use two HRNet networks to measure the key points of these two types of objects. Therefore, the present invention needs to detect and classify the dutar and the hand pressing the frets through the target detection network. Because the hand for strumming the strings and the hand for pressing the frets are both hands, in order to avoid key point detection for both hands, and because the palm of the hand for pressing the frets is facing up and the back of the hand for strumming the strings is facing up, they can be easily detected in target detection. Therefore, the hand of the present invention is divided into the hand for pressing the frets and the hand for strumming the strings, and the detected dutar and the hand for pressing the frets are input into the corresponding key point detection network. In the present invention, the network model of the target detection module used in the present invention is YoloV5.

[0015] The function of the key point detection module is to perform key point detection on the dutar and the hand pressing the grade detected by the target detection module. The position coordinates of each grade of the dutar and the position coordinates of the fingers of the hand are detected, and the state of the current grade is predicted through the spatial relationship between the position coordinates of each grade of the dutar and the position coordinates of the fingers of the hand. The key point detection model used in the present invention is the HRNet network.

[0016] The present invention determines the current fret state through target detection and key point detection modules. Because pressing a fret does not necessarily involve only one strum, but multiple strums. Therefore, detecting the current strum state is also a challenging problem. The following introduces two solutions for determining the strum state contemplated by the present invention, and explains why the sound recognition module is used.

[0017] Solution 1: Use target detection to detect the movement of the hand below and determine the current strumming status.

[0018] Solution 2: Detect the strumming state through sound recognition. The upper strings of the dutar are generally tuned to low pitch, and the lower strings are generally tuned to high pitch. The sounds produced by the dutar strumming upward and downward are different, and there is a certain difference. People with a certain dutar professional foundation can distinguish whether the string is strumming upward or downward by the sound of the playing. Therefore, the present invention has come up with a second solution to determine the current strumming state through sound recognition.

[0019] When the present invention adopts the first solution, problems arise in the case of continuous downward (upward) strumming. The problems are as follows: Case 1: The player will first strum the string downward (upward), then move the hand upward (downward) to the initial position without strumming the string, and then strum the string downward (upward) for the second time. In this case, if solution 1 is used, the open string in the second step will be detected as a strumming state, resulting in an error.

[0020] Case 2: When the player continuously strums downward (upward), he may use part of the fingers to strum downward (upward), and then use another part of the fingers to strum downward (upward) again to complete the continuous strumming. At this time, because the hand only moves downward (upward) once, if the first solution is used, these two steps will be detected as only one strumming, resulting in an error.

[0021] When using the second scheme, the present invention only needs to detect two states, namely, downward strumming and upward strumming. The current strumming state can be determined by identifying these three states through the voice recognition module, and the current playing state can be determined by combining the fret state. In the present invention, the voice recognition module uses a conformer-based voice recognition network.

[0022] (2) Data annotation: During data labeling, target detection and key point detection data sets are labeled simultaneously. The present invention uses the Labelme tool to label data. The target detection data is labeled to label the entire dutar, the upper hand, and the lower hand. Because only the key positions of the fingers of the upper hand and the positions of each grade of the dutar need to be known, the key point detection data of the present invention labels 5 fingers and 34 dutar grade key points, as shown in the attached figure. Figure 2 shown.

[0023] Because it is a bit troublesome to judge the current strumming state by hearing when marking the strumming state data, the error rate is high and the marking efficiency is low, so in order to facilitate data marking, first collect standard correct dutar music and corresponding music scores, and determine the strumming state corresponding to the current note through the music scores. The present invention marks the downward strumming state as "down" and the upward strumming state as "up", as shown in the attached figure Figure 3 , as shown in 4.

[0024] (3) Linear regression algorithm: The Dutar grade coordinates are linearly regressed. The algorithm works as follows: The first function is that the current video post has 34 dutar fret points and is divided into two columns with 17 fret points as one column, and the 17 fret points in each column are on a straight line. Therefore, the present invention finds the linear regression equation corresponding to the 17 fret points in each column through their coordinates, and moves the fret points that are not on this equation to the function, and ensures that the 17 fret points are on a straight line through the linear regression equation. Before adding the linear regression algorithm, as shown in the attached figure Figure 5 As shown in the attached figure, after adding the linear regression algorithm Figure 6 .

[0025] The second function is that this linear regression equation will be used in the taste calculation algorithm. The current taste state is determined by the positional relationship between the coordinates of the key points of the hand and this function. This will be explained in detail when introducing the taste calculation algorithm.

[0026] The present invention sets the following grade coordinates as ( )={( , ),( , ),…,( , )}, the above coordinates are ( )={( , ),( , ),…,( , )}, the linear regression algorithm steps are as follows (taking the following grade as an example): First step calculation average value

[0027]

[0028] The second calculation Average Error:

[0029]

[0030] Step 3 Calculation The squared error:

[0031] Step 4 Calculate the slope b:

[0032] The fifth step is to calculate the intercept:

[0033] The above formula is the linear regression equation of the current post column grade obtained by the present invention, and finally the column grade is adjusted by this equation. coordinate:

[0034] The last modified grade coordinates of this column are: )={( , ),( , ),…,( , )}.

[0035] (4) Grade range algorithm: First, calculate the upper and lower grade spacing and the front and back grade spacing. The upper and lower grade spacing is because when pressing a grade, the coordinates of the key points of the hand are not necessarily on the grade column linear regression equation, and there may be a slight error, which is greater or less than the grade regression equation function value of the point. Therefore, the present invention proposes a variable upper and lower grade spacing, and the y value of this grade point is allowed to be added and subtracted from the regression function value. The error can be reduced by increasing the upper and lower column spacing. The grade range before adding the upper and lower column spacing algorithm is shown in the attached figure. Figure 7 , the grade range of this grade after adding the above and below grade spacing algorithms is shown in the attached figure Figure 8 Because the upper and lower grades correspond to each other, the distance between the upper and lower grades refers to the distance between the corresponding grades. To indicate, as shown in the attached figure Figure 7 As shown:

[0036] Among them, the front and back grade spacing is because when pressing a certain grade, there is a possibility that the finger may be at the end of this grade range. At this time, the key point of the finger has exceeded the range of the grade. Because generally when pressing a certain grade, the end section of the grade range or the grade point is pressed, and when pressing a certain grade, the beginning section of the grade range is generally not pressed. Therefore, based on these two reasons, the present invention proposes a front and back grade spacing and moves each grade range forward. To solve this problem, the previous grade is considered as the range without adding the grade spacing. Fig. 9 As shown in the attached figure, the grade after adding the grade interval calculation is the range Fig.10 As shown, the present invention uses To indicate, as shown in the attached figure Fig. 9 As shown:

[0037] Because this is not the top grade range, so add a fixed As the uppermost grade range value and The value of will become .

[0038] (5) Grade status and grade time algorithm: The range of this grade and the previous grade of the linear regression equation is the grade range of this grade as shown in the attached figure. Figure 7 As shown in the figure, based on the y-axis of the grade range obtained above, plus and minus one-third of the grade above and below the grade, the new grade range of the current grade is obtained as shown in the attached figure. Figure 8 As shown in the figure, based on the above grade range, the new current grade range is obtained by subtracting one quarter of the distance between the previous and next grades of the grade from the two segments of the x-axis. Fig.10 shown.

[0039] The x-axis coordinates of the key points of the Dutar are obtained through the HRNet network, and the x-axis coordinate range between this grade and the previous grade is set as the x-axis coordinate range of this grade as shown in the attached figure. Fig.11 As shown, the x-axis coordinate of each key point of the hand finger is obtained through the HRNet network and it is determined that the x-axis coordinate range of the grade of each finger belongs to. Each x-axis coordinate of the key point of the hand finger obtained through the HRNet network is substituted into each column of the linear regression equation to find out the function value of each finger coordinate x-axis substituted into the linear regression equation. Based on the x-axis coordinate range of the grade to which the x-axis coordinate of each finger belongs, the function value and grade range of each finger x-axis coordinate substituted into the linear regression equation are used to calculate which grade each finger belongs to or is not in. Based on the grade to which each finger belongs, the high-grade grade is extracted as the current grade state and the current time and the time corresponding to the current grade are recorded.

[0040] (6) Strumming state and strumming time algorithm The timestamp algorithm of the Conformer speech recognition module, because the present invention needs to know the corresponding time of each strumming state, the present invention proposes a timestamp algorithm, the algorithm steps are as follows: Because the present invention uses Conformer streaming speech recognition, a portion of the strumming state is output every period of time during speech recognition and the cycle continues until the end. The present invention sets this period of time as Ti, and the number of characters output is n. Then the corresponding time interval between the strumming states output in this period is:

[0041] If the start time of this section is set to Ts, then the time of the first word is Ts+ti, the second is Ts+2ti, and the nth is Ts+nti.

[0042] (7) The algorithm for combining the fret position and the strumming state. The present invention proposes two solutions for this part: The first solution: In the final result, the correctness of the strumming state is the focus. Because each strumming state corresponds to a playing state, the formula is as follows:

[0043] in is the fret state corresponding to the i-th strumming state, For all taste time collection, is the time of the i-th strumming state. First, find the fret time that is less than the i-th strumming state time from all fret time sets, then find the index of the maximum value among these fret times, and then find the fret corresponding to the index from the fret state set.

[0044] The second solution: In the final result, the correctness of the fret state is the focus. Therefore, the second solution is proposed, which first determines a strumming state for each fret, and then combines the same playing state time period of no more than 0.25 seconds to get the final result. According to the experimental results, when the prediction rate of the strumming state is not high, the second method is more accurate, especially when the fret prediction accuracy is higher.

[0045] (8) Training of the target detection network YOLOv5: Each module of the framework in this method is trained separately, and all module training is performed on an NVIDIA RTX2080ti GPU in Ubuntu 18.04 and pytorch 1.9. Because there is no dataset for this direction, each module is trained on a self-built dataset. This target detection experiment collected and annotated 718 images because there is a lot of relevant dutar information on the Internet, but there is very little data of good quality that meets the requirements of this experiment. According to the specific implementation content (2) data annotation, extract the dutar, the annotation box information of the hand pressing the frets and the hand strumming from each annotation file, and convert them into the YOLO dataset format. This module uses the stochastic gradient descent optimizer (SGD) for 200 rounds of training, with a batch size of 4, an input image resolution of 640×640, and an initial learning rate of 0.01. The test results after training are shown in the attached figure. Fig.12 shown.

[0046] (9) HRNet network training for hand keypoint and dutar grade keypoint detection modules: The key point dataset is uniformly marked on the target detection dataset, as shown in the attached figure. Figure 2 As shown. The data set table is uniformly annotated. During training, the data annotation content required for each module in the data set is extracted and the format is changed. According to the specific implementation content (2) data annotation, the hand key point annotation information of the dutar and pressing grade that we need is extracted from each annotation file and uniformly converted into the COCO data set format. Then training is carried out. This module uses the stochastic gradient descent optimizer (SGD) for 200 rounds of training, with a batch size of 64, an input image resolution of 192×192, a heat map resolution of 48×48, and an initial learning rate of 0.01. The number of key points in the parameter settings of the hand key points and dutar grade key points is different. There are 5 hand key points and 34 dutar grade key points. The test results after training are shown in the attached figure. Fig.13 , as shown in 14.

[0047] (10) Training of the Conformer network in the sound recognition (strumming state recognition) module: For the sound recognition (strumming state recognition) experiment, 257 numbers were collected. Because it is difficult for audio data annotators who are not music professionals to annotate correctly, this paper collects data to avoid incorrect annotations. Some standard songs with music scores (without vocals) are found as data, so the amount of data found is very small. For a sound recognition model, these data are far from enough, so this paper performs data augmentation, randomly adding another audio after some audio. After data augmentation, the amount of data in this paper reaches 11292 newly generated data plus 257 original data, a total of 11549 audios. The sampling path of each audio in this paper is 16000HZ, and the length is between 0.5 seconds and 30 seconds. While expanding the audio data set, the same operation is also performed to amplify the annotation file (for example: if the third audio is added after the first audio, the content of the first audio annotation file is added to the content of the third audio annotation file). This module uses the stochastic gradient descent optimizer (SGD), trained for 200 rounds, with a batch size of 16 and an initial learning rate of 0.001.

[0048] 11) Overall operation of a method for dutar performance recognition and score generation based on deep learning Step 1: After finishing all the above work, start running based on the attached figure Figure 1A method for identifying dutar performance and producing music scores in the framework shown. Audio is extracted from the input video through the subprocess library, and the audio is input into the Conformer-based speech recognition network of the trained strumming state recognition module. The network first converts the input audio signal into a digital signal. This process is usually called "sampling", which converts the continuous time domain signal into a discrete time domain signal. In this process, the sampling frequency set by the inventive method is 16000HZ. Next, the digital signal needs to be preprocessed to better adapt to the model. This process is usually called "feature extraction", which converts the digital signal into a set of numerical features. The FBANK method used in the method of the present invention can better represent the acoustic characteristics of the audio signal, such as pitch, speech rate and noise level. Then, the feature sequence is sent to the Conformer model. Load the pre-trained Conformer model weights. These weights are obtained by training on a large number of audio data sets and contain the features and parameters learned by the model. In the first layer of the model, a convolution module is used to capture local context information. This module usually consists of multiple convolutional layers and pooling layers, which can gradually reduce the spatial dimension of the feature sequence to better capture local features, and batch normalization (BN) is also performed in this process. Next, the output of the convolutional module is input into the self-attention module. This module can capture global context information, such as word order and grammatical structure (Dutar strumming also has certain rules and playing methods, such as seven beats, continuous up and down strumming, one down and two up, etc.). It usually consists of multiple self-attention layers and residual connections, which can gradually increase the spatial dimension of the feature sequence to better capture global features. In this process, we can also use multi-head self-attention mechanisms and relative position encoding to improve the expressiveness and generalization ability of the model. Finally, the output of the self-attention module is sent to the feedforward module. This module can enhance the expressiveness of the model, such as by adding nonlinear transformations and residual connections to increase the complexity and flexibility of the model. In this process, we can also use batch normalization and Dropout to prevent overfitting and improve the generalization ability of the model. Finally, the output of the feedforward module is sent to the output layer to map the feature sequence to the corresponding text token or pinyin sequence. This process is usually called "decoding", which can generate the most likely text mark or pinyin sequence according to the probability distribution of the feature sequence. The decoding method used in this invention is CTC_Beam_Search. Because the Conformer-based strumming state recognition network we use is a streaming speech recognition network, a part of the strumming state S=[s1,s2,s3,…,s n ], and obtain the time T corresponding to each strumming state through the timestamp algorithm s =[t1,t2,t3,…,tn ].

[0049] Step 2: Extract the video input to the image processing part one by one and input it into the YOLOv5 target detection network. First, the input image needs to be preprocessed, including resizing the image to meet the input requirements of the model. The input size used by YOLOv5 is 640x640 pixels. Convert the image to the data format required by the model, from RGB to BGR format. Load the pre-trained YOLOv5 model weights. These weights are trained on a large number of image datasets and contain the features and parameters learned by the model. Input the pre-processed image into the YOLOv5 model, and the model will process the image layer by layer to extract image features. YOLOv5 uses anchor boxes to predict the location of the target. These anchor boxes are predefined boxes of different shapes and sizes that are used to cover possible target areas in the image. The model predicts the bounding box of each anchor box, including the center coordinates, width, and height of the bounding box. In addition to the bounding box, the model also predicts the confidence of each anchor box, that is, the probability that the anchor box contains the target. The model also predicts the class probability for each anchor box, that is, the probability that the object in the anchor box belongs to a specific class. The predicted bounding boxes, confidences, and class probabilities are post-processed, usually including threshold screening, retaining only predictions with confidences above a certain threshold. The Non-Maximum Suppression (NMS) algorithm is applied to remove overlapping prediction boxes and retain only the best predictions. Finally, the detected objects (dutar, hand pressing frets, hand strumming) and their categories and bounding boxes are output.

[0050] Step 3: Extract the dutar and hand pressing the grade detected by YOLOv5 in the second step, and input them into the dutar grade key point detection HRNet network and the hand key point detection HRNet network respectively. First, before the input image enters the network, some preprocessing operations need to be performed on it so that the network can better identify the key points. These preprocessing operations include random rotation, random scaling, random horizontal flipping, etc. These operations can make the network better adapt to various scenes and angles. At the same time, in order to avoid the input image being too large or too small, the image needs to be cropped or padded so that the aspect ratio of the image meets the requirements of the network. Then, the HRNet network uses residual blocks and jump connections to extract the feature information of the input image. In this process, the network will gradually reduce the receptive field range and gradually increase the resolution of the feature map to better capture the location information of the key points. Specifically, the HRNet network uses multiple residual blocks, each of which contains a convolutional layer, a batch normalization layer, a nonlinear activation function, and a jump connection. These residual blocks can help the network better capture the feature information in the input image. Next, in the process of feature extraction, the HRNet network also uses Exchange Blocks to improve the receptive field of the network. ExchangeBlocks is a special module that can fuse feature maps from different scales to expand the receptive field of the network. Specifically, Exchange Blocks contains two submodules, namely Level-wise UpsamplingModule and Multi-scale Feature Fusion Module. The former is used to upsample low-resolution feature maps into high-resolution feature maps, and the latter is used to fuse feature maps from different scales. In this way, the HRNet network can better capture the location information of key points. Finally, after feature extraction and Exchange Blocks, the HRNet network will upsample the feature map, then compare it with the GT heat map, calculate the mean square error (MSE) loss, and adjust the location information of each key point according to certain weights. Finally, the network will output the location information of the key points, which can be used for tasks such as Dutar grade key point estimation and pressing grade hand key point estimation.

[0051] Step 4: Perform linear regression on the two columns of Dutar grade key points output by the Dutar grade key point detection HRNet network, add the points that are not in the grade linear regression equation, and calculate the grade range of the grade through the linear regression equation and the key point coordinates of each grade point. And calculate the current grade state P=[p1,p2,p3,…,p m], and calculate the quality time T corresponding to the current quality by the number of images in the video in which the current image is p =[t1,t2,t3,…,t m ].

[0052] Step 5: According to the strumming state, strumming state time, fret state, and fret state time obtained in the first step, the music score of the song in the input dutar performance video is calculated through the performance state algorithm. The music score includes: fret state P e =[p1,p2,p3,…,p k ], strumming state S e =[s1,s2,s3,…,s k ], each state corresponds to the time interval T e =[t1,t2,t3,…,t k ], the final result is shown in the attached figure Fig.15 shown.

Claims

1. A method for recognizing dutar performance and generating music scores based on deep learning, characterized in that: Extract audio from the input dutar performance video; input the extracted audio into the speech recognition network to identify the strumming state and calculate the time corresponding to each strumming state through the timestamp algorithm, and record the strumming state and the time corresponding to the strumming state; input each post of the input dutar performance video into the target detection network to detect the dutar, the hand pressing the frets and the hand strumming; The dutar and the hand pressing the grade detected by the target detection network are input into the key point detection network separately to detect the grade of the dutar and the key points of the hand fingers; linear regression is performed on the two columns of grade key points of the dutar; the grade range of the current grade is calculated through the linear regression equation of each column, the distance between the previous and next grades and the distance between the upper and lower grades; the current grade state is determined according to the relationship between the grade range of each grade and the spatial position of the key points of the hand fingers and the grade time of the current grade is recorded; The music in the dutar playing video is output according to the fret state, fret time, strum state and strum time; this method can be extended to instruments with similar playing methods to the dutar.

2. A method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The method of extracting audio from the input dutar performance video includes: using a subprocess library to extract audio from the input video, and resetting the sampling rate of the extracted audio to 16000 Hz.

3. A method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The speech recognition network and timestamp algorithm include: the speech recognition network is based on the Conformer speech recognition network, which can perform streaming speech recognition; the Conformer streaming speech recognition outputs a part of the strumming state every time period during speech recognition and keeps looping until the end. Set this time period as Ti, the number of characters output as n, t i Ti divided by n, the start time of this section is set to Ts, then the time of the first strumming state of this section is Ts+t i , the second one is Ts+2t i , the nth one is Ts+nt i , and finally we get our strumming state and the corresponding strumming time.

4. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The target detection network needs to detect the dutar, the hand pressing the frets and the hand strumming the strings, including: inputting the dutar performance videos one by one into the YOLOv5 target detection network, and detecting the dutar, the hand pressing the frets and the hand strumming the strings; and extracting the detected dutar and the hand pressing the frets.

5. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The key point detection network performs key point detection on the dutar and the hand pressing the frets, including: a method for dutar performance recognition and music score generation based on deep learning according to claim 4, characterized in that the dutar extracted by the YOLOv5 target detection network is input into the key point detection HRNet network, and two columns of fret points of the dutar are detected, with 17 frets in each column and a total of 34 fret points; a method for dutar performance recognition and music score generation based on deep learning according to claim 4, characterized in that the hand pressing the frets extracted by the YOLOv5 target detection network is input into the key point detection HRNet network, and the key points of 5 fingers are detected.

6. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The method of performing linear regression on two columns of dutar frets separately comprises: a method for dutar performance recognition and score generation based on deep learning according to claim 5, characterized in that the key point coordinates of the two columns of dutar frets are extracted, and a linear regression equation of each column of frets is calculated; according to the calculated linear regression equation of the column, the frets in the column that are not on the linear regression equation are moved to the linear regression equation of the column to ensure that the frets in each column are on a straight line.

7. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The method of calculating the taste range of the current taste by the linear regression equation of each column, the distance between the front and back tastes and the distance between the upper and lower tastes includes: a method for recognizing dutar performance and generating music scores based on deep learning according to claim 6, characterized in that the range of the taste and the previous taste of the linear regression equation is the taste range of the taste; based on the y-axis of the above-obtained taste range, one-third of the upper and lower tastes of the taste is added and subtracted to obtain the new taste range of the current taste; based on the two segments of the above-obtained taste range, one-quarter of the distance between the front and back tastes of the taste is subtracted to obtain the new taste range of the current taste.

8. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The method of determining the current taste state and recording the taste time of the current taste according to the taste range of each taste and the spatial position relationship of the key points of the fingers of the hand comprises: a method of identifying dutar performance and generating music scores based on deep learning according to claim 5, characterized in that the x-coordinate of the key point of the dutar is obtained through the HRNet network and the x-coordinate range between the taste and the previous taste is set as the x-axis coordinate range of the taste; a method of identifying dutar performance and generating music scores based on deep learning according to claim 5, characterized in that the x-coordinate of each key point of the hand finger is obtained through the HRNet network and it is determined which taste the x-axis coordinate range of each finger belongs to; a method of identifying dutar performance and generating music scores based on deep learning according to claim 5, characterized in that , each x-coordinate of the key points of the fingers of the hand obtained by the HRNet network is brought into the method for recognizing dutar performance and generating music scores based on deep learning according to claim 6, characterized in that, in the linear regression equation, the function value of each finger coordinate x brought into the linear regression equation is found; based on the grade x-coordinate range to which each finger x-coordinate belongs and the function value of each finger coordinate x brought into the linear regression equation and according to claim 7, the grade range calculates which grade each finger belongs to or is not on; based on the grade to which each finger belongs, the high grade is extracted as the current grade state and the current time and the time corresponding to the current grade are recorded.

9. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The music score of the song in the input dutar performance video outputted according to the fret state, fret time, strum state and strum time comprises: a method for identifying dutar performance and generating music score based on deep learning according to claim 2, characterized in that the fret state, strum time and the fret time are used to calculate the overall music score of the dutar song in the input video; there are two schemes for music score calculation, the first is to focus on the accuracy of the strum state, and find out the fret state corresponding to the fret time of the strum time, and the second is to focus on the accuracy of the fret state and find out the strum state corresponding to the fret time of the strum time.

10. The method for recognizing dutar performance and generating music scores based on deep learning according to claim 1, characterized in that: The method can be extended to musical instruments with similar playing methods to the dutar, including: it can be extended to the tanbul, whose playing method is to strum or pull the strings with one hand and press the frets with the other hand, which is similar to the playing method of the dutar; it can be extended to the rewap, whose playing method is also to strum or pull the strings with one hand and press the frets with the other hand, which is similar to the playing method of the dutar; it can be extended to the dombra, whose playing method is also to strum or pull the strings with one hand and press the frets with the other hand, which is similar to the playing method of the dutar.

Images