A multi-modal piano transcription device based on graph convolution and a training method thereof
By using a graph convolution-based multimodal piano transcription device that combines audio and hand skeleton features for feature fusion, the problem of insufficient transcription accuracy in existing technologies is solved, and higher transcription accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-07
- Publication Date
- 2026-03-27
AI Technical Summary
Existing piano transcription methods are mostly unimodal, which cannot effectively utilize hand position and hand movement information, resulting in insufficient transcription accuracy. In particular, in multimodal transcription, there are problems of over-detection outside the hand area and under-detection within the hand area.
A multimodal piano transcription device based on graph convolution is adopted, including an audio transcription module, a hand skeleton feature transcription module, and a feature fusion transcription module. Pose learning is performed through the ST-GCN network, and feature fusion is performed by combining audio and hand skeleton features. Graph convolution is used to achieve feature aggregation in spatial and temporal dimensions, thereby improving transcription accuracy.
It effectively captures hand features, reduces the deficiencies of single-modality transcription, improves the accuracy of piano transcription results, reduces the problems of multiple detections outside the hand area and missed detections within the hand area, and improves the accuracy of transcription.
Smart Images
Figure CN115985270B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of piano transcription, and more particularly relates to a multi-modal piano transcription device based on graph convolution and a training method thereof. BACKGROUND
[0002] At the present stage, piano teaching needs experienced teachers to cultivate excellent students, and based on the scarcity of excellent teachers and the popularity of piano education, students often need to pay expensive economic costs and time costs to get effective guidance. In view of the above factors, whether teachers or students, piano teaching is an activity that requires a lot of time and effort. Today, with the rapid development of science and technology, computer technology is used to assist the development of music education at home and abroad. Automatic music transcription (AMT) can transcribe the performance of each student. In the case of one-to-many teaching by piano teachers, after comparing the transcription results and standard music scores, independent improvement suggestions are given, and immediate feedback is given to students to enhance the interactivity of teaching.
[0003] Automatic music transcription is the process of converting the original sound signal into a symbolic representation, which describes the key information of the basic unit in music - the note, generally including the onset, pitch and offset. The current piano transcription method mostly uses a single-modal transcription model, which cannot achieve satisfactory results in actual performance scenarios. Some multi-modal transcription methods use pictures that are differentiated or intercepted as the original input, without fully utilizing hand position and hand movement information, and the accuracy of the transcription results needs to be improved. SUMMARY
[0004] In view of the above defects or improvement needs of the prior art, the present application provides a multi-modal piano transcription device based on graph convolution and a training method thereof, thereby solving the technical problem that the accuracy of the existing piano transcription method needs to be improved.
[0005] To achieve the above-mentioned purpose, according to the first aspect of the present application, a multi-modal piano transcription device based on graph convolution is provided, comprising:
[0006] An audio transcription module is configured to convert the piano performance audio into frequency spectrum information, and transcribe after slicing according to a preset time length, to obtain a pitch frame-level detection result.
[0007] A hand skeleton feature transcription module is configured to learn a pose based on an ST-GCN network according to an input spatio-temporal graph, and output a hand skeleton feature transcription result. The spatio-temporal graph includes the spatial and temporal connection relationship of the hand key points of the continuous T frame images in the piano performance video.
[0008] The feature fusion transcription module is configured to perform time dimension and frame rate matching between the pitch frame level detection result and the hand skeleton feature transcription result, so as to obtain a transcription result of the piano playing video.
[0009] According to a second aspect of the present application, a training method of a graph convolution based multi-modal piano transcription device is provided, the multi-modal piano transcription device being the graph convolution based multi-modal piano transcription device according to the first aspect, and the training method comprising:
[0010] The multi-modal piano transcription model is trained with the objective of minimizing the difference between the start event, pitch frame level, end event and key speed detection results and the respective true labels, the difference between the hand skeleton feature transcription result and the posture learning label, and the difference between the piano playing video transcription result and the true label thereof.
[0011] According to a third aspect of the present application, a graph convolution based multi-modal piano transcription method is provided, comprising:
[0012] A trained multi-modal piano transcription device is obtained, the multi-modal piano transcription device being trained according to the training method of the graph convolution based multi-modal piano transcription device according to the second aspect;
[0013] A piano playing video to be transcribed is input into the training method of the graph convolution based multi-modal piano transcription device, so as to obtain a transcription result.
[0014] According to a fourth aspect of the present application, a computer storage medium is provided, and the computer storage medium stores instructions, when the instructions are executed on a computer, the computer is caused to execute the method according to the second aspect, or execute the method according to the second aspect.
[0015] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:
[0016] The application provides a multi-modal piano transcription device based on graph convolution and a training method thereof. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 A structure schematic diagram of the multi-modal piano transcription device based on graph convolution is provided for the embodiment of the application.
[0018] Figure 2 A structure schematic diagram of the audio transcription module is provided for the embodiment of the application.
[0019] Figure 3 A key matching result schematic diagram is provided for the embodiment of the application.
[0020] Figure 4 A structure schematic diagram of the ST-CGN network is provided for the embodiment of the application.
[0021] Figure 5 A hand key point connection schematic diagram is provided for the embodiment of the application.
[0022] Figure 6 A structure schematic diagram of the hand skeleton feature transcription module is provided for the embodiment of the application.
[0023] Figure 7 A multi-modal piano transcription method based on graph convolution provided by the embodiment of the application and a piano transcription result obtained by only using an audio transcription module are shown in the schematic diagram. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.
[0025] The embodiment of the application provides a multi-modal piano transcription device based on graph convolution, which comprises an audio transcription module, a skeleton transcription module and a fusion transcription module. Figure 1As shown, comprising:
[0026] An audio transcription module is configured to convert the piano performance audio into spectrum information, and transcribe the spectrum information after random slicing according to a preset time length to obtain a pitch frame-level detection result.
[0027] Further, the audio transcription module comprises a preprocessing module and a spectrum transcription module.
[0028] Specifically, the audio information input by the audio transcription module is converted into spectrum information through Mel transformation, and the spectrum is randomly and equally sliced and sent to a CNN and an RNN to learn frequency dimension features and time sequence features.
[0029] The audio preprocessing firstly converts the audio file into a digital signal after sampling, and then obtains spectrum information through time-frequency transformation. Preferably, the Mel transformation is adopted as the time-frequency transformation method. The spectrum information is sliced in a traversal manner according to a fixed time length and used as the input of the audio transcription model (i.e., the spectrum transcription module). The audio transcription model comprises four branches, namely, a multi-pitch onset detection branch, a multi-pitch frame-level detection branch, a multi-pitch end detection branch, and a key speed detection branch (i.e., a multi-pitch onset event detection module, a multi-pitch pitch frame-level detection module, a multi-pitch end event detection module, and a key speed detection module). The multi-pitch represents the state classification of 88 piano keys, which are independent of each other. The audio features are obtained through the correction of the model parameters by the joint action of the four branches. That is, as shown in Figure 2 As shown, the spectrum transcription module comprises a multi-pitch onset event detection module, a multi-pitch pitch frame-level detection module, a multi-pitch end event detection module, and a key speed detection module.
[0030] The multi-pitch onset event detection module and the multi-pitch end event detection module each comprise a CNN network and an RNN network, and are configured to obtain the detection results of the onset event and the end event according to the input;
[0031] The multi-pitch pitch frame-level detection module comprises a CNN network, a first RNN network, and a second RNN network,
[0032] The detection results of the onset event and the end event are spliced with the results obtained after the input is processed through the CNN network and the first RNN network, and used as the input of the second RNN network to obtain the pitch frame-level detection result.
[0033] The key speed detection module comprises a CNN network, and is configured to obtain the key speed detection result according to the input.
[0034] In the start and end prediction branches, the network is CNN+RNN, and after inputting the audio spectrum information, the corresponding outputs A and B are obtained respectively; in the frame-level prediction branch, the data obtains the intermediate feature C after passing through CNN+RNN, and the A, B and C are spliced and then passed through RNN to obtain the final output of the frame-level branch, so as to help the frame-level prediction branch to better fit through the start and end branches; in the key speed prediction branch, only CNN is used to obtain the corresponding output.
[0035] The hand skeleton feature transcription module is configured to perform pose learning based on an ST-GCN network according to an inputted space-time graph, and output a hand skeleton feature transcription result; wherein the space-time graph comprises a space and time connection relationship of hand key points in T consecutive frames of images in the piano performance video; wherein T is an integer greater than 1; preferably, T is an odd integer greater than 1.
[0036] Further, the hand skeleton feature transcription module comprises:
[0037] The key point detection module is configured to extract hand key points of each frame of image in the piano performance video.
[0038] The pose learning module based on the ST-GCN network is configured to perform pose learning according to an inputted space-time graph, and output a hand skeleton feature transcription result.
[0039] Specifically, the hand skeleton feature transcription module obtains hand key point detection results of each frame through inputted video information, and inputs the results into a graph convolution network to obtain corresponding finger key pressing feature outputs. In the field of piano transcription, graph convolution is introduced, and skeleton data is inputted into an ST-GCN network to realize feature aggregation in space and time dimensions, so as to effectively capture skeleton features. The skeleton feature transcription module comprises three parts, i.e., a key point detection module, a finger key matching module and a pose learning module.
[0040] (1) Key point detection module
[0041] The input of key point detection is a piano performance video, 21 key points of each hand are obtained through hand key point detection, and fingertip key points are further identified through numbering.
[0042] Preferably, the hand skeleton information is obtained by using mediapipe proposed by Google. The model trains a palm detector instead of a hand detector, so that relatively accurate key point coordinates can be obtained by using a non-maximum suppression algorithm in the case of palm self-occlusion, thereby solving the problem of double-hand cross self-occlusion in the piano performance process.
[0043] (2) Finger key matching module
[0044] After obtaining the hand key point coordinates, according to the piano playing characteristics, the fingertip key point is matched with the position of the key, and the label required by the skeleton feature transcription module, i.e. the posture learning label, is obtained.
[0045] Since the piano keys only have black and white colors, the position of the piano key can be classified by using the semantic segmentation method to classify the pixel color of the background image not containing hands in the piano playing video, and then using the connected domain algorithm to locate each key to obtain the area corresponding to each key in the 88 keys, and then determining the corresponding fingertip key point position by using the traversal method.
[0046] That is, the finger-key matching traverses and matches the position of each fingertip key point with the position of the 88 keys, and if the position of the fingertip key point is in a certain key, it is considered that the fingertip is successfully matched with the corresponding key, as shown in Figure 3 The number on the fingertip is the key number obtained by finger-key matching, and the upper row of numbers and the lower row of numbers are the key position numbers.
[0047] (3) Posture learning module based on ST-GCN network
[0048] The position information of the hand skeleton and the constraint information of the hand joint are used as inputs, and a space-time graph convolution network (ST-GCN) is used to identify the hand action. The weight matrix inside the ST-GCN suppresses the weight of unimportant edges. The weighted adjacency matrix and the hand skeleton sequence are input into the GCN network and the TCN network to realize the aggregation of spatial dimension features and temporal dimension features.
[0049] The input of the ST-GCN network is the skeleton data of continuous T frames, and the output is the probability feature information of whether the finger of the intermediate frame image is pressed, and the feature dimension is 10, which is the feature dimension of the ten fingers. Thus, the skeleton feature of each frame of the video mode is obtained.
[0050] The process of posture learning is as follows: after obtaining the finger key points by reading frame by frame, the key points are taken as graph nodes, and the natural connection of the hand structure and time is taken as graph edges to construct a space-time graph on the skeleton sequence as the input of the ST-GCN network. The model performs multi-level space-time graph operation on the input data, gradually generates higher-level feature maps, and finally performs multi-label classification to determine which finger has a pressing action.
[0051] The skeleton sequence is generally represented by two-dimensional or three-dimensional coordinates of human joints, and the present application uses a space-time graph G=(V, E) to represent the connection relationship between space and time in the hand skeleton sequence containing K key points and continuous T frames. The points V={V ti|t = 1, …, t, i = 1, …, K} represents all the joints on the skeleton sequence, which is 21 key points of each hand extracted by the key point detection module in this paper, and the feature vector F(v ti ) of each key point consists of its coordinates and prediction confidence. The edge E in the figure consists of both spatial and temporal structures, the spatial structure E S = {v ti v tj |(i, j) e H} represents the connection line between all joints in a frame, where H represents the set of natural connections of human hand joints, and the temporal structure E T = {v ti v (t+1)i} represents the connection line between the same joints in consecutive time frames.
[0052] The main body of the ST-GCN network includes multiple ST-GCN units, preferably 9, and the ST-GCN network structure is as shown in Figure 4 The adjacency matrix A represents a 21x21 matrix formed by the pairing of the 21 hand key points of each hand in a single frame (the same key points can also be paired), and the weight matrix is initially assigned according to the distance of each key point from the key point of the fingertip. Two key points are connected in physical position, which is 1, the same is 0, and not connected is inf (representing infinity), as shown in Figure 5 The corresponding connection of the 21 hand key points. The weight matrix (of the same size as the adjacency matrix) is multiplied by the adjacency matrix A by position, which gives important edges in the adjacency matrix A a larger weight and suppresses the weight of unimportant edges. The weighted adjacency matrix A and the hand skeleton sequence are input into the GCN network for learning to achieve the aggregation of spatial dimension features, and the TCN network achieves the aggregation of time dimension features. The ST-GCN also introduces a residual structure to add the output of the skeleton sequence and the TCN to enhance the learning ability of the network. Since the learning ability of the network will degrade to some extent as the number of model layers increases, the introduction of the residual network can effectively solve this problem.
[0053] A learnable mask M is added to each layer of the ST-GCN network, and the mask learns the importance weight according to each edge in the spatial structure E S to scale the contribution of node features to adjacent nodes.
[0054] The feature fusion transcription module is used to match the pitch frame-level detection result and the hand skeleton feature transcription result in time dimension and frame rate, and then obtain the transcription result of the piano performance video through the convolution layer and the fully connected layer.
[0055] It can be understood that the transcription result of the piano performance video is the pitch sequence in the piano piece played by the piano performance video and the corresponding time.
[0056] Specifically, the feature fusion transcription module is divided into two parts of feature matching and feature fusion.
[0057] Firstly, the skeleton features and the audio features are matched in the time dimension through the feature matching, so that the audio features can correspond to the video features in the same time dimension.
[0058] The embodiment of the application provides a training method of a multi-modal piano transcription device based on graph convolution, the multi-modal piano transcription device is the multi-modal piano transcription device based on graph convolution in any of the above embodiments, and the training method comprises the following steps:
[0059] The multi-modal piano transcription model is trained by taking minimizing the difference between the starting event, the pitch frame level, the ending event and the key speed detection result and the respective real label, the difference between the hand skeleton feature transcription result and the posture learning label, and the difference between the piano performance video transcription result and the real label thereof as the target.
[0060] Further, the real labels of the starting event, the pitch frame level, the ending event, the key speed detection result and the piano performance video transcription result are obtained based on the annotation of the piano performance video;
[0061] The posture learning label is obtained by matching the fingertip key point in each frame image of the piano performance video with the position of the key.
[0062] Specifically, the fingertip key point in each frame image of the piano performance video is matched with the position of the key by the key matching module in the skeleton feature transcription module, so as to obtain the posture learning label of each frame image.
[0063] Further, cross entropy is taken as the loss function of the training, and the loss function is L=L audio +L skeleton +L fusion , wherein:
[0064]
[0065]
[0066]
[0067]
[0068] L audio =L onset +L frame +L offset +L velocity (5)
[0069]
[0070]
[0071] l in formula (1) bce represents the cross-entropy loss. n p,t represents the true label corresponding to the pitch p and the time t start branch, while the predicted probability of the pitch p and the time t start branch is f in formula (2) p,t represents the true label corresponding to the pitch p and the time t frame level branch, while the predicted probability of the pitch p and the time t frame level branch is o in formula (3) p,t represents the true label corresponding to the pitch p and the time end branch, while the predicted probability of the pitch p and the time t end branch is (4) The loss of the touch key speed is calculated by the mean square error, and the true label of the start is weighted, so only the intensity of the touch key needs to be concerned. This is because the touch key speed only depends on the strength of the touch key moment. (5) is the total loss value of the audio transcription module.
[0072] The device and the training method for the multi-modal piano transcription based on the graph convolution provided by the application will be further described below with a specific example.
[0073] 1. First, the audio features are obtained, and in the audio preprocessing, the corresponding spectral information is obtained by reading the audio sequence, time-frequency transformation, and equal-length random cutting. Then, the audio features are obtained through the spectral feature transcription module.
[0074] (1) In the preprocessing module, the librosa.feature.mel function in the python librosa library is used. The librosa.feature.mel function is internally composed of a short-time Fourier transform (STFT) to generate an amplitude and a Mel filter bank required for audio transformation generated by librosa.filters.mel. The output of the Mel transformation is obtained by multiplying the two. Six parameters are required in the preprocessing process, and the specific parameters are a sampling rate sample_rate = 16000, a filter window length win_length = 2048, a frequency point number corresponding to each frame n_mels = 229, a Mel transformation minimum frequency value mel_fmin: = 27.5 Hz, a Mel transformation maximum frequency value mel_fmax = 8000, and a hop length hop_length = 512. After obtaining the Mel spectrum, the fixed-length spectral information is randomly taken and sent to the network training.
[0075] (2) The CNN in the spectrum transcription module is composed of three two-dimensional convolutions and two pooling layers, and the convolution kernel size of the convolution layer is 3; the RNN is composed of a BiLSTM with 128 units, and the BiLSTM is used to extract context information in the time dimension, and integrate the features of each frame into long and short-term memory features.
[0076] As shown in Figure 1 The final output dimensions of the above four branches are all 320*88, and the corresponding labels are compared to obtain four different losses.
[0077] In the field of piano transcription, the provided labels are generally 4 columns, which are the starting time of the note, the ending time of the note, the pressed piano key and the key speed, for example [1.778 2.635 35 40], which means that the 35th piano key is pressed from 1.778s-2.635s, and the key speed is 40. The time mapping (the ratio of the sampling rate to the frame length is multiplied by the time) to the corresponding note frame is
[5682] , which means that the 35th piano key is pressed from the 56th frame to the 82nd frame. For the 35th piano key, the first frame pressed in this time period is the 56th frame, and the label of the starting branch is assigned to the 56th frame. The label of the frame-level branch is assigned to the 56th frame to the 82nd frame, and the label of the end branch is assigned to the 82nd frame.
[0078] Among them, for the MIDI format piano music, the key strength is from 0-127, so the key speed is normalized by dividing by 128 as the label value of the speed.
[0079] 2. Secondly, the corresponding skeleton features are obtained through the piano performance video.
[0080] The hand skeleton sequence shape constructed by the application is 5*3*21, and the skeleton sequence shape formed by two hands is 5*3*42, wherein 5 represents 5 consecutive frame key point images, 42 represents 42 key points of two hands, and 3 is the three-dimensional coordinates of each key point.
[0081] As shown in Figure 6As shown, the ST-GCN network workflow is as follows: the input skeleton sequence shape is 5*3*42, and is sequentially subjected to multiple ST-GCN units, wherein the time domain convolution layer step of the 4th and 7th ST-GCN units is 2, for reducing the time dimension of the input feature map, and the remaining ST-GCN units only change the channel dimension of the feature map, for feature extraction of the input data, and finally the output feature map size of the ST-GCN network is 1*21*256. Then the feature is sent to the fully connected layer of the ST-GCN network for feature integration and completion of multi-label classification, and the output is the probability value of each finger having a pressing action in the intermediate frame, and the dimension is 10, that is, the skeleton transcription feature, which is compared with the label of the skeleton transcription module (i.e., the posture learning label) to obtain the loss value, and the parameter meaning is the same as formula (1). The label here is obtained by the finger key matching module, for example, the 35th and 53rd keys are pressed in a frame, corresponding to the 3rd and 8th fingers, and the label of the skeleton transcription module of the frame is [0, 0, 1, 0, 0, 0, 0, 1, 0, 0]
[0082] 3、Finally, the audio transcription feature and the skeleton transcription feature are fused to obtain the final transcription result. The first part is feature matching, and there are two problems in feature matching: first, the audio feature extraction extracts 320 continuous frames of features, but the skeleton feature extraction module obtains the features of the intermediate frame through five continuous frames, so it is necessary to match in the time dimension; second, the frame rates of audio and video are different, and the number of audio feature frames is more than that of video feature frames in the same time scale. The solutions to the two problems are as follows:
[0083] For the first problem, the data corresponding to 320 frames of pictures are simultaneously input into the skeleton feature extraction module to obtain features with the same dimension as the audio, and the size is batchsize x 320 x 10.
[0084] For the second problem, since the audio frame rate is often higher than the video frame rate, the video data is copied in the time dimension to ensure that each audio frame has a corresponding video frame. The corresponding relationship between the audio frame number and the video frame number is shown as follows, wherein i represents the i-th frame of audio of the whole piano music, index identifies the index-th frame of video corresponding to i, and ratio is the video sampling rate divided by the audio frame rate.
[0085] index = round(i*ratio)
[0086] The second part is a feature fusion module. Since the skeleton module mainly judges the key state, rather than the time of pressing and ending, we splice the audio frame feature and the skeleton feature, and then obtain integrated frame features through two convolution layers and a fully connected layer. The start, end, and speed features directly use the output extracted from the audio feature without fusion. The feature of the fusion module is shown in equation (7), where f p,t is the frame-level state value of the fusion model at time t for the pitch p, is the predicted frame-level state probability. The total loss is calculated as shown in equation (8). The loss of each single modality is obtained respectively, and the features of different modalities are projected into similar semantic spaces as much as possible to reduce the randomness of subsequent multi-modal feature fusion.
[0087] Finally, the probability obtained by transcription is compared with a threshold to obtain the pitch at the corresponding time, which is used to judge the accuracy of piano performance. As shown in equation (9), compared with the way of using only the audio transcription module for piano transcription, the multi-modal piano transcription method proposed in the embodiment effectively reduces the problems of multiple detections outside the hand range and missed detections within the hand range caused by using only the audio transcription module for transcription by using the position information of the key points through graph convolution, and obtains better piano transcription results. Figure 7
[0088] The embodiment of the present application provides a multi-modal piano transcription method based on graph convolution, which comprises:
[0089] Obtaining a trained multi-modal piano transcription device, wherein the multi-modal piano transcription device is trained by the training method of the multi-modal piano transcription device based on graph convolution according to any one of the above embodiments;
[0090] Inputting a piano performance video to be transcribed into the training method of the multi-modal piano transcription device based on graph convolution to obtain a transcription result.
[0091] The embodiment of the present application provides a computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions run on a computer, the computer executes the training method of the multi-modal piano transcription device based on graph convolution according to any one of the above embodiments or the multi-modal piano transcription method based on graph convolution according to any one of the above embodiments.
[0092] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A multimodal piano transcription device based on graph convolution, characterized in that, include: The audio transcription module is used to convert piano performance audio into spectral information, and then transcribe it after slicing it according to a preset duration to obtain pitch frame-level detection results; The hand skeleton feature transcription module is used to perform pose learning based on the input spatiotemporal map using the ST-GCN network and output the hand skeleton feature transcription results; wherein, the spatiotemporal map includes the spatial and temporal connectivity of hand key points in consecutive T frames of a piano performance video; The feature fusion transcription module is used to match the pitch frame-level detection results with the hand skeleton feature transcription results in terms of time dimension and frame rate to obtain the transcription results of the piano performance video.
2. The apparatus as claimed in claim 1, characterized in that, The hand skeletal feature transcription module includes: The key point detection module is used to extract the key points of the hands in each frame of the piano performance video; The pose learning module based on the ST-GCN network is used to learn pose based on the input spatiotemporal map and output the transcription results of hand skeleton features.
3. The apparatus as described in claim 2, characterized in that, The audio transcription module includes a preprocessing module and a spectral transcription module; The preprocessing module is used to sample the audio and perform time-frequency transformation to obtain spectral information, and then randomly slice the spectral information according to a preset duration as input to the spectral transcription module; The spectrum transcription module includes a multi-tone start event detection module, a multi-tone pitch frame-level detection module, a multi-tone end event detection module, and a key press speed detection module; The multi-tone start event detection module and the multi-tone end event detection module both include a CNN network and an RNN network, which are used to obtain the detection results of the start event and the end event based on the input, respectively. The multi-tone tuning high-frame-level detection module includes a CNN network, a first RNN network, and a second RNN network. The detection results of the start event and the end event are concatenated with the results obtained after the input is processed by the CNN network and the first RNN network, and used as the input of the second RNN network to obtain the pitch frame-level detection result; The key press speed detection module includes a CNN network, which is used to obtain key press speed detection results based on the input.
4. A training method for a multimodal piano transcription device based on graph convolution, characterized in that, The multimodal piano transcription device is the graph convolution-based multimodal piano transcription device as described in claim 3, and the training method includes: The multimodal piano transcription device is trained with the goal of minimizing the differences between the detection results of the start event, pitch frame level, end event, and key speed and their respective real labels, the differences between the transcription results of the hand skeleton features and the posture learning labels, and the differences between the transcription results of the piano performance video and their real labels.
5. The method as described in claim 4, characterized in that, The true labels for the start event, pitch frame level, end event, key speed detection results, and piano performance video transcription results are all obtained by labeling based on the piano performance video; The posture learning labels are obtained by matching the fingertip key points of a single frame image in a piano performance video with the positions of the piano keys.
6. A multimodal piano transcription method based on graph convolution, characterized in that, include: A trained multimodal piano transcription device is obtained, wherein the multimodal piano transcription device is trained by the training method of the graph convolution-based multimodal piano transcription device according to any one of claims 4-5; The piano performance video to be transcribed is input into the training method of the graph convolution-based multimodal piano transcription device to obtain the transcription result.
7. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 4-5, or the method as described in claim 6.