Piano playing robot control method and system

Abstract

The invention provides a piano playing robot control method and system in the technical field of piano playing robots. The method comprises the following steps: S1, creating a piano music transcription model and a key positioning model; s2, acquiring a large amount of historical piano music and historical key images, and constructing a music data set and an image data set; s3, training the piano music transcription model through the music data set, and training the key positioning model through the image data set; s4, deploying the trained piano music transcription model and the piano key positioning model to a piano playing robot; and S5, the piano playing robot obtains input real-time piano music, inputs the real-time piano music into the piano music transcription model to obtain a transcription file, inputs the collected real-time piano key image into the piano key positioning model to obtain a piano key physical position, and performs piano playing based on the transcription file and the piano key physical position. The piano playing device has the advantage that the piano playing quality is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of piano playing robots, and in particular to a piano playing robot control method and system. Background Art

[0002] A piano-playing robot is a high-tech device that combines precision mechanics, sensor technology, and complex algorithms. It can simulate the movements of a human piano player and perform complex musical performances. Piano-playing robots' performances involve piano music transcription and key visual localization. Piano music transcription is the process of converting piano music (such as audio files or sheet music) into a sequence of instructions that the piano-playing robot can understand and execute. For example, an audio file of a piano piece is converted into a series of notes, durations, and dynamics. This information is the foundation of the piano-playing robot's performance, telling it when to press which key, how long to press it, and how hard to press it. Without accurate piano music transcription, the piano-playing robot cannot determine the specific content to play. Key visual localization allows the piano-playing robot to precisely determine the location of each key, allowing its robotic arm or fingers to accurately press the correct key. Piano music transcription and key visual localization are key technologies for piano-playing robots to achieve precise performance, adapt to complex environments, and enhance their intelligence. Together, they provide the piano-playing robot with its "brain" and "eyes," enabling it to efficiently and flexibly complete various piano-playing tasks.

[0003] However, the Onsets and Frames model used in existing piano music transcription relies on a bidirectional LSTM (BiLSTM), which results in high computational complexity and many parameters, leading to poor real-time performance. Furthermore, existing piano key visual positioning is mostly based on general computer vision solutions, which have limited positioning accuracy, thus affecting the quality of piano performance.

[0004] Therefore, how to provide a piano playing robot control method and system to improve the piano playing quality has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a piano playing robot control method and system to improve the piano playing quality.

[0006] In a first aspect, the present invention provides a piano playing robot control method, comprising the following steps:

[0007] Step S1, creating a piano music transcription model and a piano key positioning model, setting a transcription loss function of the piano music transcription model, and setting a positioning loss function of the piano key positioning model;

[0008] Step S2: Acquire a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music dataset, and pre-process each of the historical piano key images to construct an image dataset;

[0009] Step S3, training a piano music transcription model using the music dataset and the transcription loss function, and training a key positioning model using the image dataset and the positioning loss function;

[0010] Step S4: deploying the trained piano music transcription model and key positioning model to the piano playing robot;

[0011] Step S5: The piano playing robot obtains input real-time piano music, inputs the real-time piano music into the deployed piano music transcription model to obtain a transcription file, inputs the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and performs piano performance based on the transcription file and the physical positions of the keys.

[0012] Furthermore, in step S1, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module, and a fault tolerance module;

[0013] The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module;

[0014] The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; the third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music;

[0015] The convolution stack is constructed by a six-layer convolutional network, and each layer of the convolutional network is constructed based on a 3×3 convolution kernel, batch normalization and ReLU activation function; the first timing modeling module and the second timing modeling module are both constructed by unidirectional LSTM.

[0016] Furthermore, in step S1, the key positioning model is constructed based on the image preprocessing module and the key position recognition module;

[0017] The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker;

[0018] The grayscale algorithm is:

[0019] Igray=0.299R+0.587G+0.114B;

[0020] Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel;

[0021] The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm;

[0022] The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, obtain a square image carrying ArUco tags, parse the binary coding structure of the square image through a preset dictionary to obtain the tag ID of the ArUco tag, perform a check operation through the check bit carried by the tag ID, and map the ArUco tags in the key image to the physical position of the key based on the tag ID.

[0023] Furthermore, the step S2 is specifically as follows:

[0024] A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

[0025] Furthermore, the step S3 is specifically as follows:

[0026] Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes;

[0027] The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

[0028] In a second aspect, the present invention provides a piano playing robot control system, comprising the following modules:

[0029] A model creation module, configured to create a piano music transcription model and a piano key positioning model, set a transcription loss function for the piano music transcription model, and set a positioning loss function for the piano key positioning model;

[0030] A data set construction module is used to obtain a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music data set, and pre-process each of the historical piano key images to construct an image data set;

[0031] A model training module, configured to train a piano music transcription model using the music dataset and the transcription loss function, and to train a key positioning model using the image dataset and the positioning loss function;

[0032] A model deployment module, for deploying the trained piano music transcription model and key positioning model to a piano playing robot;

[0033] The piano playing module is used for the piano playing robot to obtain input real-time piano music, input the real-time piano music into the deployed piano music transcription model to obtain a transcription file, input the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and perform piano playing based on the transcription file and the physical positions of the keys.

[0034] Furthermore, in the model creation module, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module, and a fault tolerance module;

[0035] The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module;

[0036] The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; the third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music;

[0037] The convolution stack is constructed by a six-layer convolutional network, and each layer of the convolutional network is constructed based on a 3×3 convolution kernel, batch normalization and ReLU activation function; the first timing modeling module and the second timing modeling module are both constructed by unidirectional LSTM.

[0038] Furthermore, in the model creation module, the key positioning model is constructed based on the image preprocessing module and the key position recognition module;

[0039] The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker;

[0040] The grayscale algorithm is:

[0041] Igray=0.299R+0.587G+0.114B;

[0042] Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel;

[0043] The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm;

[0044] The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, obtain a square image carrying ArUco tags, parse the binary coding structure of the square image through a preset dictionary to obtain the tag ID of the ArUco tag, perform a check operation through the check bit carried by the tag ID, and map the ArUco tags in the key image to the physical position of the key based on the tag ID.

[0045] Furthermore, the dataset construction module is specifically used to:

[0046] A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

[0047] Furthermore, the model training module is specifically used to:

[0048] Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes;

[0049] The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

[0050] The advantages of the present invention are:

[0051] 1. Create a piano music transcription model and a key positioning model, set the transcription loss function of the piano music transcription model, and set the positioning loss function of the key positioning model; then obtain a large amount of historical piano music and historical key images, pre-process each historical piano music to construct a music dataset, and pre-process each historical key image to construct an image dataset; then train the piano music transcription model with the music dataset and the transcription loss function, and train the key positioning model with the image dataset and the positioning loss function, and deploy the trained piano music transcription model and key positioning model to the piano playing robot; the piano playing robot obtains the input real-time piano music The real-time piano music is input into the deployed piano music transcription model to obtain a transcription file, the collected real-time piano key images are input into the key positioning model to obtain the physical positions of the keys, and the piano performance is performed based on the transcription file and the physical positions of the keys. That is, the piano playing robot is controlled to perform through the pre-trained piano music transcription model and key positioning model. Since the piano music transcription model replaces the traditional bidirectional LSTM with a unidirectional LSTM, the model parameters are effectively reduced, thereby greatly reducing the computational burden of the piano music transcription model. The key positioning model combines a variety of image preprocessing measures and perspective transformation correction to effectively improve the key positioning accuracy, thereby greatly improving the quality of piano performance.

[0052] 2. The piano music transcription model processes audio signals, while the key positioning model processes visual information, achieving audio-visual collaborative control. The fusion of dual-model outputs (transcription files and physical locations) significantly improves the spatiotemporal consistency of the robot's performance. The music transcription model focuses on temporal feature extraction through modules such as LSTM, while the key positioning model focuses on spatial positioning through image processing. This clear division of labor reduces system complexity, while ensuring end-to-end control efficiency through real-time collaboration after deployment.

[0053] 3. The starting point prediction module introduces time intervals to dynamically adjust the detection sensitivity, effectively resolving the contradiction between false detection and missed detection in dense and sparse segments of notes. Compared with the fixed threshold method, the recognition accuracy is significantly improved.

[0054] 4. By setting up a fault-tolerant module, dynamic threshold compensation and forward-looking window smoothing operations are adopted. Dynamic threshold compensation is to adaptively adjust the judgment threshold according to historical prediction results to resist environmental noise interference; window smoothing operation is to eliminate instantaneous misjudgment through multi-frame correlation to ensure the temporal consistency of note starting point detection.

[0055] 5. A time-frequency feature extraction network is constructed by stacking six convolution layers (3x3 convolution kernel + batch normalization + ReLU). Compared with shallower networks, the receptive field is expanded layer by layer, taking into account both local details and global features; batch normalization effectively suppresses overfitting and adapts to the differences in different piano timbres.

[0056] 6. By adopting the cascade processing of grayscale → hybrid noise reduction → threshold segmentation → edge enhancement, the following are achieved: ① Gaussian filtering + median filtering hybrid noise reduction: synchronously eliminates Gaussian noise and impulse noise; ② Otsu combined with adaptive thresholding: solves the binarization error caused by uneven lighting; ③ Canny edge enhancement: retains the geometric features of the piano keys while suppressing texture interference.

[0057] 7. During the key positioning process, binary code structure verification is used in combination with dictionary parsing to achieve coding error correction; check digit verification is used to prevent ID misidentification due to image distortion; homography matrix perspective correction is used to eliminate the impact of camera angle deviation and improve the mark recognition rate of low-quality images.

[0058] 8. Music data is processed using frame segmentation and silence removal to eliminate environmental noise interference. Image data is normalized and denoised to ensure model input consistency and reduce the risk of data drift during deployment.

[0059] 9. Independent verification of the piano music transcription model and the key localization model avoids cross-modal error coupling. A dual confidence testing mechanism simultaneously evaluates accuracy and confidence intervals to ensure model reliability. Dynamic control of the loss threshold prevents overfitting and underfitting, improving generalization capabilities.

[0060] 10. By setting the piano music transcription model to use unidirectional LSTM (instead of bidirectional), the inference delay is reduced while ensuring the timing modeling capability; the perspective transformation and marker recognition of the key positioning module are set to use a lightweight algorithm to meet millisecond-level response requirements.

[0061] 11. Through the dual-modal collaborative control of a piano music transcription model and a key localization model, high-precision fusion of audio and visual information is achieved: The piano music transcription model employs dynamic sensitivity adjustment, a multi-stage normalization architecture, and a fault-tolerant compensation mechanism to significantly improve the robustness of note onset detection. The key localization model combines hybrid noise reduction, multi-threshold segmentation, and ArUco marker verification technology to effectively overcome illumination interference and image distortion, ensuring the accuracy of key physical positioning. Through standardized data preprocessing, an independent model training and verification system, and a lightweight deployment design, it balances real-time response and generalization capabilities, while its modular architecture supports cross-scenario expansion. Its innovation is reflected in the cross-domain collaborative optimization of music timing modeling and visual localization technologies, as well as the combined innovation of detailed improvements such as dynamic fault tolerance and hybrid image processing, resulting in outstanding advantages in performance accuracy, anti-interference performance, and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] Figure 1 The present invention is a flow chart of a piano playing robot control method.

[0064] Figure 2 The present invention is a structural diagram of a piano playing robot control system.

[0065] Figure 3 It is a schematic diagram of the piano music transcription of the present invention.

[0066] Figure 4 Schematic diagram of the keyboard image of the present invention. DETAILED DESCRIPTION

[0067] The technical solution in the embodiments of the present application has the following overall idea: the piano playing robot is controlled to perform through a pre-trained piano music transcription model and a key positioning model. Since the piano music transcription model replaces the traditional bidirectional LSTM with a unidirectional LSTM, the model parameters are effectively reduced, thereby greatly reducing the computational burden of the piano music transcription model; and the key positioning model combines a variety of image preprocessing measures and perspective transformation correction to effectively improve the key positioning accuracy, thereby improving the quality of piano performance.

[0068] Please refer to Figures 1 to 4 As shown, a preferred embodiment of a piano playing robot control method of the present invention includes the following steps:

[0069] Step S1, creating a piano music transcription model and a piano key positioning model, setting a transcription loss function of the piano music transcription model, and setting a positioning loss function of the piano key positioning model;

[0070] The piano music transcription model processes audio signals, and the key positioning model processes visual information to achieve audio-visual collaborative control; the fusion of dual model outputs (transcription files and physical locations) significantly improves the spatiotemporal consistency of the robot's performance; the music transcription model focuses on temporal feature extraction through modules such as LSTM, while the key positioning model focuses on spatial positioning through image processing. The clear division of labor reduces system complexity, and end-to-end control efficiency is guaranteed through real-time collaboration after deployment.

[0071] Step S2: Acquire a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music dataset, and pre-process each of the historical piano key images to construct an image dataset;

[0072] Step S3, training a piano music transcription model using the music dataset and the transcription loss function, and training a key positioning model using the image dataset and the positioning loss function;

[0073] Step S4: deploying the trained piano music transcription model and key positioning model to the piano playing robot;

[0074] Step S5: The piano playing robot obtains input real-time piano music, inputs the real-time piano music into the deployed piano music transcription model to obtain a transcription file, inputs the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and performs piano performance based on the transcription file and the physical positions of the keys.

[0075] Through the dual-modal collaborative control of the piano music transcription model and the key positioning model, high-precision fusion of audio and visual information is achieved: the piano music transcription model adopts dynamic sensitivity adjustment, multi-stage normalization architecture and fault-tolerant compensation mechanism to significantly improve the robustness of note starting point detection; the key positioning model combines hybrid noise reduction, multi-threshold segmentation and ArUco marker verification technology to effectively overcome the problems of illumination interference and image distortion, ensuring the accuracy of the physical positioning of the keys; through standardized data preprocessing, independent model training and verification system and lightweight deployment design, it takes into account both real-time response and generalization capabilities, while the modular architecture supports cross-scenario expansion. Its innovation is reflected in the cross-domain collaborative optimization of music timing modeling and visual positioning technology, as well as the combined innovation of details such as dynamic fault tolerance and hybrid image processing. It has outstanding advantages in performance accuracy, anti-interference performance and system reliability.

[0076] In step S1, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module, and a fault tolerance module;

[0077] The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module;

[0078] By introducing the time interval into the starting point prediction module to dynamically adjust the detection sensitivity, the contradiction between false detection and missed detection in dense and sparse note segments can be effectively resolved. Compared with the fixed threshold method, the recognition accuracy is significantly improved.

[0079] By setting up the fault-tolerant module, dynamic threshold compensation and forward-looking window smoothing operations are adopted. Dynamic threshold compensation is to adaptively adjust the judgment threshold according to historical prediction results to resist environmental noise interference; window smoothing operation is to eliminate instantaneous misjudgment through multi-frame association to ensure the temporal consistency of note starting point detection.

[0080] The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; the third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music;

[0081] To address the limitations of unidirectional LSTM in onset detection, a dynamic compensation mechanism based on temporal distance was designed. This mechanism adjusts detection sensitivity by analyzing the time interval between the current detection position and historical onsets. When detecting shorter time intervals, the threshold is appropriately lowered to improve the ability to capture rapid note sequences; when the intervals are longer, a stricter detection standard is adopted to avoid false triggers. Simultaneously, the frame probability information output by the persistent state branch is used to perform cross-validation using a three-frame sliding window to ensure the reliability of the onset detection results. Notably, the original unidirectional LSTM structure is maintained in the processing of the persistent state branch, without the introduction of additional compensation mechanisms. This is because persistent state predictions inherently have strong temporal continuity, and the modeling capabilities of the unidirectional LSTM can already meet basic requirements. Furthermore, maintaining the simplicity of this branch helps control the overall computational complexity.

[0082] The convolutional stack is constructed using a six-layer convolutional network. Each layer is based on a 3×3 convolution kernel, batch normalization, and a ReLU activation function. This network gradually builds a feature representation with rich time-frequency characteristics. This design effectively captures the harmonic structure and transient characteristics of piano audio. The first and second time series modeling modules are both constructed using unidirectional LSTMs. Specifically, two independent unidirectional LSTM branches are established: one specifically handles note onset detection, and the other tracks note duration. The parameters of the two branches are completely independent, ensuring that each branch can focus on modeling specific features.

[0083] A time-frequency feature extraction network is constructed by stacking six convolution layers (3x3 convolution kernel + batch normalization + ReLU). Compared with shallower networks, the receptive field is expanded layer by layer, taking into account both local details and global features; batch normalization effectively suppresses overfitting and adapts to the differences in different piano timbres.

[0084] By setting the piano music transcription model to use unidirectional LSTM (instead of bidirectional), the inference delay is reduced while ensuring the timing modeling capability; the perspective transformation and marker recognition of the key positioning module are set to use lightweight algorithms to meet millisecond-level response requirements.

[0085] In step S1, the key positioning model is constructed based on the image preprocessing module and the key position recognition module;

[0086] The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker;

[0087] The grayscale algorithm is:

[0088] Igray=0.299R+0.587G+0.114B;

[0089] Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel;

[0090] This algorithm fully considers the differences in sensitivity of the human visual system to different color wavelengths, with the green channel having the largest weight, followed by red, and the blue least. This weight distribution can retain important edge information in the image to the greatest extent. Compared with simple arithmetic averaging or maximum value methods, the weighted grayscale algorithm has obvious advantages in preserving image details, especially when processing scenes with obvious black and white contrast features such as piano keyboards.

[0091] The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm;

[0092] Noise suppression uses a combined strategy of Gaussian and median filtering to address different types of noise interference. First, the image is smoothed using a 5×5 Gaussian filter with a standard deviation of 1.0. This filter constructs a convolution kernel using a two-dimensional Gaussian function, which can effectively suppress Gaussian noise in the image while maintaining good edge characteristics. The image is then processed using a 3×3 median filter. This nonlinear filter replaces the central pixel value with the median of neighboring pixels, making it particularly suitable for removing impulse noise such as salt and pepper noise. This combined filtering strategy fully leverages the advantages of both filters, eliminating noise while maximally preserving image detail features, providing a cleaner input image for subsequent processing.

[0093] Threshold segmentation adopts a combination of the maximum inter-class difference method, the Otsu global threshold method and the adaptive local threshold method to deal with the common problem of uneven illumination in piano keyboard images; the Otsu algorithm is an automatic threshold selection method based on histogram analysis, which realizes image binarization by finding the threshold that maximizes the inter-class variance. This method has high computational efficiency and works well under conditions of uniform illumination; to address the problem of uneven illumination that may occur in actual scenes, an adaptive threshold method based on local mean is also adopted. This method divides the image into local areas of 31×31 pixels and independently calculates the threshold in each area to adapt to the illumination changes in different areas; this combination of global and local strategies not only ensures the consistency of the overall segmentation effect, but also effectively deals with the problems caused by local illumination changes.

[0094] Edge enhancement uses the classic Canny edge detection algorithm, which is considered one of the standard algorithms in the field of edge detection. First, a 5×5 Gaussian filter is used to smooth the image to eliminate the interference of small noise. Then, the Sobel operator is used to calculate the gradient amplitude and direction of the image. The Sobel operator calculates the gradient in the horizontal and vertical directions respectively through two 3×3 convolution kernels. Next, non-maximum suppression processing is performed to retain the local maximum points in the gradient direction and refine the edges. Finally, the true edge is determined through dual-threshold detection, where the low threshold is set to 30 and the high threshold is set to 90. This setting can effectively suppress false edge responses caused by noise while ensuring edge continuity.

[0095] The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, eliminate other interfering contours, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, eliminate the deformation caused by the shooting angle, and obtain a square image carrying the ArUco mark. The binary encoding structure of the square image is parsed through a preset dictionary to obtain the tag ID of the ArUco tag. The check bit carried by the tag ID is used to perform a check operation. Based on the tag ID, the ArUco tag in the key image is mapped to the physical position of the key. In specific implementation, the verified tag ID is further used to accurately locate the coordinates of the four corner points using a sub-pixel algorithm. Then, combined with the preset tag-key mapping relationship, the image coordinates are converted into the corresponding physical position of the key, completing the precise mapping from image space to piano keyboard space.

[0096] By adopting the cascade processing of grayscale → hybrid noise reduction → threshold segmentation → edge enhancement, the following are achieved: ① Gaussian filtering + median filtering hybrid noise reduction: synchronously eliminating Gaussian noise and impulse noise; ② Otsu combined with adaptive threshold: solving the binarization error caused by uneven lighting; ③ Canny edge enhancement: retaining the geometric features of the piano keys while suppressing texture interference.

[0097] During the key positioning process, binary coding structure verification is used in combination with dictionary parsing to achieve coding error correction; check digit verification is used to prevent ID misidentification due to image distortion; and homography matrix perspective correction is used to eliminate the impact of camera angle deviation and improve the mark recognition rate of low-quality images.

[0098] The step S2 is specifically as follows:

[0099] A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

[0100] Music data is processed using frame segmentation and silence removal to eliminate environmental noise interference. Image data is normalized and denoised to ensure model input consistency and reduce the risk of data drift during deployment.

[0101] The step S3 is specifically as follows:

[0102] Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes;

[0103] The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

[0104] Independent verification of the piano music transcription model and the key positioning model avoids cross-modal error coupling; a dual confidence testing mechanism simultaneously evaluates accuracy and confidence intervals to ensure model reliability; and dynamic control of the loss threshold prevents overfitting and underfitting, thereby improving generalization capabilities.

[0105] A preferred embodiment of a piano playing robot control system of the present invention includes the following modules:

[0106] A model creation module, configured to create a piano music transcription model and a piano key positioning model, set a transcription loss function for the piano music transcription model, and set a positioning loss function for the piano key positioning model;

[0107] The piano music transcription model processes audio signals, and the key positioning model processes visual information to achieve audio-visual collaborative control; the fusion of dual model outputs (transcription files and physical locations) significantly improves the spatiotemporal consistency of the robot's performance; the music transcription model focuses on temporal feature extraction through modules such as LSTM, while the key positioning model focuses on spatial positioning through image processing. The clear division of labor reduces system complexity, and end-to-end control efficiency is guaranteed through real-time collaboration after deployment.

[0108] A data set construction module is used to obtain a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music data set, and pre-process each of the historical piano key images to construct an image data set;

[0109] A model training module, configured to train a piano music transcription model using the music dataset and the transcription loss function, and to train a key positioning model using the image dataset and the positioning loss function;

[0110] A model deployment module, for deploying the trained piano music transcription model and key positioning model to a piano playing robot;

[0111] The piano playing module is used for the piano playing robot to obtain input real-time piano music, input the real-time piano music into the deployed piano music transcription model to obtain a transcription file, input the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and perform piano playing based on the transcription file and the physical positions of the keys.

[0112] Through the dual-modal collaborative control of the piano music transcription model and the key positioning model, high-precision fusion of audio and visual information is achieved: the piano music transcription model adopts dynamic sensitivity adjustment, multi-stage normalization architecture and fault-tolerant compensation mechanism to significantly improve the robustness of note starting point detection; the key positioning model combines hybrid noise reduction, multi-threshold segmentation and ArUco marker verification technology to effectively overcome the problems of illumination interference and image distortion, ensuring the accuracy of the physical positioning of the keys; through standardized data preprocessing, independent model training and verification system and lightweight deployment design, it takes into account both real-time response and generalization capabilities, while the modular architecture supports cross-scenario expansion. Its innovation is reflected in the cross-domain collaborative optimization of music timing modeling and visual positioning technology, as well as the combined innovation of details such as dynamic fault tolerance and hybrid image processing. It has outstanding advantages in performance accuracy, anti-interference performance and system reliability.

[0113] In the model creation module, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module and a fault tolerance module;

[0114] The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module;

[0115] By introducing the time interval into the starting point prediction module to dynamically adjust the detection sensitivity, the contradiction between false detection and missed detection in dense and sparse note segments can be effectively resolved. Compared with the fixed threshold method, the recognition accuracy is significantly improved.

[0116] By setting up the fault-tolerant module, dynamic threshold compensation and forward-looking window smoothing operations are adopted. Dynamic threshold compensation is to adaptively adjust the judgment threshold according to historical prediction results to resist environmental noise interference; window smoothing operation is to eliminate instantaneous misjudgment through multi-frame association to ensure the temporal consistency of note starting point detection.

[0117] The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; the third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music;

[0118] To address the limitations of unidirectional LSTM in onset detection, a dynamic compensation mechanism based on temporal distance was designed. This mechanism adjusts detection sensitivity by analyzing the time interval between the current detection position and historical onsets. When detecting shorter time intervals, the threshold is appropriately lowered to improve the ability to capture rapid note sequences; when the intervals are longer, a stricter detection standard is adopted to avoid false triggers. Simultaneously, the frame probability information output by the persistent state branch is used to perform cross-validation using a three-frame sliding window to ensure the reliability of the onset detection results. Notably, the original unidirectional LSTM structure is maintained in the processing of the persistent state branch, without the introduction of additional compensation mechanisms. This is because persistent state predictions inherently have strong temporal continuity, and the modeling capabilities of the unidirectional LSTM can already meet basic requirements. Furthermore, maintaining the simplicity of this branch helps control the overall computational complexity.

[0119] The convolutional stack is constructed using a six-layer convolutional network, each built with a 3×3 convolution kernel, batch normalization, and ReLU activation function. Both the first and second time series modeling modules are constructed using unidirectional LSTMs, gradually building feature representations with rich time-frequency characteristics. This design effectively captures the harmonic structure and transient characteristics of piano audio. This involves establishing two independent unidirectional LSTM branches: one dedicated to note onset detection and the other to tracking note duration. The parameters of the two branches are completely independent, ensuring that each branch can focus on modeling specific features.

[0120] A time-frequency feature extraction network is constructed by stacking six convolution layers (3x3 convolution kernel + batch normalization + ReLU). Compared with shallower networks, the receptive field is expanded layer by layer, taking into account both local details and global features; batch normalization effectively suppresses overfitting and adapts to the differences in different piano timbres.

[0121] By setting the piano music transcription model to use unidirectional LSTM (instead of bidirectional), the inference delay is reduced while ensuring the timing modeling capability; the perspective transformation and marker recognition of the key positioning module are set to use lightweight algorithms to meet millisecond-level response requirements.

[0122] In the model creation module, the key positioning model is constructed based on the image preprocessing module and the key position recognition module;

[0123] The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker;

[0124] The grayscale algorithm is:

[0125] Igray=0.299R+0.587G+0.114B;

[0126] Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel;

[0127] This algorithm fully considers the differences in sensitivity of the human visual system to different color wavelengths, with the green channel having the largest weight, followed by red, and the blue least. This weight distribution can retain important edge information in the image to the greatest extent. Compared with simple arithmetic averaging or maximum value methods, the weighted grayscale algorithm has obvious advantages in preserving image details, especially when processing scenes with obvious black and white contrast features such as piano keyboards.

[0128] The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm;

[0129] Noise suppression uses a combined strategy of Gaussian and median filtering to address different types of noise interference. First, the image is smoothed using a 5×5 Gaussian filter with a standard deviation of 1.0. This filter constructs a convolution kernel using a two-dimensional Gaussian function, which can effectively suppress Gaussian noise in the image while maintaining good edge characteristics. The image is then processed using a 3×3 median filter. This nonlinear filter replaces the central pixel value with the median of neighboring pixels, making it particularly suitable for removing impulse noise such as salt and pepper noise. This combined filtering strategy fully leverages the advantages of both filters, eliminating noise while maximally preserving image detail features, providing a cleaner input image for subsequent processing.

[0130] Threshold segmentation adopts a combination of the maximum inter-class difference method, the Otsu global threshold method and the adaptive local threshold method to deal with the common problem of uneven illumination in piano keyboard images; the Otsu algorithm is an automatic threshold selection method based on histogram analysis, which realizes image binarization by finding the threshold that maximizes the inter-class variance. This method has high computational efficiency and works well under conditions of uniform illumination; to address the problem of uneven illumination that may occur in actual scenes, an adaptive threshold method based on local mean is also adopted. This method divides the image into local areas of 31×31 pixels and independently calculates the threshold in each area to adapt to the illumination changes in different areas; this combination of global and local strategies not only ensures the consistency of the overall segmentation effect, but also effectively deals with the problems caused by local illumination changes.

[0131] Edge enhancement uses the classic Canny edge detection algorithm, which is considered one of the standard algorithms in the field of edge detection. First, a 5×5 Gaussian filter is used to smooth the image to eliminate the interference of small noise. Then, the Sobel operator is used to calculate the gradient amplitude and direction of the image. The Sobel operator calculates the gradient in the horizontal and vertical directions respectively through two 3×3 convolution kernels. Next, non-maximum suppression processing is performed to retain the local maximum points in the gradient direction and refine the edges. Finally, the true edge is determined through dual-threshold detection, where the low threshold is set to 30 and the high threshold is set to 90. This setting can effectively suppress false edge responses caused by noise while ensuring edge continuity.

[0132] The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, eliminate other interfering contours, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, eliminate the deformation caused by the shooting angle, and obtain a square image carrying the ArUco mark. The binary encoding structure of the square image is parsed through a preset dictionary to obtain the tag ID of the ArUco tag. The check bit carried by the tag ID is used to perform a check operation. Based on the tag ID, the ArUco tag in the key image is mapped to the physical position of the key. In specific implementation, the verified tag ID is further used to accurately locate the coordinates of the four corner points using a sub-pixel algorithm. Then, combined with the preset tag-key mapping relationship, the image coordinates are converted into the corresponding physical position of the key, completing the precise mapping from image space to piano keyboard space.

[0133] By adopting the cascade processing of grayscale → hybrid noise reduction → threshold segmentation → edge enhancement, the following are achieved: ① Gaussian filtering + median filtering hybrid noise reduction: synchronously eliminating Gaussian noise and impulse noise; ② Otsu combined with adaptive threshold: solving the binarization error caused by uneven lighting; ③ Canny edge enhancement: retaining the geometric features of the piano keys while suppressing texture interference.

[0134] During the key positioning process, binary coding structure verification is used in combination with dictionary parsing to achieve coding error correction; check digit verification is used to prevent ID misidentification due to image distortion; and homography matrix perspective correction is used to eliminate the impact of camera angle deviation and improve the mark recognition rate of low-quality images.

[0135] The dataset construction module is specifically used for:

[0136] A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

[0137] Music data is processed using frame segmentation and silence removal to eliminate environmental noise interference. Image data is normalized and denoised to ensure model input consistency and reduce the risk of data drift during deployment.

[0138] The model training module is specifically used for:

[0139] Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes;

[0140] The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

[0141] Independent verification of the piano music transcription model and the key positioning model avoids cross-modal error coupling; a dual confidence testing mechanism simultaneously evaluates accuracy and confidence intervals to ensure model reliability; and dynamic control of the loss threshold prevents overfitting and underfitting, thereby improving generalization capabilities.

[0142] In summary, the advantages of the present invention are:

[0143] 1. Create a piano music transcription model and a key positioning model, set the transcription loss function of the piano music transcription model, and set the positioning loss function of the key positioning model; then obtain a large amount of historical piano music and historical key images, pre-process each historical piano music to construct a music dataset, and pre-process each historical key image to construct an image dataset; then train the piano music transcription model with the music dataset and the transcription loss function, and train the key positioning model with the image dataset and the positioning loss function, and deploy the trained piano music transcription model and key positioning model to the piano playing robot; the piano playing robot obtains the input real-time piano music The real-time piano music is input into the deployed piano music transcription model to obtain a transcription file, the collected real-time piano key images are input into the key positioning model to obtain the physical positions of the keys, and the piano performance is performed based on the transcription file and the physical positions of the keys. That is, the piano playing robot is controlled to perform through the pre-trained piano music transcription model and key positioning model. Since the piano music transcription model replaces the traditional bidirectional LSTM with a unidirectional LSTM, the model parameters are effectively reduced, thereby greatly reducing the computational burden of the piano music transcription model. The key positioning model combines a variety of image preprocessing measures and perspective transformation correction to effectively improve the key positioning accuracy, thereby greatly improving the quality of piano performance.

[0144] 2. The piano music transcription model processes audio signals, while the key positioning model processes visual information, achieving audio-visual collaborative control. The fusion of dual-model outputs (transcription files and physical locations) significantly improves the spatiotemporal consistency of the robot's performance. The music transcription model focuses on temporal feature extraction through modules such as LSTM, while the key positioning model focuses on spatial positioning through image processing. This clear division of labor reduces system complexity, while ensuring end-to-end control efficiency through real-time collaboration after deployment.

[0145] 3. The starting point prediction module introduces time intervals to dynamically adjust the detection sensitivity, effectively resolving the contradiction between false detection and missed detection in dense and sparse segments of notes. Compared with the fixed threshold method, the recognition accuracy is significantly improved.

[0146] 4. By setting up a fault-tolerant module, dynamic threshold compensation and forward-looking window smoothing operations are adopted. Dynamic threshold compensation is to adaptively adjust the judgment threshold according to historical prediction results to resist environmental noise interference; window smoothing operation is to eliminate instantaneous misjudgment through multi-frame correlation to ensure the temporal consistency of note starting point detection.

[0147] 5. A time-frequency feature extraction network is constructed by stacking six convolution layers (3x3 convolution kernel + batch normalization + ReLU). Compared with shallower networks, the receptive field is expanded layer by layer, taking into account both local details and global features; batch normalization effectively suppresses overfitting and adapts to the differences in different piano timbres.

[0148] 6. By adopting the cascade processing of grayscale → hybrid noise reduction → threshold segmentation → edge enhancement, the following are achieved: ① Gaussian filtering + median filtering hybrid noise reduction: synchronously eliminates Gaussian noise and impulse noise; ② Otsu combined with adaptive thresholding: solves the binarization error caused by uneven lighting; ③ Canny edge enhancement: retains the geometric features of the piano keys while suppressing texture interference.

[0149] 7. During the key positioning process, binary code structure verification is used in combination with dictionary parsing to achieve coding error correction; check digit verification is used to prevent ID misidentification due to image distortion; homography matrix perspective correction is used to eliminate the impact of camera angle deviation and improve the mark recognition rate of low-quality images.

[0150] 8. Music data is processed using frame segmentation and silence removal to eliminate environmental noise interference. Image data is normalized and denoised to ensure model input consistency and reduce the risk of data drift during deployment.

[0151] 9. Independent verification of the piano music transcription model and the key localization model avoids cross-modal error coupling. A dual confidence testing mechanism simultaneously evaluates accuracy and confidence intervals to ensure model reliability. Dynamic control of the loss threshold prevents overfitting and underfitting, improving generalization capabilities.

[0152] 10. By setting the piano music transcription model to use unidirectional LSTM (instead of bidirectional), the inference delay is reduced while ensuring the timing modeling capability; the perspective transformation and marker recognition of the key positioning module are set to use a lightweight algorithm to meet millisecond-level response requirements.

[0153] 11. Through the dual-modal collaborative control of a piano music transcription model and a key localization model, high-precision fusion of audio and visual information is achieved: The piano music transcription model employs dynamic sensitivity adjustment, a multi-stage normalization architecture, and a fault-tolerant compensation mechanism to significantly improve the robustness of note onset detection. The key localization model combines hybrid noise reduction, multi-threshold segmentation, and ArUco marker verification technology to effectively overcome illumination interference and image distortion, ensuring the accuracy of key physical positioning. Through standardized data preprocessing, an independent model training and verification system, and a lightweight deployment design, it balances real-time response and generalization capabilities, while its modular architecture supports cross-scenario expansion. Its innovation is reflected in the cross-domain collaborative optimization of music timing modeling and visual localization technologies, as well as the combined innovation of detailed improvements such as dynamic fault tolerance and hybrid image processing, resulting in outstanding advantages in performance accuracy, anti-interference performance, and system reliability.

[0154] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A piano playing robot control method, characterized in that: The steps include: Step S1, creating a piano music transcription model and a piano key positioning model, setting a transcription loss function of the piano music transcription model, and setting a positioning loss function of the piano key positioning model; Step S2: Acquire a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music dataset, and pre-process each of the historical piano key images to construct an image dataset; Step S3, training a piano music transcription model using the music dataset and the transcription loss function, and training a key positioning model using the image dataset and the positioning loss function; Step S4: deploying the trained piano music transcription model and key positioning model to the piano playing robot; Step S5: The piano playing robot obtains input real-time piano music, inputs the real-time piano music into the deployed piano music transcription model to obtain a transcription file, inputs the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and performs piano performance based on the transcription file and the physical positions of the keys.

2. A piano playing robot control method as claimed in claim 1, characterized in that: In step S1, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module, and a fault tolerance module; The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module; The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; The third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music; The convolution stack is constructed by a six-layer convolutional network, and each layer of the convolutional network is constructed based on a 3×3 convolution kernel, batch normalization and ReLU activation function; the first timing modeling module and the second timing modeling module are both constructed by unidirectional LSTM.

3. A piano playing robot control method as claimed in claim 1, characterized in that: In step S1, the key positioning model is constructed based on the image preprocessing module and the key position recognition module; The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker; The grayscale algorithm is: Igray=0.299R+0.587G+0.114B; Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel; The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm; The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, obtain a square image carrying ArUco tags, parse the binary coding structure of the square image through a preset dictionary to obtain the tag ID of the ArUco tag, perform a check operation through the check bit carried by the tag ID, and map the ArUco tags in the key image to the physical position of the key based on the tag ID.

4. A piano playing robot control method as claimed in claim 1, characterized in that: The step S2 is specifically as follows: A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

5. The piano playing robot control method according to claim 1, wherein: The step S3 is specifically as follows: Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes; The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

6. A piano playing robot control system, characterized in that: Includes the following modules: A model creation module, configured to create a piano music transcription model and a piano key positioning model, set a transcription loss function for the piano music transcription model, and set a positioning loss function for the piano key positioning model; A data set construction module is used to obtain a large amount of historical piano music and historical piano key images, pre-process each of the historical piano music to construct a music data set, and pre-process each of the historical piano key images to construct an image data set; A model training module, configured to train a piano music transcription model using the music dataset and the transcription loss function, and to train a key positioning model using the image dataset and the positioning loss function; A model deployment module, for deploying the trained piano music transcription model and key positioning model to a piano playing robot; The piano playing module is used for the piano playing robot to obtain input real-time piano music, input the real-time piano music into the deployed piano music transcription model to obtain a transcription file, input the collected real-time piano key images into the key positioning model to obtain the physical positions of the keys, and perform piano playing based on the transcription file and the physical positions of the keys.

7. A piano playing robot control system as claimed in claim 6, characterized in that: In the model creation module, the piano music transcription model is constructed based on an input module, a feature extraction module, a first time series modeling module, a second time series modeling module, a first feature conversion module, a second feature conversion module, a third feature conversion module, a starting point prediction module, a frame prediction module and a fault tolerance module; The input module is used to convert the input piano music into a logarithmic Mel spectrum graph; the feature extraction module is used to extract time-frequency features from the logarithmic Mel spectrum graph through a convolution stack; the first time series modeling module is used to perform time series modeling on the time-frequency features to obtain a first hidden state feature; the first feature conversion module is used to perform a normalization operation on the first hidden state feature to obtain a first normalized feature; the starting point prediction module is used to perform a starting point prediction on the first normalized feature and dynamically adjust the detection sensitivity based on the time interval between the current detection position and the starting point; the fault tolerance module is used to perform dynamic threshold compensation and forward-looking window smoothing operations on the starting point output by the starting point prediction module; The second feature conversion module is used to perform a normalization operation on the time-frequency feature to obtain a second normalized feature; the second time series modeling module is used to perform time series modeling on the second normalized feature and the starting point output by the fault tolerance module to obtain a second hidden state feature; The third feature conversion module is used to perform a normalization operation on the second hidden state feature to obtain a third normalized feature; the frame prediction module is used to perform music frame prediction on the third normalized feature and output a transcription file corresponding to the piano music; The convolution stack is constructed by a six-layer convolutional network, and each layer of the convolutional network is constructed based on a 3×3 convolution kernel, batch normalization and ReLU activation function; the first timing modeling module and the second timing modeling module are both constructed by unidirectional LSTM.

8. A piano playing robot control system as claimed in claim 6, characterized in that: In the model creation module, the key positioning model is constructed based on the image preprocessing module and the key position recognition module; The image preprocessing module is used to sequentially perform grayscale conversion, noise suppression, threshold segmentation, and edge enhancement on the input piano key image; each key in the piano key image is provided with an ArUco marker; The grayscale algorithm is: Igray=0.299R+0.587G+0.114B; Among them, Igray represents the grayscale pixel value; R represents the red component of the pixel; G represents the green component of the pixel; B represents the blue component of the pixel; The noise suppression is combined with Gaussian filtering and median filtering; the threshold segmentation is combined with the maximum inter-class difference method, Otsu global threshold method and adaptive local threshold method; the edge enhancement adopts the Canny edge detection algorithm; The key position recognition module is used to perform contour detection on the key image output by the image preprocessing module to obtain quadrilateral candidate areas, calculate the homography matrix to perform perspective transformation correction on each of the quadrilateral candidate areas, obtain a square image carrying ArUco tags, parse the binary coding structure of the square image through a preset dictionary to obtain the tag ID of the ArUco tag, perform a check operation through the check bit carried by the tag ID, and map the ArUco tags in the key image to the physical position of the key based on the tag ID.

9. The piano playing robot control system according to claim 6, wherein: The dataset construction module is specifically used for: A large amount of historical piano music and historical key images are obtained, and each piece of historical piano music is sequentially subjected to format conversion, reuse, silence removal, signal framing, and feature extraction. Each piece of historical piano music is then transcribed and annotated to complete preprocessing, and a music dataset is constructed based on each piece of preprocessed historical piano music. Each piece of historical key image is sequentially subjected to format conversion, size adjustment, grayscale conversion, and noise reduction. Each piece of historical key image is then preprocessed by annotating the key position of each piece of historical key image. An image dataset is constructed based on each piece of preprocessed historical key image.

10. The piano playing robot control system according to claim 6, wherein: The model training module is specifically used for: Dividing the music data set into a first training set, a first validation set, and a first test set according to a preset first ratio, training the piano music transcription model using the first training set until a loss value of the transcription loss function is less than a preset first loss threshold, verifying the trained piano music transcription model by calculating the transcription accuracy using the first validation set, and testing the piano music transcription model that has passed the verification by calculating the confidence score using the first test set, and terminating the training if the test passes; The image data set is divided into a second training set, a second verification set and a second test set according to a preset second ratio. The key positioning model is trained with the second training set until the loss value of the positioning loss function is less than a preset second loss threshold. The positioning accuracy is calculated with the second verification set to verify the trained key positioning model. The confidence is calculated with the second test set to test the verified key positioning model. The training ends if the test passes.

Images