A Machine Learning-Based Real-Time Piano Timbre Simulation Method and System

Abstract

This invention discloses a real-time piano timbre simulation method and system based on machine learning, relating to the field of audio signal processing technology. The method includes: data acquisition and multi-dimensional annotation, acquiring multiple types of piano audio, covering techniques and seven dynamic levels, and simultaneously acquiring information such as key presses and techniques; audio preprocessing, including pre-emphasis compensation for high frequencies, Hanning window framing, Fourier transform to frequency domain, spectral subtraction for noise reduction and normalization; multi-dimensional feature extraction, extracting static features such as MFCC and spectral parameters, dynamic features such as first- and second-order differences, and overtone structures; two-stage model training, using stacked autoencoders for dimensionality reduction; real-time parsing, filtering and converting acquired performance data into parameter sequences; timbre synthesis, where the model generates a spectrum and performs an inverse Fourier transform into a waveform; and dynamic optimization, receiving user feedback. This invention solves the problems of traditional simulation methods; the two-stage model enhances timbre coherence, dynamic control achieves low latency, and multi-scenario adaptation and feedback optimization meet specific needs.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to a method and system for real-time simulation of piano timbre based on machine learning. Background Technology

[0002] Piano tone, with its rich overtone structure and dynamic expressiveness, has become an indispensable carrier of timbre in music composition and performance. Different piano types (grand piano, harpsichord, etc.), playing techniques (legato, staccato, etc.), and dynamic levels all create unique timbre characteristics. With the development of digital music and smart musical instruments, the demand for real-time piano tone simulation technology is becoming increasingly urgent. It not only needs to reproduce the timbre details of a real piano, but also needs to meet the requirements of low latency and dynamic response in real-time performance, adapting to diverse scenarios such as concert halls and home performances.

[0003] Traditional piano tone simulation techniques are mainly divided into two categories: sampling synthesis and physical modeling, both of which have significant limitations. Sampling synthesis technology involves pre-recording a large number of piano samples and calling the corresponding samples based on the performance parameters during playback. However, it relies on massive storage resources, and gaps easily appear at the sample splicing points, making it difficult to reproduce the smooth transition of tone under techniques such as legato and tremolo. At the same time, the sampled data is fixed and cannot flexibly adapt to dynamic changes in different dynamics and scenarios. The loss of detail in high-frequency overtones also leads to insufficient tonal layering. Physical modeling technology constructs vibration equations based on the acoustic principles of the piano. Although it can achieve parametric adjustments, the modeling process requires accurately depicting the resonance characteristics of components such as strings and soundboard, resulting in extremely high computational complexity. Real-time performance is difficult to guarantee, and its dynamic adaptability to performance techniques is poor, often resulting in a stiff simulated tone.

[0004] In recent years, machine learning technology has begun to be applied to timbre simulation, but existing solutions still have many shortcomings. Feature extraction is mostly limited to static time-frequency features, ignoring the temporal correlation of dynamic features during performance. The processing of overtone structures does not consider the nonlinear effects of dynamics and piano shape, resulting in insufficient accuracy of feature representation. Model training often uses a single network structure, lacking effective dimensionality reduction and core information enhancement for high-dimensional features, making it prone to overfitting. Furthermore, the lack of temporal constraints on performance parameters results in poor consistency of generated timbre. Real-time control often uses fixed frame length processing, without dynamic adjustment based on performance rhythm, leading to large latency fluctuations. Simultaneously, the lack of user feedback loops and scene adaptation mechanisms makes it impossible to optimize timbre output according to user preferences and usage environment, failing to meet the needs of professional performances and diverse scenarios. These problems collectively make it difficult for existing technologies to achieve "high fidelity, low latency, and strong adaptability" real-time simulation of piano timbre, necessitating the development of an optimized solution integrating deep learning and audio processing technologies. Summary of the Invention

[0005] The present invention proposes a real-time piano timbre simulation method and system based on machine learning to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a real-time piano timbre simulation method based on machine learning, comprising the following steps:

[0007] Data acquisition and multi-dimensional annotation: Collect raw audio data of grand pianos, upright pianos, electric pianos and harpsichords, and simultaneously collect performance parameters including key speed, key duration, pedal depth, etc. Annotate the audio data with piano model, playing technique, dynamic level, overtone characteristics and performance scene.

[0008] Audio signal preprocessing involves pre-emphasis processing of the labeled audio data, using Hanning window framing with a frame length covering two pitch cycles, and converting it into a frequency domain signal through fast Fourier transform.

[0009] Multidimensional timbre feature extraction extracts static and dynamic features from the preprocessed signal. Static features include Mel frequency cepstral coefficients, spectral centroid, spectral bandwidth, spectral flatness, overtone structure, and fundamental frequency. Dynamic features include the first-order difference, second-order difference, and feature similarity between adjacent frames of the feature parameters.

[0010] The machine learning model is trained in two stages. In the first stage, a stacked autoencoder is used to reduce the dimensionality of the feature vector set and encode the features. In the second stage, a timbre generation model combining a generative adversarial network and a long short-term memory network is constructed and trained using the Adam optimizer. The parameters are optimized by minimizing the adversarial loss function.

[0011] Real-time performance data analysis: Real-time performance data is collected through sensor modules, including key trigger signals, velocity values, key duration, pedal opening and closing status and pedal depth changes. Discrete signals are smoothed and filtered, and the technical characteristics and velocity levels of the current performance are extracted simultaneously.

[0012] Real-time timbre synthesis and modulation involves inputting the performance parameter sequence into a trained model, combining the overtone structure parameters with the velocity mapping relationship, converting it into a time-domain waveform through inverse Fourier transform, and using waveform splicing technology and parameter modulation algorithm to optimize the smoothness of the transition.

[0013] Dynamic optimization and feedback iteration are implemented, receiving subjective ratings and correction suggestions from users for the synthesized timbre. The rating dimensions include similarity, naturalness, coherence, and hierarchy. The rating data is correlated with feature parameters, and an incremental learning algorithm is used to update the model parameter weights.

[0014] Furthermore, it also includes adaptive adjustment of feature weights in the multi-dimensional timbre feature extraction step, determining the weight coefficients by calculating the cumulative time impact of dynamic features, specifically through... Implementation, where wk is the weight coefficient of the k-th dynamic feature, and T is the duration of the audio signal. (t) represents the value of the k-th dynamic feature at time t, and α is the attenuation coefficient. Under legato technique, the value is 0.1 to 0.2 to preserve the influence of long-term features, and under staccato technique, the value is 0.4 to 0.5 to highlight the contribution of immediate features. At the same time, a sliding window mechanism is introduced to highlight the influence of recent dynamic features on the current timbre and solve the problem that the difference in dynamic feature contribution under different techniques is ignored.

[0015] Furthermore, it also includes optimizing the pre-emphasis processing flow in the audio signal preprocessing step, using a first-order high-pass filter to achieve pre-emphasis, adding a real-time fundamental frequency detection unit, locating the fundamental frequency period through autocorrelation, and introducing an additional high-frequency gain compensation factor to address the high-frequency overtone attenuation characteristics of grand pianos and harpsichords.

[0016] Furthermore, in the two-stage machine learning model training step, the hidden layer of the stacked autoencoder adopts a 3-layer structure. The number of neurons in the first layer is twice the feature vector dimension, the second layer is the feature vector dimension, and the third layer is half the feature vector dimension. The training process adopts a strategy that combines layer-by-layer pre-training with overall fine-tuning. First, the parameters of each layer are pre-trained through unsupervised learning, and then supervised fine-tuning is performed by combining labeled data. The Dropout mechanism is introduced, and L2 regularization is used to constrain the model parameters. An early stopping method is added to control the number of training rounds. The feature reconstruction error on the validation set is used as the indicator. At the same time, an overtone feature enhancement layer is added at the output of the autoencoder.

[0017] Furthermore, low-latency control is implemented in the real-time timbre synthesis and modulation steps. Latency optimization is achieved by dynamically adjusting the synthesis frame length and processing priority. The latency control employs... The delay adjustment rate is calculated, where dl / dt is the delay change rate, β is the adjustment coefficient ranging from 0.3 to 0.7, and l(t) is the actual delay at time t. By monitoring the performance rhythm characteristics in real time and introducing a preloading mechanism, the subsequent feature trend is predicted based on the performance parameters of the first 3 frames. Combined with the inverse Fourier transform accelerated by FPGA hardware, the processing time of a single frame is compressed to less than 5ms. This operation stabilizes the synthesized delay within the perceptually delay-free range.

[0018] Furthermore, in the multi-dimensional timbre feature extraction step, the overtone structure feature extraction adopts a harmonic superposition model. The theoretical frequencies of each order of overtones are calculated by octet of the fundamental frequency. The peak detection algorithm is used to locate the actual overtone peaks. The overtone amplitude ratio, overtone frequency deviation, and overtone decay time constant are extracted. The overtone amplitude ratio is the ratio of the amplitude of each order of overtone to the amplitude of the fundamental frequency. The overtone decay time constant is obtained by exponential fitting. A detailed mapping relationship is established according to seven dynamic levels from ppp to fff. For missing overtones, an interpolation algorithm between adjacent overtones is used for compensation. At the same time, combined with the characteristics of the piano resonator, the frequency shift of high-frequency overtones is corrected. The influence of resonance on overtones is simulated by a finite element model, and the correction factor is incorporated into the overtone frequency calculation.

[0019] Furthermore, in the dynamic optimization and feedback iteration steps, a quantitative evaluation of timbre similarity is introduced, and similarity calculation is performed through... Achieve, where S is the similarity between the synthesized timbre and the target timbre. For the i-th feature parameter value of the synthesized timbre, Let be the value of the i-th feature parameter of the target timbre, and n be the total number of feature parameters. A weighted fusion strategy of static feature similarity and dynamic feature similarity is adopted. A similarity and rating mapping model is established by combining the subjective ratings of users from 1 to 5. The model weights are updated collaboratively based on the similarity results and user ratings. For feature combinations with low similarity and low ratings, their weight coefficients in the loss function are increased. At the same time, a user preference feature library is constructed.

[0020] A system for real-time simulation of piano timbre based on machine learning includes the following modules:

[0021] Data acquisition and annotation module: Equipped with a high-fidelity microphone array and a 24-bit ADC converter, the microphone array features cardioid polarity and spacing, and is synchronously connected to piano keyboard and pedal position sensors to collect key pressure and pedal depth data. It is also equipped with a touch-screen annotation terminal.

[0022] Audio preprocessing module: integrates digital signal processing unit, including pre-emphasis filter, frame segmenter, windower, Fourier transform and noise suppressor. The pre-emphasis filter is a first-order high-pass filter, the frame segmenter supports variable frame length, the windower has built-in Hanning window, Hamming window and Blackman window options, and the noise suppressor adopts a combination algorithm of spectral subtraction and Wiener filtering.

[0023] Feature extraction module: It consists of time-frequency feature analysis, overtone feature parsing and dynamic feature calculation unit. The time-frequency feature analysis unit calculates MFCC and its difference parameters. The overtone feature parsing unit locates the overtone peak through harmonic decomposition. The dynamic feature calculation unit outputs the first and second order differences of features of adjacent frames and similarity parameters.

[0024] Model training module: Equipped with a GPU acceleration unit, it supports multi-GPU parallel computing, has built-in stacked autoencoder, generative adversarial network and long short-term memory network algorithm libraries, integrates Adam and SGD optimizers, and supports model version management and incremental training;

[0025] Real-time analysis module: It adopts FPGA high-speed processing unit, integrates signal filtering and parameter conversion unit, performs debouncing and smoothing processing on key and pedal signals, converts discrete signals into continuous performance parameter sequence, and synchronously outputs technique recognition results;

[0026] The timbre synthesis module consists of a spectrum generation, waveform conversion, and parameter modulation unit. The spectrum generation unit receives the characteristic parameters output by the model, the waveform conversion unit generates a time-domain waveform through inverse Fourier transform, and the parameter modulation unit adjusts the delay and attenuation characteristics according to the pedal parameters. It also has a built-in waveform splicing optimization algorithm.

[0027] Optimized feedback module: Equipped with a touch-screen interactive interface, it supports users to rate the synthesized timbre from 1 to 5 points, receive text correction suggestions, and includes a rating data processing and model update triggering unit. It associates and stores the rating with feature parameters and triggers the incremental training process.

[0028] Storage module: Uses an SSD array to store raw audio data, feature vector sets and trained model parameters, supports data classification indexing, and is equipped with a data backup unit.

[0029] Furthermore, it also includes a low-latency transmission module, which adopts a PCIe 4.0 high-speed bus, supports burst data transmission, integrates a data verification unit and a timing synchronization unit, reduces transmission latency through a cache pre-read mechanism, and uses a circular cache strategy to avoid data overflow. The data verification unit adopts the CRC32 verification algorithm, and the timing synchronization unit realizes cross-module clock calibration based on GPS timing signals. The performance parameter signal is set to the highest priority, the feature data is set to the medium priority, and the log data is set to the low priority.

[0030] Furthermore, it also includes a multi-scene adaptation module, which contains a scene parameter configuration unit and a timbre template library. The scene parameter configuration unit supports parameter presets for three scenarios: concert hall, home performance, and recording studio. The timbre template library stores the baseline timbre characteristics of different piano models. Users can customize scene parameters and save them as new templates. The module adjusts the synthesis strategy by calling the feature parameters of the corresponding template to adapt to the timbre requirements of different usage scenarios. It also adds seamless transition technology for scene switching, using a cross-fade algorithm to achieve smooth switching of scene parameters. In addition, it integrates an environmental acoustic detection unit, which collects current environmental noise and reverberation data through a built-in microphone and automatically recommends suitable scene templates.

[0031] Compared with existing technologies, the beneficial effects of this invention are:

[0032] At the feature representation level, this invention innovatively constructs a multi-dimensional timbre feature system that takes into account both static time-frequency features and dynamic temporal features. Through a dynamic weight adaptive adjustment mechanism, it accurately matches the differences in feature contributions under different playing techniques, solving the problem of traditional methods ignoring the influence of dynamic feature temporal sequence. The overtone structure extraction, combined with velocity mapping and resonance correction, fully restores the overtone details of a real piano, avoiding the loss of high-frequency features and nonlinear deviations. This makes the feature representation more closely resemble the acoustic essence of real timbre, laying a precise feature foundation for high-quality simulation.

[0033] In terms of model performance, the two-stage machine learning model achieves effective dimensionality reduction of high-dimensional features and enhancement of core information through stacked autoencoders. Combined with layer-by-layer pre-training and multiple anti-overfitting strategies, it improves the model's generalization ability and feature learning depth. The fusion architecture of generative adversarial networks and long short-term memory networks introduces temporal constraints on performance parameters, which significantly improves the coherence and naturalness of synthesized timbres. It solves the problems of stiff timbres and discontinuous transitions in traditional models, making the simulated timbres more closely resemble real performances when switching techniques.

[0034] In terms of real-time performance and adaptability, the dynamic latency control mechanism, combined with hardware acceleration and preloading strategies, achieves perceptible latency-free timbre output, completely resolving the lag issue in real-time performance. The multi-scene adaptation module, through scene parameter presets and automatic recommendations, quickly matches the acoustic requirements of different usage environments. Seamless transition technology avoids abrupt timbre changes during scene switching, enhancing the system's practical flexibility. The user feedback closed-loop and preference learning mechanism combines subjective ratings with quantitative similarity to specifically optimize model output, making the timbre more closely match user needs and breaking through the rigid limitations of traditional "one-size-fits-all" technologies.

[0035] At the system support level, the low-latency transmission module ensures the synchronous transmission of performance data and characteristic signals through a high-speed bus and priority scheduling, avoiding latency fluctuations caused by data congestion. The entire signal optimization process, from dynamic pre-emphasis in preprocessing to waveform modulation in the synthesis stage, forms a complete technical chain, ensuring that every step from data acquisition to timbre output maximizes the preservation of timbre details and dynamic characteristics. Overall, this invention provides a professional and practical solution for real-time piano timbre simulation, widely adaptable to various scenarios such as professional performance, music composition, and intelligent musical instruments. Attached Figure Description

[0036] Figure 1 This is a schematic block diagram of the real-time piano timbre simulation method based on machine learning proposed in this invention;

[0037] Figure 2 This is a schematic block diagram of the system for the real-time simulation method of piano timbre based on machine learning proposed in this invention.

[0038] Figure 3 This is a graph showing the evolution of the training effect of the two-stage model.

[0039] Figure 4 Comparison chart of real-time performance delay control effects;

[0040] Figure 5 Radar chart adaptable to multiple scenarios;

[0041] Figure 6 A trend chart showing the closed-loop optimization effect based on user feedback. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0044] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0045] Reference Figures 1 to 6A method for real-time simulation of piano timbre based on machine learning, comprising the following steps:

[0046] Data acquisition and multidimensional annotation: Collect raw audio data of different types of pianos, including grand pianos, upright pianos, electric pianos and harpsichords. Playing techniques include legato, staccato, glissando, and vibrato. Playing dynamics are divided into seven levels from ppp to fff. Playing parameters such as key speed, key duration, and pedal depth are collected simultaneously. The collected audio data is annotated, and the annotation content includes piano model, playing technique, dynamic level, overtone characteristics, and playing scenario.

[0047] Audio signal preprocessing involves pre-emphasis on the labeled audio data to compensate for high-frequency signal attenuation, framing using a Hanning window with a frame length covering two pitch cycles and an overlap rate of 50% between adjacent frames, conversion of the time-domain signal to the frequency-domain signal using fast Fourier transform, removal of environmental noise using spectral subtraction, and normalization of the signal to unify the signal amplitude range.

[0048] Multidimensional timbre feature extraction extracts static and dynamic features from the preprocessed signal. Static features include Mel frequency, cepstral coefficients, spectrum, centroid, spectrum, bandwidth, spectrum flatness, overtone structure, and fundamental frequency. Dynamic features include first-order and second-order differences of feature parameters and feature similarity between adjacent frames. Overtone structure features are extracted by harmonic decomposition to extract the amplitude frequency and attenuation rate of each overtone. A timbre feature vector set with a dimension of no less than 30 is constructed.

[0049] The machine learning model training is carried out in two stages. In the first stage, a stacked autoencoder is used to reduce the dimensionality and encode the feature vector set, retaining the core features with a cumulative variance contribution rate of not less than 95%, and outputting low-dimensional feature codes. In the second stage, a timbre generation model combining generative adversarial network and long short-term memory network is constructed. The low-dimensional feature codes and performance parameters are used as inputs, and the spectrogram of the original labeled audio is used as output. The model is trained using the Adam optimizer. The model parameters are optimized by minimizing the adversarial loss function between the generator and the discriminator. Temporal dependency constraints of performance parameters are introduced to improve the coherence of generation.

[0050] Real-time performance data analysis involves collecting real-time performance data through a sensor module, including key trigger signal strength value, key duration, pedal opening and closing state, and pedal depth changes. The collected discrete signals are smoothed and filtered to convert them into a performance parameter sequence consistent with the training data format, and the technical characteristics and dynamic level of the current performance are extracted simultaneously.

[0051] Real-time timbre synthesis and modulation involves inputting the analyzed performance parameter sequence into a trained timbre generation model to generate corresponding spectral features. Combining the overtone structure parameters and velocity mapping relationship, the frequency domain features are converted into time domain waveforms through inverse Fourier transform. Waveform splicing technology and parameter modulation algorithms are used to optimize the waveform transition smoothness. The timbre sustain characteristics are adjusted according to the pedal parameters to achieve low-latency timbre simulation output.

[0052] Dynamic optimization and feedback iteration receive subjective ratings and correction suggestions from users for synthesized timbres. Rating dimensions include similarity, naturalness, coherence, and hierarchy. The rating data is associated with corresponding feature parameters, and incremental learning algorithms are used to update the parameter weights of the timbre generation model, enhancing the generation capability of high-scoring feature combinations and forming a closed-loop process of data collection, model training, simulated output, and optimization iteration.

[0053] This invention also includes adaptive adjustment of feature weights in the multi-dimensional timbre feature extraction step, determining the weight coefficients by calculating the cumulative time impact of dynamic features, specifically through... Implementation, where wk is the weight coefficient of the k-th dynamic feature, and T is the duration of the audio signal. (t) represents the value of the k-th dynamic feature at time t, and α is the attenuation coefficient, ranging from 0.1 to 0.5. The attenuation coefficient α is dynamically adapted according to the playing technique. Under legato technique, the value is 0.1 to 0.2 to retain the influence of long-term features, and under staccato technique, the value is 0.4 to 0.5 to highlight the contribution of immediate features. At the same time, a sliding window mechanism is introduced, with the window length set to 5 to 10 frames, and the weight coefficient is updated in real time. This method highlights the influence of recent dynamic features on the current timbre, improves the accuracy of feature expression, and solves the problem that the difference in dynamic feature contribution under different techniques is ignored.

[0054] This invention also includes optimizing the pre-emphasis processing flow in the audio signal preprocessing step. A first-order high-pass filter is used for pre-emphasis, and the filter cutoff frequency is dynamically adjusted according to the piano's fundamental frequency. A cutoff frequency of 50Hz is set for the bass range (80Hz to 200Hz), 80Hz for the midrange (200Hz to 500Hz), and 120Hz for the treble range (500Hz to 4000Hz). A real-time fundamental frequency detection unit is added, and the fundamental frequency period is located using the autocorrelation method. The cutoff frequency parameter is updated once per frame of signal. For the high-frequency overtone attenuation characteristics of grand pianos and harpsichords, an additional high-frequency gain compensation factor is introduced. The compensation coefficient increases linearly from 1.0 to 1.5 as the frequency increases. Pre-emphasis enhances the extraction effect of high-frequency overtone components, avoids the loss of high-frequency features in different ranges and piano types, and improves the completeness of subsequent feature extraction.

[0055] In this invention, during the two-stage machine learning model training step, the hidden layer of the stacked autoencoder adopts a three-layer structure. The number of neurons in the first layer is twice the dimension of the feature vector, the second layer is the same as the dimension of the feature vector, and the third layer is half the dimension of the feature vector. The training process adopts a strategy combining layer-by-layer pre-training and overall fine-tuning. First, the parameters of each layer are pre-trained through unsupervised learning, and then supervised fine-tuning is performed using labeled data. A Dropout mechanism is introduced with a probability of 0.2. At the same time, L2 regularization is used to constrain the model parameters with a regularization coefficient of 0.001. An early stopping method is added to control the number of training rounds. The feature reconstruction error on the validation set is used as an indicator. If the error does not decrease for 10 consecutive rounds, training is stopped to effectively avoid model overfitting. At the same time, an overtone feature enhancement layer is added at the output of the autoencoder to perform secondary encoding of core features such as overtone amplitude ratio and decay rate, thereby improving the ability of low-dimensional features to represent key timbre information.

[0056] In this invention, low-latency control is performed in the real-time timbre synthesis and modulation steps. Latency optimization is achieved by dynamically adjusting the synthesis frame length and processing priority. The latency control employs… The delay adjustment rate is calculated, where dl / dt is the delay change rate, β is the adjustment coefficient ranging from 0.3 to 0.7, l0 is the target delay threshold set to within 20ms, and l(t) is the actual delay at time t. By monitoring the performance rhythm characteristics in real time, β is set to 0.6 to 0.7 for fast-paced performances to accelerate delay adjustment, and β is set to 0.3 to 0.4 for slow-paced performances to maintain synthesis stability. At the same time, a preloading mechanism is introduced to predict the subsequent feature trends based on the performance parameters of the first 3 frames, and generate a semi-synthesized spectrum in advance. Combined with the inverse Fourier transform accelerated by FPGA hardware, the processing time of a single frame is compressed to within 5ms. This operation stabilizes the synthesis delay within the perceptible delay-free range, solving the problem of timbre output lag in real-time performance.

[0057] In this invention, the harmonic structure feature extraction step employs a harmonic superposition model. The theoretical frequencies of each harmonic are calculated by multiplying the fundamental frequency. A peak detection algorithm is used to locate the actual harmonic peaks. The harmonic amplitude ratio, harmonic frequency deviation, and harmonic decay time constant are extracted. The harmonic amplitude ratio is the ratio of the amplitude of each harmonic to the fundamental frequency amplitude. The harmonic decay time constant is obtained through exponential fitting. Addressing the nonlinear characteristics of harmonic amplitude being affected by dynamics, a dynamics-harmonic amplitude mapping table is introduced, establishing a detailed mapping relationship according to seven dynamic levels from ppp to fff. Missing harmonics are compensated using an interpolation algorithm between adjacent harmonics. Simultaneously, considering the characteristics of the piano resonator, the frequency shift of high-frequency harmonics is corrected. The influence of resonance on harmonics is simulated using a finite element model, and the correction factor is incorporated into the harmonic frequency calculation, improving the matching degree between harmonic features and the true piano timbre.

[0058] In this invention, a quantitative evaluation of timbre similarity is introduced in the dynamic optimization and feedback iteration steps. The similarity calculation is performed through... The goal is to achieve this, where S represents the similarity between the synthesized timbre and the target timbre, with a value ranging from 0 to 1. For the i-th feature parameter value of the synthesized timbre, Let n be the value of the i-th feature parameter of the target timbre, and n be the total number of feature parameters. A weighted fusion strategy of static feature similarity and dynamic feature similarity is adopted, with the static feature weight set to 0.4 and the dynamic feature weight set to 0.6. A similarity-rating mapping model is established by combining the subjective ratings of users from 1 to 5 points. A rating of 4 to 5 points corresponds to S≥0.9, a rating of 2 to 3 points corresponds to 0.6≤S<0.9, and a rating of 1 point corresponds to S<0.6. The model weights are updated collaboratively based on the similarity results and user ratings. For feature combinations with low similarity and low ratings, their weight coefficients in the loss function are increased. At the same time, a user preference feature library is constructed to record the preferences of different users for parameters such as overtone richness and duration of sustain. The model output is optimized in a targeted manner to improve the user adaptability of the synthesized timbre.

[0059] A system for real-time simulation of piano timbre based on machine learning includes the following modules:

[0060] The data acquisition and annotation module is equipped with a high-fidelity microphone array and a 24-bit ADC converter. The sampling rate is set to 48kHz. The microphone array adopts a cardioid polar pattern with a spacing of 10cm. It is synchronously connected to the piano keyboard sensor and the pedal position sensor. The accuracy of key force acquisition is 1024 levels, the pedal depth resolution is 8 bits, and it is equipped with a touch-screen annotation terminal that supports manual supplementation and correction of automatic annotation results.

[0061] The audio preprocessing module integrates a digital signal processing unit, including a pre-emphasis filter, a frame divider, a windower, a Fourier transform, and a noise suppressor. The pre-emphasis filter is a first-order high-pass filter, the frame divider supports variable frame length settings, the windower has built-in Hanning window, Hamming window, and Blackman window options, the Fourier transform accuracy is 1024 points, and the noise suppressor adopts an algorithm combining spectral subtraction and Wiener filtering.

[0062] The feature extraction module consists of a time-frequency feature analysis unit, an overtone feature parsing unit, and a dynamic feature calculation unit. The time-frequency feature analysis unit calculates the MFCC and its difference parameters, and extracts 13-dimensional MFCC plus 26-dimensional difference parameters. The overtone feature parsing unit locates the overtone peak through harmonic decomposition. The dynamic feature calculation unit outputs the first and second order differences of features of adjacent frames and similarity parameters.

[0063] The model training module is equipped with a GPU acceleration unit, supports multi-GPU parallel computing, has built-in algorithm libraries for stacked autoencoder generative adversarial networks and long short-term memory networks, integrates two optimizers, AdamSGD, includes a loss function calculation unit and a model parameter storage unit, and supports model version management and incremental training.

[0064] The real-time analysis module uses a high-speed FPGA processing unit with a response latency of less than 1ms. It integrates a signal filtering unit and a parameter conversion unit to perform de-jittering and smoothing processing on key and pedal signals, converting discrete signals into a continuous sequence of performance parameters and synchronously outputting the technique recognition results.

[0065] The timbre synthesis module consists of a spectrum generation unit, a waveform conversion unit, and a parameter modulation unit. The spectrum generation unit receives the feature parameters output by the model, the waveform conversion unit generates a time-domain waveform through inverse Fourier transform, and the parameter modulation unit adjusts the delay and attenuation characteristics according to the pedal parameters. It also has a built-in waveform splicing optimization algorithm.

[0066] The feedback module is optimized and equipped with a touch-screen interactive interface, which allows users to rate the synthesized timbre from 1 to 5 points and receive correction suggestions in text form. It includes a rating data processing unit and a model update triggering unit, which associates and stores the rating with feature parameters and triggers the incremental training process.

[0067] The storage module uses an SSD array to store the original audio data feature vector set and the trained model parameters. It supports data classification indexing and is equipped with a data backup unit to ensure data security and traceability.

[0068] This invention also includes a low-latency transmission module. This module uses a PCIe 4.0 high-speed bus, is equipped with a 256MB high-speed cache, supports burst data transmission, integrates a data verification unit and a timing synchronization unit, reduces transmission latency through a cache pre-read mechanism with a pre-read depth of 8 frames of data, and employs a circular caching strategy to avoid data overflow. The data verification unit uses the CRC32 check algorithm to detect the integrity of transmitted data in real time. The timing synchronization unit uses GPS timing signals to achieve cross-module clock calibration, with synchronization errors controlled within 1μs, ensuring signal synchronization between the real-time parsing module and the timbre synthesis module, and controlling transmission latency within 50μs. At the same time, differentiated transmission priorities are adopted for different data types: performance parameter signals are set to the highest priority, feature data to the medium priority, and log data to the low priority. Priority scheduling avoids data congestion and further optimizes transmission latency stability.

[0069] This invention also includes a multi-scene adaptation module, which comprises a scene parameter configuration unit and a timbre template library. The scene parameter configuration unit supports parameter presets for three scenarios: concert hall, home performance, and recording studio. The concert hall scenario is configured with a reverberation time of 1.5 to 2.0 seconds and a sound field diffusion of 0.8 to 0.9 seconds; the home performance scenario is configured with a reverberation time of 0.3 to 0.5 seconds and a sound field diffusion of 0.4 to 0.5 seconds; and the recording studio scenario is configured with a reverberation time of 0.1 to 0.2 seconds and a sound field diffusion of 0.2 to 0.3 seconds. The timbre template library stores the baseline timbre characteristics of different piano models, covering more than 20 common pianos such as Steinway grand pianos and Yamaha upright pianos. Each template contains core parameters such as fundamental frequency range, overtone order, and attenuation characteristics. Users can customize scene parameters and save them as new templates. The module adjusts the synthesis strategy by calling the feature parameters of the corresponding template to adapt to the timbre requirements of different usage scenarios. It also adds seamless transition technology for scene switching, using a cross-fade algorithm to achieve smooth switching of scene parameters, with a transition time set to 100ms to 200ms to avoid abrupt changes in timbre during scene switching. In addition, it integrates an environmental acoustic detection unit, which collects current environmental noise and reverberation data through a built-in microphone and automatically recommends suitable scene templates to improve the system's intelligent adaptation capabilities.

[0070] The following two examples further illustrate the specific implementation of this system:

[0071] Example 1: Real-time simulation of a concert hall scene using a Steinway grand piano

[0072] This embodiment is designed for professional concert hall performances and focuses on the timbre simulation of the Steinway D-274 grand piano. It is essential to ensure the accuracy of overtone reproduction, the continuity of technique, and low latency response. The specific implementation is as follows.

[0073] 1. Detailed Implementation of Technical Solutions

[0074] Data Acquisition and Multidimensional Annotation: A three-array Neumann U87 high-fidelity microphone (cardioid polar pattern, 10cm spacing) was used with a 24-bit ADC converter (48kHz sampling rate) to acquire audio data from the Steinway D-274 in a concert hall environment. A Roland TP-100 keyboard sensor (1024 levels of key velocity accuracy) and a Korg DS-1H pedal sensor (8-bit depth resolution) were simultaneously connected to acquire parameters such as key speed (0.5-10m / s), key duration (10ms-5s), and pedal depth (0-255). Performance was covered including legato (C4-E4-G4 legato), staccato (G5 short note), staccato (A3 staccato), glissando (C4 to E4), and tremolo (F4 tremolo), with dynamics evenly distributed from ppp (20 levels) to fff (1000 levels). The labeling was done using LabelStudio, which automatically labeled the piano model "Steinway D-274", technique type, and dynamic level. The harmonic characteristics (amplitude ratio of the first 8 harmonics) and the scene "concert hall (reverberation 1.8s)" were manually corrected, resulting in 100,000 labeled samples.

[0075] Audio signal preprocessing: Pre-emphasis uses a first-order high-pass filter, and the fundamental frequency is detected in real time using the autocorrelation method. The cutoff frequency is updated every 10ms frame—50Hz for the low range (80-200Hz, e.g., C265.4Hz), 80Hz for the mid-range (200-500Hz, e.g., C4261.6Hz), and 120Hz for the high range (500-4000Hz, e.g., C61046.5Hz). To address the rapid decay of high-frequency overtones in grand pianos, a high-frequency gain compensation factor is introduced: 1.0 below 1kHz, and linearly increased to 1.5 above 4kHz. Framing uses a Hanning window with a frame length of 20ms (covering two fundamental frequency cycles) and 50% overlap. A 1024-point FFT is used to transform the signal to the frequency domain, and spectral subtraction is used to remove environmental noise (threshold set to 0.05 times the signal amplitude), normalizing the signal to the [-1,1] interval.

[0076] Multidimensional timbre feature extraction: Static features include 13-dimensional MFCC, spectral centroid (reflecting timbre brightness), spectral bandwidth (overtone distribution range), spectral flatness (timbre richness), fundamental frequency (extracted by autocorrelation method), and overtone structure; dynamic features include first and second order MFCC differences (26 dimensions in total) and adjacent frame feature similarity (cosine similarity calculation). Overtone extraction uses a harmonic superposition model, obtaining theoretical overtone frequencies by doubling the fundamental frequency, and a peak detection algorithm (threshold 0.02 times the fundamental amplitude) to locate the actual peak, extracting overtone amplitude ratios (each overtone / fundamental amplitude), frequency deviation (actual - theoretical frequency), and decay time constant. To address the nonlinear effects of dynamics, a 7-level dynamics-overtone mapping table is established (ppp corresponds to a 0.3 decrease in overtone amplitude ratio, and fff an increase of 0.2), with missing overtones compensated by linear interpolation. Feature weights are calculated using the formula... The calculations were performed with T=2s and α=0.15 for legato techniques and T=0.3s and α=0.45 for staccato techniques. The sliding window was set to 8 frames, and the weights were updated in real time to finally construct a 35-dimensional feature vector set.

[0077] Two-stage model training: A stacked autoencoder input with 35-dimensional features and a 70-35-17 neuron hidden layer structure were used for layer-by-layer pre-training (unsupervised reconstruction error of the first layer <0.01), followed by fine-tuning with labeled data. Dropout 0.2 and L2 regularization 0.001 were applied. Early stopping was used to monitor the reconstruction error on the validation set, stopping after 10 consecutive rounds without any descent. In the generative adversarial network (GAN), the generator used a 3-layer LSTM (17-dimensional input + 4-dimensional performance parameters), outputting a 1024-point spectrogram; the discriminator used a 2-layer CNN (3×3 convolutional kernels), with an Adam optimizer (learning rate 0.0002). The loss function was adversarial loss + 0.3 times the temporal constraint loss (spectral similarity between adjacent frames >0.95). An NVIDIA RTX 3090 GPU was used, and multi-GPU parallel training was performed for 200 rounds until the model converged.

[0078] Real-time parsing and synthesis: Real-time parsing uses a Xilinx Zynq FPGA with a response latency of <1ms. Key signals are debounced (5ms threshold), smoothed, and filtered (moving average window of 3 frames) to convert them into a sequence of performance parameters. Technique recognition uses an SVM model (98% accuracy). Synthesis latency is calculated using a formula. Controls are applied with β=0.65 for fast-paced vibrato and β=0.35 for slow-paced legato, with l0=15ms. Pre-load parameters from the first 3 frames to predict the spectral trend, use FPGA to accelerate IFFT (4ms per frame), use cross fade for waveform splicing (5ms transition), extend the sustain decay time (from 2s to 4s) when pedal depth >128, and match scene parameters of 1.8s reverberation and 0.85 sound field diffusion in a concert hall.

[0079] Optimization Feedback: The touchscreen receives user ratings (1-5 points), and similarity is calculated using a formula. Calculate n=35, static feature weights 0.4, dynamic weights 0.6. After collecting 50 rating data points, trigger incremental training. For samples with S<0.8 and scores <3, increase the loss function weights by 1.5 times and update the model parameters.

[0080] 2. Effectiveness verification data

[0081] Table 1: Comparison of Simulation Performance of Grand Pianos

[0082]

[0083] Table 1 data verifies the advantages of this invention in professional scenarios. Traditional sampling synthesis suffers from low fidelity and coherence due to missing overtone samples and splicing gaps; while physical modeling provides good overtone simulation, its computational complexity leads to high latency. This invention achieves 95% overtone fidelity through dynamic mapping and resonance correction of overtone extraction; the timing constraints of the two-stage model improve the coherence of technique switching to 94%; the dynamic delay formula combined with FPGA acceleration reduces latency to 12ms (perceptually zero latency); and the multi-scenario modules accurately match the acoustic parameters of concert halls, achieving 96% adaptability. High user satisfaction stems from the dual improvement in timbre naturalness and real-time response, confirming the synergistic effectiveness of feature optimization, model architecture, and scenario adaptation, and solving the core requirements of professional performance for high fidelity and low latency.

[0084] Example 2: Real-time simulation of a domestic scene with an 18th-century harpsichord

[0085] This embodiment is designed for home retro performance needs, focusing on the simulation of the timbre of an 18th-century harpsichord (clavichord). It needs to highlight the retro overtone characteristics, scene adaptation flexibility, and user preference adaptation. The specific implementation is as follows.

[0086] 1. Detailed Implementation of Technical Solutions

[0087] Data Acquisition and Multidimensional Annotation: Two sets of Rohde NT1 high-fidelity microphones (cardioid, 8cm spacing) and a 24-bit ADC (48kHz sampling rate) were used to acquire audio from a replica 18th-century harpsichord. A MIDI keyboard sensor (512-level key speed accuracy) and an analog pedal sensor (8-bit depth) were simultaneously connected to acquire key press duration (50ms-3s), pedal open / closed state (0 / 1), and velocity (pp-ff, 50-800 levels). The performance focused on legato (D3-F3-A3) and glissando (G3 to B3). Harpsichords do not have vibrato techniques, so these were not acquired. The model name was labeled "18th-century harpsichord (replica)," overtone characteristics (emphasis on the first four overtones), and the scene was "family living room." LabelStudio automatically annotated the samples, and the overtone attenuation parameters were manually corrected, resulting in 80,000 annotated samples.

[0088] Audio signal preprocessing: Dynamic adjustment of pre-emphasis cutoff frequency—50Hz for bass (100-200Hz), 80Hz for midrange (200-400Hz), and 120Hz for treble (400-3000Hz), with a high-frequency gain compensation factor of 1.2-1.6 (the harpsichord's high frequencies are weaker). Framing using a Hanning window with a frame length of 25ms (adapting to the longer fundamental period of the harpsichord, such as D3146.8Hz with a period of 6.8ms), 50% overlap; 1024-point FFT, spectral subtraction denoising (threshold 0.04 times the signal amplitude), and normalization to [-1, 1].

[0089] Multidimensional timbre feature extraction: Static features include 13-dimensional MFCC, spectral centroid (lower, reflecting a retro, somber timbre), spectral flatness, fundamental frequency, and overtone structure (primarily the first four orders); dynamic features are MFCC difference (26-dimensional) and frame similarity. Overtone extraction uses a harmonic superposition model combined with a finite element model of the harpsichord resonator to correct high-frequency overtone frequency shifts (4th order overtone correction + 2Hz). The velocity-overtone mapping table is simplified to 4 levels (pp-ff), and missing overtones are interpolated using adjacent elements. In the feature weight formula, legato T = 1.5s, α = 0.18; glissando T = 1s, α = 0.2, with a sliding window of 6 frames, constructing a 32-dimensional feature vector set.

[0090] Two-stage model training: A stacked autoencoder with 3 layers (64-32-16 neurons) was pre-trained layer by layer (reconstruction error <0.015), Dropout 0.2, L2 regularization 0.001, and early stopping for 10 epochs without descent. An overtone enhancement layer was added to the output (emphasizing the encoding of the first 4 overtones). The generative adversarial network (GAN) generator's LSTM temporal window was increased to 10 frames (adapting to the long sustain of the harpsichord), the discriminator was a CNN (5×5 convolutional kernels), and an Adam optimizer (learning rate 0.0003) was used. The loss function included a 0.4x overtone constraint loss (amplitude ratio error of the first 4 overtones <5%). An NVIDIA RTX 4070 GPU was used, and training converged after 180 epochs.

[0091] Real-time analysis and synthesis: FPGA is the same as in Example 1, with analysis latency <1ms, key signal debouncing 8ms, and outputting technique recognition results after parameter conversion (legato / glissando accuracy 97%). In the synthesis latency formula, β=0.4 (for a smooth home performance rhythm), l0=20ms, preload 2 frames of parameters, IFFT processing 5ms, and waveform splicing transition 8ms. The delay decay time when the pedal is closed is set to 3s, matching the preset parameters of 0.4s reverb and 0.45 sound field diffusion for a home scene, and a cross fade of 150ms is used for scene switching.

[0092] Optimize feedback: Combine touchscreen ratings with text suggestions (e.g., "softer overtones"), similarity formula n=32, static weight 0.3, dynamic weight 0.7 (dynamic transitions are more critical for harpsichords). Build a user preference library to record preference tags such as "retro overtones" and "short sustain," strengthen the corresponding feature weights during incremental training, and update the model every 40 data points collected.

[0093] 2. Effectiveness verification data

[0094] Table 2: Comparison of the simulation performance of the harpsichord

[0095]

[0096] Table 2 data highlights the advantages of this invention in retro scenarios and user adaptation. Traditional sampling synthesis struggles to reproduce the unique first four overtone structures of the harpsichord, resulting in noticeable abrupt transitions between scenes. While physical modeling provides good overtone simulation, it suffers from high resource consumption and poor preference adaptation. This invention achieves 93% retro overtone similarity through overtone enhancement encoding and resonance correction; cross-fade transition technology smooths scene transitions to 95%; collaborative updates of the user preference library and similarity-rating improve adaptability to 92%; spectral subtraction combined with noise suppression algorithms achieves 94% anti-interference capability; and stacked autoencoder dimensionality reduction reduces resource consumption to 65% (lower than traditional methods). Data validates the effectiveness of overtone optimization, seamless scene switching, and preference learning, solving the problems of insufficient retro charm and rigid scene adaptation in harpsichord simulation, thus meeting the needs of home retro performances.

[0097] Reference Figure 3 This figure reveals the training advantages of the two-stage model. Traditional single-network models, due to high-dimensional feature redundancy and a lack of overfitting prevention strategies, still have a loss value of 0.35 after 200 rounds, with an overtone reproduction rate of only 81%. This invention, through stacked autoencoders for dimensionality reduction (preserving 95% of the variance contribution), Dropout, and early stopping to prevent overfitting, reduces the loss value to 0.3 after 50 rounds and converges to below 0.12 after 150 rounds. Simultaneously, the introduction of temporal constraint loss steadily improves the overtone reproduction rate to 95% with each training round. The data demonstrates the synergistic effect of "dimensionality reduction encoding + adversarial generation + temporal constraints," solving the problems of overfitting and poor overtone reproduction in traditional models, laying the foundation for high-quality timbre generation.

[0098] Reference Figure 4 This figure verifies the effectiveness of the real-time latency control mechanism. Traditional fixed frame length processing results in a latency of 45-50ms during fast-paced staccato and technique switching, far exceeding the perceived latency threshold. This invention adjusts the processing rate through a dynamic latency formula (β adapts to the rhythm: 0.65 for fast rhythms and 0.35 for slow rhythms), combined with FPGA acceleration and preloading strategies. This reduces the latency to 12ms for fast rhythms and only 14ms for complex technique switching, with an average latency of 15ms, consistently below the target threshold. The data confirms that the low-latency solution of "algorithm optimization + hardware acceleration" solves the core pain point of lagging tone output in real-time performance, meeting the immediate needs of professional performances.

[0099] Reference Figure 5This diagram comprehensively demonstrates the advantages of the multi-scene adaptation module. Traditional methods, lacking scene-specific parameter templates, generally achieve reverberation matching accuracy below 82%, and transition smoothness at only 55%-65%. This invention, through scene parameter presets (concert hall reverberation 1.5-2.0s, recording studio 0.1-0.2s) and cross-fade transition technology (100-200ms), achieves reverberation matching accuracy exceeding 95% for all scenes, with transition smoothness improved to 94%-96%. Overtone adaptation, combined with scene acoustic characteristics, adjusts overtone attenuation parameters, resulting in user satisfaction exceeding 92 points. Data validates the effectiveness of the scene template library and automatic recommendation mechanism, solving the problem of "rigid scene adaptation" in traditional technologies.

[0100] Reference Figure 6 This diagram highlights the continuous optimization value of the feedback loop. The initial model, failing to adapt to user preferences, achieved a similarity of 82% and a score of only 7.2. In the first iteration, incorporating the user's suggestion of "softer overtones," the model used a similarity formula (S-weighted fusion of static and dynamic features) to locate low-matching features, reducing the loss value to 0.14. After three iterations, the similarity reached 93%, and the score reached 9.4, indicating the model converged to a stable state. This is because the feedback mechanism correlates subjective scores with quantified similarity, specifically strengthening the weights of high-preference features, thus solving the problem of "undifferentiated output" in traditional models and making the timbre more aligned with user needs.

[0101] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for real-time simulation of piano timbre based on machine learning, characterized in that, Includes the following steps: Data acquisition and multi-dimensional annotation: Collect raw audio data of grand pianos, upright pianos, electric pianos and harpsichords, and simultaneously collect performance parameters including key speed, key duration, pedal depth, etc. Annotate the audio data with piano model, playing technique, dynamic level, overtone characteristics and performance scene. Audio signal preprocessing involves pre-emphasis processing of the labeled audio data, using Hanning window framing with a frame length covering two pitch cycles, and converting it into a frequency domain signal through fast Fourier transform. Multidimensional timbre feature extraction extracts static and dynamic features from the preprocessed signal. Static features include Mel frequency cepstral coefficients, spectral centroid, spectral bandwidth, spectral flatness, overtone structure, and fundamental frequency. Dynamic features include the first-order difference, second-order difference, and feature similarity between adjacent frames of the feature parameters. The two-stage machine learning model training uses a stacked autoencoder to reduce the dimensionality of the feature vector set and encode the features in the first stage. The second stage constructs a timbre generation model that combines a generative adversarial network and a long short-term memory network, and trains it using the Adam optimizer, optimizing the parameters by minimizing the adversarial loss function; Real-time performance data analysis: Real-time performance data is collected through sensor modules, including key trigger signals, velocity values, key duration, pedal opening and closing status and pedal depth changes. Discrete signals are smoothed and filtered, and the technical characteristics and velocity levels of the current performance are extracted simultaneously. Real-time timbre synthesis and modulation involves inputting the performance parameter sequence into a trained model, combining the overtone structure parameters with the velocity mapping relationship, converting it into a time-domain waveform through inverse Fourier transform, and using waveform splicing technology and parameter modulation algorithm to optimize the smoothness of the transition. Dynamic optimization and feedback iteration are implemented, receiving subjective ratings and correction suggestions from users for the synthesized timbre. The rating dimensions include similarity, naturalness, coherence, and hierarchy. The rating data is correlated with feature parameters, and an incremental learning algorithm is used to update the model parameter weights.

2. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, Also included in the multi-dimensional timbre feature extraction step is feature weight self-adaptive adjustment, with weight coefficients determined by calculating the time cumulative influence of dynamic features, specifically by implemented, wherein is the weight coefficient of the kth dynamic feature, T is the time length of the audio signal, is the value of the kth dynamic feature at time t, and a is a decay coefficient, with a value of 0.1 to 0.2 under the legato technique to retain the influence of long-time sequence features, and a value of 0.4 to 0.5 under the staccato technique to highlight the contribution of immediate features, while a sliding window mechanism is introduced to highlight the influence of recent dynamic features on the current timbre through feature weight self-adaptive adjustment.

3. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, It also includes optimizing the pre-emphasis processing flow in the audio signal preprocessing step, using a first-order high-pass filter to achieve pre-emphasis, adding a real-time fundamental frequency detection unit, locating the fundamental frequency period through autocorrelation, and introducing an additional high-frequency gain compensation factor to address the high-frequency overtone attenuation characteristics of grand pianos and harpsichords.

4. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, In the two-stage machine learning model training step, the hidden layer of the stacked autoencoder adopts a 3-layer structure. The number of neurons in the first layer is twice the feature vector dimension, the second layer is the feature vector dimension, and the third layer is half the feature vector dimension. The training process adopts a strategy that combines layer-by-layer pre-training with overall fine-tuning. First, the parameters of each layer are pre-trained through unsupervised learning, and then supervised fine-tuning is performed by combining labeled data. The Dropout mechanism is introduced, and L2 regularization is used to constrain the model parameters. An early stopping method is added to control the number of training rounds. The feature reconstruction error on the validation set is used as the indicator. At the same time, an overtone feature enhancement layer is added at the output of the autoencoder.

5. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, Low latency control is implemented in the real-time timbre synthesis and modulation steps. Latency optimization is achieved by dynamically adjusting the synthesis frame length and processing priority. Latency control employs... The delay adjustment rate is calculated, where dl / dt is the delay change rate, β is the adjustment coefficient ranging from 0.3 to 0.7, and l(t) is the actual delay at time t. By monitoring the performance rhythm characteristics in real time and introducing a preloading mechanism, the subsequent feature trend is predicted based on the performance parameters of the first 3 frames. Combined with the inverse Fourier transform accelerated by FPGA hardware, the processing time of a single frame is compressed to less than 5ms. The synthesis delay is stabilized within the perceptual delay-free range through low-latency control operations.

6. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, In the multidimensional timbre feature extraction step, the harmonic structure feature extraction adopts a harmonic superposition model. The theoretical frequencies of each order of overtones are calculated by octet of the fundamental frequency. The peak detection algorithm is used to locate the actual overtone peaks. The overtone amplitude ratio, overtone frequency deviation, and overtone decay time constant are extracted. The overtone amplitude ratio is the ratio of the amplitude of each order of overtone to the amplitude of the fundamental frequency. The overtone decay time constant is obtained by exponential fitting. A subdivided mapping relationship is established according to seven dynamic levels from ppp to fff. For missing overtones, an interpolation algorithm between adjacent overtones is used for compensation. At the same time, combined with the characteristics of the piano resonator, the frequency shift of high-frequency overtones is corrected. The influence of resonance on overtones is simulated by a finite element model, and the correction factor is incorporated into the overtone frequency calculation.

7. The real-time piano timbre simulation method based on machine learning according to claim 1, characterized in that, In the dynamic optimization and feedback iteration steps, a quantitative evaluation of timbre similarity is introduced, and similarity calculation is performed through... Achieve, where S is the similarity between the synthesized timbre and the target timbre. For the i-th feature parameter value of the synthesized timbre, Let be the value of the i-th feature parameter of the target timbre, and n be the total number of feature parameters. A weighted fusion strategy of static feature similarity and dynamic feature similarity is adopted. A similarity and rating mapping model is established by combining the subjective ratings of users from 1 to 5. The model weights are updated collaboratively based on the similarity results and user ratings. For feature combinations with low similarity and low ratings, their weight coefficients in the loss function are increased. At the same time, a user preference feature library is constructed.

8. A system for applying the real-time piano timbre simulation method based on machine learning as described in any one of claims 1-7, characterized in that, Includes the following modules: Data acquisition and annotation module: Equipped with a high-fidelity microphone array and a 24-bit ADC converter, the microphone array features cardioid polarity and spacing, and is synchronously connected to piano keyboard and pedal position sensors to collect key pressure and pedal depth data. It is also equipped with a touch-screen annotation terminal. Audio preprocessing module: integrates digital signal processing unit, including pre-emphasis filter, frame segmenter, windower, Fourier transform and noise suppressor. The pre-emphasis filter is a first-order high-pass filter, the frame segmenter supports variable frame length, the windower has built-in Hanning window, Hamming window and Blackman window options, and the noise suppressor adopts a combination algorithm of spectral subtraction and Wiener filtering. Feature extraction module: It consists of time-frequency feature analysis, overtone feature parsing and dynamic feature calculation unit. The time-frequency feature analysis unit calculates MFCC and its difference parameters. The overtone feature parsing unit locates the overtone peak through harmonic decomposition. The dynamic feature calculation unit outputs the first and second order differences of features of adjacent frames and similarity parameters. Model training module: Equipped with a GPU acceleration unit, it supports multi-GPU parallel computing, has built-in stacked autoencoder, generative adversarial network and long short-term memory network algorithm libraries, integrates Adam and SGD optimizers, and supports model version management and incremental training; Real-time analysis module: It adopts FPGA high-speed processing unit, integrates signal filtering and parameter conversion unit, performs debouncing and smoothing processing on key and pedal signals, converts discrete signals into continuous performance parameter sequence, and synchronously outputs technique recognition results; The timbre synthesis module consists of a spectrum generation, waveform conversion, and parameter modulation unit. The spectrum generation unit receives the characteristic parameters output by the model, the waveform conversion unit generates a time-domain waveform through inverse Fourier transform, and the parameter modulation unit adjusts the delay and attenuation characteristics according to the pedal parameters. It also has a built-in waveform splicing optimization algorithm. Optimized feedback module: Equipped with a touch-screen interactive interface, it supports users to rate the synthesized timbre from 1 to 5 points, receive text correction suggestions, and includes a rating data processing and model update triggering unit. It associates and stores the rating with feature parameters and triggers the incremental training process. Storage module: Uses an SSD array to store raw audio data, feature vector sets and trained model parameters, supports data classification indexing, and is equipped with a data backup unit.

9. The system for real-time simulation of piano timbre based on machine learning according to claim 8, characterized in that, It also includes a low-latency transmission module, which uses a PCIe 4.0 high-speed bus, supports burst data transmission, integrates a data verification unit and a timing synchronization unit, reduces transmission latency through a cache pre-read mechanism, and uses a circular cache strategy to avoid data overflow. The data verification unit uses the CRC32 verification algorithm, and the timing synchronization unit uses GPS timing signals to achieve cross-module clock calibration. The performance parameter signal is set to the highest priority, the feature data is set to the medium priority, and the log data is set to the low priority.

10. The system for real-time simulation of piano timbre based on machine learning according to claim 8, characterized in that, It also includes a multi-scene adaptation module, which contains a scene parameter configuration unit and a timbre template library. The scene parameter configuration unit supports parameter presets for three scenarios: concert hall, home performance, and recording studio. The timbre template library stores the baseline timbre characteristics of different piano models. Users can customize scene parameters and save them as new templates. The module adjusts the synthesis strategy by calling the feature parameters of the corresponding template to adapt to the timbre requirements of different usage scenarios. It also adds seamless transition technology for scene switching, using a cross-fade algorithm to achieve smooth switching of scene parameters. In addition, it integrates an environmental acoustic detection unit, which collects current environmental noise and reverberation data through a built-in microphone and automatically recommends suitable scene templates.

Images