A vocal automatic tuning method, device, equipment and medium of a digital musical instrument

Abstract

The present application relates to a kind of vocal automatic tuning method, device, equipment and medium of digital musical instrument, wherein, method includes: collecting human voice waveform, and the human voice waveform is converted into PCM data stream by analog-digital conversion;The PCM data stream is stripped off timbre information by inverse filter, and only excitation residual signal containing vocal cord vibration frequency is obtained;Target fundamental frequency is generated based on PCM data stream, keyboard interface input and external accompaniment audio;According to the excitation residual signal and target fundamental frequency, sound synthesis is carried out, and the modified human voice waveform is obtained.The present application can realize high-fidelity sound source synthesis and AI real-time tuning with formant protection.

Description

Technical Field

[0001] This invention relates to the field of digital music processing technology, and in particular to a method, apparatus, device, and medium for automatic vocal correction of digital musical instruments. Background Technology

[0002] In current digital musical instrument applications such as digital pianos and synthesizers, real-time vocal auto-tuning has become an important function for enhancing performance. However, existing embedded auto-tuning solutions have the following drawbacks:

[0003] Lack of formant protection: Simple time-domain or frequency-domain modulation algorithms destroy the physical characteristics of the human voice, resulting in heavy sound correction and failing to meet the needs of professional singing.

[0004] Limited analytical dimensions: It cannot analyze the tonality of external audio input (such as Bluetooth or USB input) in real time, which makes it easy for vocals to conflict with the accompaniment in terms of music theory.

[0005] Poor synchronization performance: In the dual-chip architecture, there is a phase difference (jitter) between sound source synthesis and human voice processing due to the buffer and data bus. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for automatic vocal correction of digital musical instruments, which can realize high-fidelity sound source synthesis and AI real-time correction with formant protection.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide an automatic vocal correction method for digital musical instruments, comprising the following steps:

[0008] The human voice waveform is acquired and converted into a PCM data stream through analog-to-digital conversion;

[0009] The PCM data stream is stripped of its timbre information by an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency.

[0010] The target fundamental frequency is generated based on PCM data stream, piano key interface input, and external accompaniment audio.

[0011] Based on the excitation residual signal and the target fundamental frequency, sound synthesis is performed to obtain the modified human voice waveform;

[0012] The inverse filter is constructed based on LPC prediction coefficients, which are obtained by using the PCM data stream as input to the source-filter acoustic model and solving the Yule-Walker equations to obtain LPC prediction coefficients of orders 12-18.

[0013] The generation of the target fundamental frequency based on PCM data stream, piano key interface input, and external accompaniment audio is specifically as follows:

[0014] When only human voice waveforms exist, the pure human voice automatic tone correction mode is used to generate the target fundamental frequency based on the PCM data stream;

[0015] When there is a piano key interface input, the piano key linkage tone correction mode is adopted to generate the target fundamental frequency based on the key notes and key force input through the piano key interface;

[0016] When there is external accompaniment audio but no keyboard input, the accompaniment-driven pitch correction mode is used to generate the target base frequency based on the external accompaniment audio and PCM data stream.

[0017] The generation of the target base frequency based on the PCM data stream specifically includes:

[0018] The time-domain PCM data stream is converted into a time-frequency graph using short-time Fourier transform;

[0019] The time-frequency graph is input into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency;

[0020] Based on the original baseband frequency, the Viterbi algorithm is used for dynamic programming, and the optimal continuous state sequence is found by combining the historical frame states to obtain the target baseband frequency.

[0021] The process of generating the target fundamental frequency based on the key notes and key pressure input from the piano key interface specifically includes:

[0022] A heuristic weighting algorithm based on key pressure is used to identify the key with the greatest key pressure as the main melody note;

[0023] The key notes of the main melody are converted into absolute frequencies by looking up a table, and these absolute frequencies are used as the target fundamental frequency.

[0024] The process of generating the target fundamental frequency based on the external accompaniment audio and PCM data stream specifically includes:

[0025] CQT analysis was performed on the external accompaniment audio, and the full-frequency energy was folded and projected onto 12 semitones to obtain pitch contour features;

[0026] The pitch contour features are input into a pre-trained bidirectional gated recurrent unit to analyze the evolution of the pitch contour sequence over time and generate a tonality mask.

[0027] The time-domain PCM data stream is converted into a time-frequency graph using short-time Fourier transform;

[0028] The time-frequency graph is input into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency;

[0029] The original fundamental frequency is compared with the 12-tone equal temperament chart to find the nearest standard pitch;

[0030] If the corresponding bit in the tone mask of the nearest standard tone is the first value, then the original fundamental frequency is taken as the target fundamental frequency;

[0031] If the corresponding bit of the nearest standard pitch in the tonality mask is the second value, find the first value in the tonality mask that is closest to the original fundamental frequency, and use it as the target fundamental frequency.

[0032] When using the pure vocal automatic tone correction mode or the accompaniment-driven tone correction mode, the step of synthesizing sound based on the excitation residual signal and the target fundamental frequency to obtain the modified vocal waveform specifically includes:

[0033] Based on the target fundamental frequency, the fundamental frequency synchronization point of the excitation residual signal is located using a waveform synchronization superposition algorithm. The pulse spacing is stretched or compressed according to the target fundamental frequency without changing the waveform envelope shape to obtain the frequency-shifted excitation residual signal.

[0034] The frequency-shifted excitation residual signal is input into the synthesis filter to obtain the modified human voice waveform;

[0035] The synthesized filter is constructed based on LPC prediction coefficients.

[0036] When using the piano key linkage tone correction mode, the step of synthesizing sound based on the excitation residual signal and the target fundamental frequency to obtain the modified human voice waveform specifically includes:

[0037] Map the pressure applied to the tracking step size;

[0038] Using the tracking step size as the pitch correction speed, the fundamental frequency synchronization point of the excitation residual signal is located according to the target fundamental frequency using a waveform synchronization superposition algorithm. The pulse spacing is stretched or compressed according to the target fundamental frequency without changing the waveform envelope shape to obtain the frequency-shifted excitation residual signal.

[0039] The frequency-shifted excitation residual signal is input into the synthesis filter to obtain the modified human voice waveform;

[0040] The synthesized filter is constructed based on LPC prediction coefficients.

[0041] The technical solution adopted by this invention to solve its technical problem is: to provide an automatic vocal correction device for digital musical instruments, comprising:

[0042] The acquisition and conversion module is used to acquire human voice waveforms and convert the human voice waveforms into PCM data streams through analog-to-digital conversion;

[0043] The filtering module is used to strip the timbre information from the PCM data stream through an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency.

[0044] The generation module is used to generate the target fundamental frequency based on the human voice waveform, piano key interface input, and external accompaniment audio.

[0045] The synthesis module is used to synthesize sound based on the excitation residual signal and the target fundamental frequency to obtain a modified human voice waveform.

[0046] The inverse filter is constructed based on LPC prediction coefficients, which are obtained by using the PCM data stream as input to the source-filter acoustic model and solving the Yule-Walker equations to obtain LPC prediction coefficients of orders 12-18.

[0047] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned automatic vocal correction method for digital musical instruments.

[0048] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of the above-mentioned automatic vocal correction method for digital musical instruments.

[0049] Beneficial effects

[0050] By employing the aforementioned technical solutions, this invention offers the following advantages and positive effects compared to existing technologies: This invention acquires human voice waveforms and converts them into PCM data streams. Based on LPC prediction coefficients, an inverse filter is constructed to effectively separate the human voice into timbre information and an excitation residual signal containing only vocal cord vibration frequencies. This preserves the original formant characteristics of the singer's vocal tract from the source, avoiding problems such as formant distortion and "chipmunk voice" that occur during modulation in traditional pitch correction techniques, significantly improving the naturalness and fidelity of pitch correction. This method integrates human voice waveforms, keyboard interface input, and external accompaniment audio to generate a target fundamental frequency, achieving multimodal signal collaborative constraints. It supports both automatic pitch correction of pure human voices and zero-latency keyboard linkage and intelligent accompaniment tonality following, resulting in fast pitch correction response and strong music theory compatibility. The overall method is computationally efficient, adaptable to embedded platforms, and achieves low-latency real-time processing with limited computing power. It also boasts high sound quality, strong interactivity, and high system integration, making it directly applicable to live digital instrument performances and professional recording scenarios, significantly enhancing product practicality and performance expressiveness. Attached Figure Description

[0051] Figure 1This is a flowchart of the automatic vocal correction method for digital musical instruments according to the first embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0053] The first embodiment of the present invention relates to an automatic vocal correction method for digital musical instruments, such as... Figure 1 As shown, it includes the following steps:

[0054] Step S100: Acquire human voice waveform and convert the human voice waveform into PCM data stream through analog-to-digital conversion.

[0055] In this step, the human voice waveform can be collected by a microphone, and the collected human voice waveform can be input into an ADC converter to convert it into a PCM data stream. The PCM data stream can be windowed and framed, and the resulting frame of PCM data can be copied into two channels, which are used in steps S200 and S300 respectively.

[0056] Step S200: The PCM data stream is stripped of timbre information by an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency.

[0057] The inverse filter in this step is constructed based on LPC prediction coefficients, which can be obtained as follows: Based on the source-filter acoustic model, the PCM data stream is used as the input to the source-filter acoustic model, and LPC prediction coefficients of orders 12-18 are obtained by solving the Yule-Walker equations. This set of LPC prediction coefficients represents the physical formant envelope (i.e., timbre characteristics) of the singer's vocal tract. After passing the PCM data stream through the inverse filter constructed with these LPC prediction coefficients, the timbre information can be stripped away, resulting in a pure excitation residual signal containing only the vocal cord vibration frequencies.

[0058] Step S300: Generate the target fundamental frequency based on the PCM data stream, the keyboard interface input, and the external accompaniment audio. In this step, when only a human voice waveform exists, a pure human voice automatic pitch correction mode can be used to generate the target fundamental frequency based on the PCM data stream; when a keyboard interface input exists, a keyboard-linked pitch correction mode can be used to generate the target fundamental frequency based on the key notes and key pressure input from the keyboard interface; when an external accompaniment audio exists but no keyboard interface input exists, an accompaniment-driven pitch correction mode is used to generate the target fundamental frequency based on the external accompaniment audio and the PCM data stream.

[0059] When using the pure vocal automatic tone correction mode, the target fundamental frequency is generated based on the PCM data stream, specifically including:

[0060] First, the time-domain PCM data stream is converted into a time-frequency graph using short-time Fourier transform (STFT);

[0061] Then, the time-frequency graph is input into a pre-trained one-dimensional convolutional neural network (CNN), which automatically filters out environmental noise and harmonic interference through the convolution kernel to extract the original fundamental frequency;

[0062] Finally, based on the original fundamental frequency, dynamic programming using the Viterbi algorithm is employed, and the optimal continuous state sequence is found by combining historical frame states to obtain the target fundamental frequency. This method prevents abrupt changes in the target fundamental frequency at critical points, thus obtaining a smooth target fundamental frequency.

[0063] When using the key-linked pitch correction mode, the target fundamental frequency is generated based on the key notes and key pressure input from the key interface. Specifically, this includes:

[0064] First, a heuristic weighting algorithm based on key pressure is used to identify the key with the greatest key pressure as the main melody note;

[0065] Then, the key notes of the main melody are converted into absolute frequencies by looking up a table, and the absolute frequencies are used as the target fundamental frequency.

[0066] In the key-linked pitch correction mode, this implementation method uses an event-driven activation note stack. When multiple keys are pressed simultaneously (in chord mode), a heuristic weighting algorithm based on key pressure is used to identify the key with the greatest pressure as the main melody note. The key notes of the main melody note are converted into absolute frequencies by looking up a table. Compared with using an AI model, this method can achieve true zero-latency target fundamental frequency locking.

[0067] Using an accompaniment-driven pitch correction mode, the target fundamental frequency is generated based on the external accompaniment audio and PCM data stream, specifically including the following steps:

[0068] First, CQT analysis was performed on the external accompaniment audio, and the full-frequency energy was folded and projected onto 12 semitones to obtain the pitch contour features.

[0069] Next, the pitch contour features are input into a pre-trained bidirectional gated recurrent unit to analyze the evolution of the pitch contour sequence over time and generate a tonality mask. This tonality mask is generated only when the probability of a certain tonality predicted by the bidirectional gated recurrent unit exceeds a threshold.

[0070] Then, the time-domain PCM data stream is converted into a time-frequency graph using a short-time Fourier transform;

[0071] Then, the time-frequency graph is input into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency;

[0072] The original fundamental frequency is then compared with the 12-tone equal temperament chart to find the nearest standard pitch;

[0073] If the corresponding bit in the tone mask of the nearest standard tone is the first value, then the original fundamental frequency is taken as the target fundamental frequency;

[0074] If the corresponding bit of the nearest standard pitch in the tonality mask is the second value, find the first value in the tonality mask that is closest to the original fundamental frequency, and use it as the target fundamental frequency.

[0075] Through the above steps, the original fundamental frequency is compared with the 12-tone equal temperament table. If the corresponding bit of the nearest standard tone in the tonality mask is "0" (i.e., the second value), it means that it belongs to the out-of-tone tone and will produce dissonance. At this time, the scale with the nearest distance of "1" from the original fundamental frequency to the tonality mask is taken as the target fundamental frequency. In this way, the human voice can be absorbed, thereby ensuring that no matter how out of tune the original singing is, the output human voice always conforms to the macroscopic music theory tonality of the accompaniment stream, thus solving the problem of disharmony between the human voice and the background music.

[0076] Step S400: Sound synthesis is performed based on the excitation residual signal and the target fundamental frequency to obtain the modified human voice waveform.

[0077] In this step, when using the pure vocal automatic tone correction mode or the accompaniment-driven tone correction mode, during synthesis, firstly, the fundamental frequency synchronization point of the excitation residual signal is located using a waveform synchronization superposition algorithm based on the target fundamental frequency. Then, without changing the waveform envelope shape, the pulse interval is stretched or compressed according to the target fundamental frequency to obtain the frequency-shifted excitation residual signal. Next, the frequency-shifted excitation residual signal is input into the synthesis filter to obtain the modified vocal waveform. The synthesis filter is constructed based on LPC prediction coefficients. Since the LPC prediction coefficients of the synthesis filter remain unchanged, the final generated sound achieves the target pitch while perfectly preserving the singer's original timbre, avoiding "chipmunk voice" distortion, and completing blind testing and automatic correction of the vocal waveform signal input to the microphone.

[0078] When using the key-linked pitch correction mode, during synthesis, firstly, the key pressure is mapped to a tracking step size; then, using the tracking step size as the pitch correction speed, a waveform synchronization superposition algorithm is used to locate the fundamental frequency synchronization point of the excitation residual signal based on the target fundamental frequency. The pulse spacing is stretched or compressed according to the target fundamental frequency without changing the waveform envelope shape to obtain the frequency-shifted excitation residual signal; finally, the frequency-shifted excitation residual signal is input into a synthesis filter to obtain the modified vocal waveform. The synthesis filter is constructed based on LPC prediction coefficients. In this method, if the performer hits the key hard (i.e., with a large key pressure), the mapped tracking step size is large, allowing the pitch correction to be completed instantly, producing a mechanical, leaping feel similar to electronic music. If the performer touches the key lightly (i.e., with a large key pressure), the mapped tracking step size is small. This method smoothly approaches the target frequency within several frames using glissando techniques to simulate natural vocal transitions.

[0079] It is easy to see that this invention, by acquiring human voice waveforms and converting them into PCM data streams, constructs an inverse filter based on LPC prediction coefficients, effectively separating the human voice into timbre information and an excitation residual signal containing only vocal cord vibration frequencies. This preserves the singer's original vocal tract formant characteristics from the source, avoiding problems such as formant distortion and "chipmound voice" that occur during modulation in traditional pitch correction techniques, thus significantly improving the naturalness and fidelity of pitch correction. This method integrates human voice waveforms, keyboard interface input, and external accompaniment audio to generate a target fundamental frequency, achieving multi-modal signal collaborative constraints. It supports both automatic pitch correction of pure human voices and zero-delay keyboard linkage and intelligent accompaniment tonality following, resulting in fast pitch correction response and strong music theory compatibility.

[0080] The second embodiment of the present invention relates to an automatic vocal correction device for a digital musical instrument, comprising:

[0081] The acquisition and conversion module is used to acquire human voice waveforms and convert the human voice waveforms into PCM data streams through analog-to-digital conversion;

[0082] The filtering module is used to strip the timbre information from the PCM data stream through an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency.

[0083] The generation module is used to generate the target fundamental frequency based on the human voice waveform, piano key interface input, and external accompaniment audio.

[0084] The synthesis module is used to synthesize sound based on the excitation residual signal and the target fundamental frequency to obtain a modified human voice waveform.

[0085] The inverse filter is constructed based on LPC prediction coefficients, which are obtained by using the PCM data stream as input to the source-filter acoustic model and solving the Yule-Walker equations to obtain LPC prediction coefficients of orders 12-18.

[0086] The generation module includes:

[0087] The first generation unit is used to generate the target fundamental frequency based on the PCM data stream when only human voice waveforms exist, using a pure human voice automatic tone correction mode.

[0088] The second generation unit is used to generate the target fundamental frequency based on the key notes and key force input from the key interface when there is a key interface input.

[0089] The third generation unit is used to generate the target fundamental frequency based on the external accompaniment audio and PCM data stream when there is external accompaniment audio but no piano key interface input, using the accompaniment-driven pitch correction mode.

[0090] The first generation unit includes:

[0091] The conversion unit is used to convert the time-domain PCM data stream into a time-frequency diagram through short-time Fourier transform;

[0092] The extraction unit is used to input the time-frequency diagram into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency;

[0093] The dynamic programming unit is used to perform dynamic programming based on the original baseband frequency using the Viterbi algorithm, and to find the optimal continuous state sequence by combining the historical frame states to obtain the target baseband frequency.

[0094] The second generation unit includes:

[0095] The recognition unit is used to identify the key with the greatest key pressure as the main melody note using a heuristic weighting algorithm based on key pressure.

[0096] The lookup table unit is used to convert the key notes of the main melody into absolute frequencies through a lookup table conversion method, and use the absolute frequencies as the target fundamental frequency.

[0097] The third generation unit includes:

[0098] The analysis projection unit is used to perform CQT analysis on the external accompaniment audio, folding and projecting the full-frequency energy onto 12 semitones to obtain pitch contour features;

[0099] The prediction unit is used to input pitch contour features into a pre-trained bidirectional gated recurrent unit to analyze the evolution of the pitch contour sequence over time and generate a tonality mask.

[0100] The conversion unit is used to convert the time-domain PCM data stream into a time-frequency diagram through short-time Fourier transform;

[0101] The extraction unit is used to input the time-frequency diagram into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency;

[0102] The comparison unit is used to compare the original fundamental frequency with the 12-tone equal temperament table to find the nearest standard tone;

[0103] The first determining unit is used to determine the original fundamental frequency as the target fundamental frequency when the corresponding bit in the tonality mask of the nearest standard tone is a first value;

[0104] The second determining unit is used to find the pitch of the first value closest to the original fundamental frequency in the tonality mask when the corresponding bit of the nearest standard pitch in the tonality mask is the second value, and determine it as the target fundamental frequency.

[0105] When using the pure vocal automatic pitch correction mode or the accompaniment-driven pitch correction mode, the synthesis module includes:

[0106] The first frequency shifting unit is used to locate the fundamental frequency synchronization point of the excitation residual signal according to the target fundamental frequency using a waveform synchronization superposition algorithm, and to stretch or compress the pulse spacing according to the target fundamental frequency without changing the waveform envelope shape, so as to obtain the frequency-shifted excitation residual signal.

[0107] The first synthesis unit is used to input the frequency-shifted excitation residual signal into the synthesis filter to obtain the modified human voice waveform.

[0108] The synthesized filter is constructed based on LPC prediction coefficients.

[0109] When using the key-linked pitch correction mode, the synthesis module includes:

[0110] The mapping unit is used to map the key pressure to a tracking step size;

[0111] The second frequency shifting unit is used to locate the fundamental frequency synchronization point of the excitation residual signal according to the target fundamental frequency using the tracking step size as the pitch correction speed and the waveform synchronization superposition algorithm, and to stretch or compress the pulse spacing according to the target fundamental frequency without changing the waveform envelope shape, so as to obtain the frequency-shifted excitation residual signal.

[0112] The second synthesis unit is used to input the frequency-shifted excitation residual signal into the synthesis filter to obtain the modified human voice waveform.

[0113] The synthesized filter is constructed based on LPC prediction coefficients.

[0114] The present invention will be further illustrated by a specific embodiment below.

[0115] In this embodiment, the core of the device is a DSP chip with 1.5 TOPS of neural network acceleration computing power, and the specific physical connection is as follows:

[0116] Audio acquisition: The microphone signal is processed by the ADC and then enters the DSP through I2S IN1; the external analog audio is processed by the ADC and then enters the DSP through I2S IN2.

[0117] Wireless and digital input: The Bluetooth module receives audio and inputs it via I2S IN3; the USB interface is configured in full-duplex UAC (USB Audio Class) mode, supporting audio input and data transmission after processing.

[0118] Control input: The piano key interface scans and transmits key codes (Note) and velocity signals in real time.

[0119] Main output: The processed mix data is sent to the DAC for amplification via I2S OUT1.

[0120] To run both the sound synthesizer and the tone correction engine on a single chip, the device implements a scheduling strategy based on time slices and isolation of computing units.

[0121] The Synth engine utilizes a floating-point unit (FPU) and a DMA controller to process PCM sample data reading, linear interpolation, and ADSR envelope calculation for multiple repeating notes in real time. The AI ​​analysis module uses a 1.5 TOPS NPU unit to run a quantized lightweight CNN (convolutional neural network) for pitch evaluation and a recurrent neural network (i.e., a bidirectional gated recurrent unit) for tonality prediction, thereby freeing up CPU core computing power. The note information generated by the internal Synth engine is directly broadcast to the pitch correction module via registers, serving as the "true value" benchmark for vocal correction without any audio analysis.

[0122] In the pure human voice automatic sound correction mode, the device relies entirely on internal AI to perform blind testing and automatic correction of the microphone input signal.

[0123] First, the human voice waveform captured by the microphone is converted into a PCM data stream by the ADC and enters the DSP's I2S IN1. The DMA controller moves this data to a double buffer (Ping-Pong Buffer) and performs windowing and framing. This frame of PCM data is mirrored into two paths, which enter the control stream and the audio stream respectively.

[0124] In the control flow, the one-dimensional PCM data in the time domain is first converted into a two-dimensional time-frequency graph using a Short-Time Fourier Transform (STFT). A lightweight one-dimensional convolutional neural network (CNN) is then run using an NPU to automatically filter environmental noise and harmonic interference through convolutional kernels, extracting the periodic features of the fundamental frequency and outputting a probability vector. Finally, the Viterbi algorithm is used for dynamic programming, combining historical frame states to find the optimal continuous state sequence, preventing abrupt changes in the fundamental frequency at critical points, and outputting a smooth target reference frequency, Target F0.

[0125] In the audio stream, based on the source-filter acoustic model, the system solves the Yule-Walker equations in the floating-point unit (FPU) to calculate linear LPC prediction coefficients of orders 12-18. These LPC prediction coefficients represent the physical formant envelope (i.e., timbre characteristics) of the singer's vocal tract. The original signal is then passed through an inverse filter constructed using these LPC prediction coefficients to remove the timbre information, resulting in a pure excitation residual signal containing only the vocal cord vibration frequencies.

[0126] Then, the waveform synchronization overlay method (PSOLA) precisely locates the fundamental synchronization points (epochs) of the residual waveform based on the input target reference frequency Target F0. Through overlay and overlay techniques, the pulse spacing is stretched or compressed to change the fundamental frequency without altering the waveform envelope shape. Finally, the frequency-shifted residual is fed back into a synthesis filter composed of LPC prediction coefficients. Since the LPC prediction coefficients of the synthesis filter remain unchanged (i.e., the sound characteristics remain unchanged), the final generated sound achieves the target pitch while perfectly preserving the singer's original timbre, avoiding "chipmunk voice" distortion.

[0127] In the key-linked pitch correction mode, this embodiment breaks through the latency bottleneck of external AI blind testing and directly injects the high-priority hard real-time signals of the physical keys into the processing stream.

[0128] In this mode, an event-driven stack of active notes is first established. When multiple keys are pressed simultaneously (in chord mode), a heuristic weighting algorithm based on key pressure is used. The system identifies the key with the greatest pressure as the main melody note and converts its Note value into an absolute frequency using a lookup table. This absolute frequency is then used as the target fundamental frequency, Target F0. By directly overriding the CNN output in the pure vocal automatic pitch correction mode, true zero-latency reference locking can be achieved.

[0129] During synthesis, the key pressure is non-linearly mapped to the tracking step size (Speed) of the PSOLA algorithm. If the performer hits the key hard (i.e., the key pressure is large), the step size is set to the maximum, and the pitch correction is completed instantly, producing a mechanical jump similar to electronic music (hard pitch correction); if the performer touches the key lightly (i.e., the key pressure is small), the algorithm will smoothly approach the target frequency in the form of glissando over several frames, simulating a natural vocal transition.

[0130] The accompaniment-driven audio editing mode incorporates the external musical environment (accompaniment) into the analysis loop, resolving the issue of inconsistency between vocals and background music.

[0131] In this mode, CQT analysis is performed on the external accompaniment stream, folding and projecting the full-frequency energy onto 12 chromatic scales to extract the chroma (tone profile) features of the harmonic structure. The NPU-run bidirectional gated loop unit (Bi-GRU) analyzes the evolution of the chroma sequence over time. When the predicted probability of a certain key (such as C Major) exceeds the 80% confidence threshold, the system generates a binary valid chromatic scale mask (e.g., in the mask array, bits C, D, E, F, G, A, and B are 1, and the remaining bits are 0).

[0132] Then, the system compares the distance between the original fundamental frequency F0 and the 12-tone equal temperament table. If the nearest standard note corresponds to 0 in the mask (belonging to an out-of-key tone, which will produce dissonance), the system will forcibly calculate the minimum distance from the original fundamental frequency F0 to the scale with a value of 1 in the mask and absorb the human voice there, thereby ensuring that no matter how off-key the original singing is, the output human voice always conforms to the macroscopic musical theory tonality of the accompaniment stream.

[0133] In this embodiment, the USB controller is configured as a multi-channel bidirectional interface. The DSP's internal processing link sends the audio to I2S OUT1 while simultaneously mirroring the digital stream to the USB bus, enabling the device to function as both an audio output device and a high-performance digital sound card for high-quality audio editing and recording on a PC.

[0134] In this embodiment, computationally intensive CNN convolution calculations and RNN matrix operations are handled by the NPU acceleration unit within the DSP (using INT8 quantization to improve throughput); the highly real-time LPC analysis and PSOLA superposition operations are handled by the floating-point unit (FPU). A double-buffering (Ping-Pong Buffer) mechanism combined with DMA (Direct Memory Access) ensures that I2S data transfer does not consume CPU clock speeds. The entire algorithm chain is optimized to a single processing time of less than 2ms. With hardware buffering, the end-to-end latency of the entire system can be controlled within 5-8ms, which is completely imperceptible to the listener.

[0135] As can be seen, the present invention is computationally efficient, adaptable to embedded platforms, and achieves low-latency real-time processing with limited computing power. It also features high sound quality, strong interactivity, and high system integration, and can be directly applied to live performances and professional recording scenarios with digital musical instruments, significantly improving the practicality and performance of the product.

[0136] The third embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the automatic vocal correction method for a digital musical instrument according to the first embodiment.

[0137] The fourth embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the automatic vocal correction method for a digital musical instrument according to the first embodiment.

[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic vocal correction in a digital musical instrument, characterized in that, Includes the following steps: The human voice waveform is acquired and converted into a PCM data stream through analog-to-digital conversion; The PCM data stream is stripped of its timbre information by an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency. The target fundamental frequency is generated based on PCM data stream, piano key interface input, and external accompaniment audio. Based on the excitation residual signal and the target fundamental frequency, sound synthesis is performed to obtain the modified human voice waveform; The inverse filter is constructed based on LPC prediction coefficients, which are obtained by using the PCM data stream as input to the source-filter acoustic model and solving the Yule-Walker equations to obtain LPC prediction coefficients of orders 12-18.

2. The automatic vocal correction method for digital musical instruments according to claim 1, characterized in that, The generation of the target fundamental frequency based on PCM data stream, piano key interface input, and external accompaniment audio is specifically as follows: When only human voice waveforms exist, the pure human voice automatic tone correction mode is used to generate the target fundamental frequency based on the PCM data stream; When there is a piano key interface input, the piano key linkage tone correction mode is adopted to generate the target fundamental frequency based on the key notes and key force input through the piano key interface; When there is external accompaniment audio but no keyboard input, the accompaniment-driven pitch correction mode is used to generate the target base frequency based on the external accompaniment audio and PCM data stream.

3. The automatic vocal correction method for digital musical instruments according to claim 2, characterized in that, The generation of the target base frequency based on the PCM data stream specifically includes: The time-domain PCM data stream is converted into a time-frequency graph using short-time Fourier transform; The time-frequency graph is input into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency; Based on the original baseband frequency, the Viterbi algorithm is used for dynamic programming, and the optimal continuous state sequence is found by combining the historical frame states to obtain the target baseband frequency.

4. The automatic vocal correction method for digital musical instruments according to claim 2, characterized in that, The process of generating the target fundamental frequency based on the key notes and key pressure input from the piano key interface specifically includes: A heuristic weighting algorithm based on key pressure is used to identify the key with the greatest key pressure as the main melody note; The key notes of the main melody are converted into absolute frequencies by looking up a table, and these absolute frequencies are used as the target fundamental frequency.

5. The automatic vocal correction method for digital musical instruments according to claim 2, characterized in that, The process of generating the target fundamental frequency based on the external accompaniment audio and PCM data stream specifically includes: CQT analysis was performed on the external accompaniment audio, and the full-frequency energy was folded and projected onto 12 semitones to obtain pitch contour features; The pitch contour features are input into a pre-trained bidirectional gated recurrent unit to analyze the evolution of the pitch contour sequence over time and generate a tonality mask. The time-domain PCM data stream is converted into a time-frequency graph using short-time Fourier transform; The time-frequency graph is input into a pre-trained one-dimensional convolutional neural network to extract the original fundamental frequency; The original fundamental frequency is compared with the 12-tone equal temperament chart to find the nearest standard pitch; If the corresponding bit in the tone mask of the nearest standard tone is the first value, then the original fundamental frequency is taken as the target fundamental frequency; If the corresponding bit of the nearest standard pitch in the tonality mask is the second value, find the first value in the tonality mask that is closest to the original fundamental frequency, and use it as the target fundamental frequency.

6. The automatic vocal correction method for digital musical instruments according to claim 2, characterized in that, When using the pure vocal automatic tone correction mode or the accompaniment-driven tone correction mode, the step of synthesizing sound based on the excitation residual signal and the target fundamental frequency to obtain the modified vocal waveform specifically includes: Based on the target fundamental frequency, the fundamental frequency synchronization point of the excitation residual signal is located using a waveform synchronization superposition algorithm. The pulse spacing is stretched or compressed according to the target fundamental frequency without changing the waveform envelope shape to obtain the frequency-shifted excitation residual signal. The frequency-shifted excitation residual signal is input into the synthesis filter to obtain the modified human voice waveform; The synthesized filter is constructed based on LPC prediction coefficients.

7. The automatic vocal correction method for digital musical instruments according to claim 2, characterized in that, When using the piano key linkage tone correction mode, the step of synthesizing sound based on the excitation residual signal and the target fundamental frequency to obtain the modified human voice waveform specifically includes: Map the pressure applied to the tracking step size; Using the tracking step size as the pitch correction speed, the fundamental frequency synchronization point of the excitation residual signal is located according to the target fundamental frequency using a waveform synchronization superposition algorithm. The pulse spacing is stretched or compressed according to the target fundamental frequency without changing the waveform envelope shape to obtain the frequency-shifted excitation residual signal. The frequency-shifted excitation residual signal is input into the synthesis filter to obtain the modified human voice waveform; The synthesized filter is constructed based on LPC prediction coefficients.

8. An automatic vocal correction device for a digital musical instrument, characterized in that, include: The acquisition and conversion module is used to acquire human voice waveforms and convert the human voice waveforms into PCM data streams through analog-to-digital conversion; The filtering module is used to strip the timbre information from the PCM data stream through an inverse filter to obtain an excitation residual signal containing only the vocal cord vibration frequency. The generation module is used to generate the target fundamental frequency based on the human voice waveform, piano key interface input, and external accompaniment audio. The synthesis module is used to synthesize sound based on the excitation residual signal and the target fundamental frequency to obtain a modified human voice waveform. The inverse filter is constructed based on LPC prediction coefficients, which are obtained by using the PCM data stream as input to the source-filter acoustic model and solving the Yule-Walker equations to obtain LPC prediction coefficients of orders 12-18.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the automatic vocal correction method for a digital musical instrument as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the automatic vocal correction method for digital musical instruments as described in any one of claims 1-7.

Images