Vocal singing sound processing method and system
Through the sound processing system of vocal singing, the segmented and cut-off analysis and feature extraction technology are used to identify and adjust the various attributes of singing sounds, solving the problem that the existing technology is difficult to correct subtle defects, and achieving a more refined and personalized sound The process has improved the professional level of vocal drills and recording.
Patent Information
- Application Number
- CN202411317758.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing vocal singing sound processing techniques are difficult to accurately identify and correct subtle flaws in singing, such as pitch deviation, timbre unevenness and inappropriate control of dynamic range, especially when dealing with the unique voices of professional vocal singers.
A sound processing method and system for vocal singing is adopted, including sound input module, sound data analysis module, sound feature extraction module and sound feature processing module. Through segmentation and cut-off analysis, MFCC parameter extraction, VQ codebook training and Euclidean distance Judgment, identify and adjust the various attributes and states of the singing sound, including pitch, volume, tone and voice method, eliminate non-standard pronunciations and make targeted adjustments.
The refinement and personalization of the vocal singing voice is realized, the technical defects in singing are identified and corrected, the expressiveness and pleasantness of the sound are enhanced, and the professional level and artistic expression of vocal drills and recordings are significantly improved.
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vocal singing sound processing, and in particular to a vocal singing sound processing method and system. Background Art
[0002] In the current field of vocal singing sound processing technology, many systems aim to improve sound quality and enhance singing performance through digital signal processing technology. Common processing technologies include automatic pitch correction, dynamic range compression, and echo and noise elimination. These methods often rely on traditional digital signal processing algorithms, such as fast Fourier transform (FFT) and linear predictive coding (LPC), as well as rule-based audio effect applications. However, these technologies are usually only applicable to general sound quality improvement, lacking in-depth analysis and personalized processing of individual vocal characteristics, especially when dealing with the unique voices of professional vocal singers, and often cannot accurately identify and correct subtle defects in singing, such as slight pitch deviations, uneven timbre, or inappropriate control of dynamic range. Summary of the invention
[0003] In view of the deficiencies in the prior art, the present invention provides a sound processing method and system for vocal singing, which solves the problems raised in the above-mentioned background technology.
[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: a sound processing method and system for vocal singing, including a sound input module, a sound data analysis module, a sound feature extraction module, a sound feature processing module, and a sound output module;
[0005] The sound input module is used to operate the sound input of the singer during the vocal performance, and uses the singer's original sound output as the basic data model for later sound processing;
[0006] The sound data analysis module is used to perform segmentation and truncation analysis on the original sound data model of the vocal singer, and divide the original sound data model into a number of data packets with equal time as the unit for independent analysis, and the data packets include feature information of N dimensions, where N is a multiple of 1;
[0007] The sound feature extraction module is used to obtain each singing sound feature in each unit data packet and use it as the basic data node for processing. It is realized by calling the MATLAB script node in LabVIEW. The input speech signal is first pre-processed by noise reduction, and then its MFCC (Mel frequency cepstral coefficient) parameters are extracted, and its VQ codebook is trained. When performing speech recognition, the Euclidean distance between the VQ codebook of the sample speech signal and the MFCC of the test speech signal is calculated to determine whether it is less than the set threshold.
[0008] The sound feature processing module is used to perform corresponding adjustment operations on the singing voice of the vocal singer, compare each data packet segmented in the sound data analysis module with the audio information of the song to be sung, determine the difference amplitude between the sound feature information of each singer and the audio information of each lyrics of the song to be sung, and adjust the data packet of the singing voice accordingly according to the difference amplitude;
[0009] The sound output module is used to integrate, reorganize and play the processed singing sound.
[0010] Optionally, the sound input module is composed of a sound collection unit, a sound recording unit, an automatic orientation correction unit, and a storage unit;
[0011] The sound collection unit is used to collect various data of the singer's personal sound signal, including the parameters of sound wave direction, sound volume, and environmental status;
[0012] The sound recording unit is used to collect the original sound data. The recording process is not limited by time, venue environment, or volume, and the original sound is recorded.
[0013] The automatic orientation correction unit adjusts the orientation of the recording device according to the determination of the orientation of the sound wave center by the sound collection unit, so that the recording device faces the direction of the sound wave center.
[0014] The storage unit is used to save the sound content of the mobile phone for use as technical data.
[0015] Optionally, the sound data analysis module includes volume analysis, timbre analysis, phonation analysis, breathing state analysis, and coherence analysis;
[0016] Volume analysis is used to uniformly analyze and judge the volume of each equal amount of data packets, independently analyze the volume of each note, and store the analysis data;
[0017] Timbre analysis is used to analyze and judge the trend of the singer's pronunciation tone of each note, and calculate the difference between the pronunciation tone and the optimal value;
[0018] Vocal analysis, used to analyze the vocal state, vocal strength, and vocal continuity of the singer during the singing process;
[0019] Breathing state analysis is used to analyze the breathing intervals of the singer during the singing process, which also includes the analysis of breathing frequency, breathing adjustment ability, transition between breathing and singing, and oxygen intake;
[0020] Coherence analysis is used to analyze the degree of connection between each two adjacent sound data packets and the tone connection ability between each two adjacent notes;
[0021] The sound data analysis module also includes a capturing unit for capturing non-standard pronunciations found during the sound data analysis process one by one.
[0022] Optionally, the sound feature extraction module is specifically used to perform frequency domain transformation processing on the chorus audio data to obtain chorus frequency domain data corresponding to the chorus audio data; based on the audio feature parameters of the singer, extract the solo frequency domain data corresponding to the audio feature parameters in the chorus frequency domain data.
[0023] Optionally, the sound feature extraction module includes fundamental frequency extraction and formant extraction;
[0024] Pitch frequency extraction, a pitch detection method used for different speakers and different environmental requirements, divides the speech signal into speech frames of a certain length, and then calculates the average pitch period for each frame of speech data;
[0025] Resonant peak extraction adopts the resonance peak extraction method based on inverse filter, decomposes the speech signal into modulation components and adopts the algorithm of extracting resonance peaks by nonlinear processing method, and the resonance peak extraction algorithm based on multi-band energy separation is based on the FM-AM nonlinear generation model of speech signal, and uses a set of bandpass filters to separate the various resonance peak components, and then uses the energy separation algorithm to obtain the instantaneous frequency and instantaneous amplitude of each resonance peak. Before filtering, short-time Fourier transform is required to estimate the frequency and bandwidth of each resonance peak.
[0026] Optionally, the sound feature processing module includes a difference value analysis unit, an adjustment mode analysis unit, and a data packet connection adjustment unit;
[0027] The difference value analysis unit is used to compare each sound feature information with the audio information of the song to be sung, and determine the difference information between the sound feature information and the audio information of the singer to be sung, including the volume, timbre, and pronunciation strength;
[0028] The adjustment mode analysis unit determines the priority of the sound features of each dimension; for the standard singing data, the sound features of the lyrics are determined in turn according to the audio information of the lyrics from the sound features of the highest priority dimension; the sound features of each dimension are compared with the sound features of the corresponding dimension of the lyrics, the difference range between the sound features of the corresponding dimension is determined, and according to the difference amplitude, the sound features with the largest amplitude are adjusted first, until all adjustments are completed;
[0029] The data packet connection adjustment unit is used to make a second adjustment to the connection of adjacent data packets after the adjustment is completed, so as to ensure the continuity between the timbre and volume of each data packet.
[0030] Optionally, the sound output module includes a data packet splicing unit and a sound data playing unit;
[0031] The data packet splicing unit is used to connect the adjusted and connected data packets to restore the complete singing sound, maintain the continuity between the singing sounds, and avoid the phenomenon of mismatch;
[0032] The sound data playback unit is used to play the singing after the sound processing is completed.
[0033] A sound processing method for vocal singing according to any one of the above claims, comprising the following specific steps:
[0034] Step 1: The vocal singing sound is uniformly recorded by the sound collection unit and the sound recording unit, wherein the sound recording direction of the recording device can be adjusted in real time by the automatic orientation correction unit during the recording process;
[0035] Step 2: The recorded sound data is divided into several data packets for independent analysis, and marked respectively, such as: A00; A01; A02...Z98; Z99; the data packet includes feature information of N dimensions, where N is a multiple of 1; and the non-standard pronunciations in the divided data packets are captured and marked one by one;
[0036] Step 3: Analyze the volume, timbre, and vocalization status of each data packet independently, and also analyze the singer's breathing status and pronunciation continuity during the singing process;
[0037] Step 4: collecting the vocal features of the singer in the analysis process in step 3, wherein the special extraction includes fundamental frequency extraction and formant extraction;
[0038] Step 5: Compare the extracted sound features with the sample data to obtain difference data, and make targeted adjustments and records according to the range of the difference data;
[0039] Step 6: Make a second adjustment to the connection points of adjacent data packets after the adjustment is completed to ensure the continuity of the timbre and volume of each data packet;
[0040] Step 7: Restore the connected data packets to complete continuous sound data, and play or record after maintaining the continuity between the singing sounds.
[0041] The present invention provides a method and system for processing vocal singing sound, which has the following beneficial effects:
[0042] The sound processing method and system for vocal singing automatically recognize and classify various attributes and states of sound, including but not limited to pitch, volume, timbre and vocalization mode, through the cooperation between a sound data analysis module, a sound feature extraction module and a sound feature processing module; adopt an abnormal sound elimination algorithm to accurately recognize non-standard pronunciation and technical defects in singing, such as falsetto and broken voice; and recognize defects in singing skills, such as unstable vibrato or pitch problems, and provide targeted correction suggestions; the sound adjustment module intelligently adjusts the processing parameters according to the analysis results, such as balancing the sound spectrum by adjusting the frequency response of the equalizer, adjusting the compressor settings to optimize the dynamic range, and modifying the sound texture to enhance the expressiveness and clarity of the sound;
[0043] The innovation of the vocal singing sound processing method and system lies in the ability to provide more refined and personalized sound processing, and to conduct a comprehensive and in-depth analysis and improvement of the singer's voice; by eliminating abnormal sounds and correcting singing defects, the singer's technical flaws are effectively reduced, while enhancing the expressiveness and pleasantness of the sound; in addition, the system's adaptive adjustment mechanism makes the sound processing no longer limited to general settings, but can be optimized according to the specific needs of each singer, thereby significantly improving the professional level and artistic expression of vocal rehearsal and recording; this system has important application value for professional vocal education, singing practice and commercial music production, provides a technical innovation support for vocal singing, and promotes the development of audio processing technology. DETAILED DESCRIPTION
[0044] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0045] Embodiment 1: A sound processing system for vocal singing, comprising a sound input module, a sound data analysis module, a sound feature extraction module, a sound feature processing module, and a sound output module;
[0046] The sound input module is used to operate the sound input of the singer during the vocal performance, and uses the singer's original sound output as the basic data model for later sound processing;
[0047] The sound data analysis module is used to perform segmentation and truncation analysis on the original sound data model of the vocal singer, and divide the original sound data model into a number of data packets with equal time as the unit for independent analysis, and the data packets include feature information of N dimensions, where N is a multiple of 1;
[0048] The sound feature extraction module is used to obtain each singing sound feature in each unit data packet and use it as the basic data node for processing. It is realized by calling the MATLAB script node in LabVIEW. The input speech signal is first pre-processed by noise reduction, and then its MFCC (Mel frequency cepstral coefficient) parameters are extracted, and its VQ codebook is trained. When performing speech recognition, the Euclidean distance between the VQ codebook of the sample speech signal and the MFCC of the test speech signal is calculated to determine whether it is less than the set threshold.
[0049] The sound feature processing module is used to perform corresponding adjustment operations on the singing voice of the vocal singer, compare each data packet segmented in the sound data analysis module with the audio information of the song to be sung, determine the difference amplitude between the sound feature information of each singer and the audio information of each lyrics of the song to be sung, and adjust the data packet of the singing voice accordingly according to the difference amplitude;
[0050] The sound output module is used to integrate, reorganize and play the processed singing sound.
[0051] In this embodiment, the sound input module is composed of a sound collection unit, a sound recording unit, an automatic orientation correction unit, and a storage unit;
[0052] The sound collection unit is used to collect various data of the singer's personal sound signal, including the parameters of sound wave direction, sound volume, and environmental status;
[0053] The sound recording unit is used to collect the original sound data. The recording process is not limited by time, venue environment, or volume, and the original sound is recorded.
[0054] The automatic orientation correction unit adjusts the orientation of the recording device according to the determination of the orientation of the sound wave center by the sound collection unit, so that the recording device faces the direction of the sound wave center.
[0055] The storage unit is used to save the sound content of the mobile phone for use as technical data.
[0056] In this embodiment, the sound data analysis module includes volume analysis, timbre analysis, phonation analysis, breathing state analysis, and coherence analysis;
[0057] Volume analysis is used to uniformly analyze and judge the volume of each equal amount of data packets, independently analyze the volume of each note, and store the analysis data;
[0058] Tone analysis is used to analyze and judge the trend of the singer's pronunciation tone of each note, and calculate the difference between the pronunciation tone and the optimal value;
[0059] Vocal analysis, used to analyze the vocal state, vocal strength, and vocal continuity of the singer during the singing process;
[0060] Breathing state analysis is used to analyze the breathing intervals of the singer during the singing process, which also includes the analysis of breathing frequency, breathing adjustment ability, transition between breathing and singing, and oxygen intake;
[0061] Coherence analysis is used to analyze the degree of connection between each two adjacent sound data packets and the tone connection ability between each two adjacent notes;
[0062] The sound data analysis module also includes a capturing unit for capturing non-standard pronunciations found during the sound data analysis process one by one.
[0063] In this embodiment, the sound feature extraction module is specifically used to perform frequency domain transformation processing on the chorus audio data to obtain chorus frequency domain data corresponding to the chorus audio data; based on the singer's audio feature parameters, the solo frequency domain data corresponding to the audio feature parameters are extracted from the chorus frequency domain data.
[0064] In this embodiment, the sound feature extraction module includes fundamental frequency extraction and formant extraction;
[0065] Pitch frequency extraction, a pitch detection method used for different speakers and different environmental requirements, divides the speech signal into speech frames of a certain length, and then calculates the average pitch period for each frame of speech data;
[0066] Resonant peak extraction adopts the resonance peak extraction method based on inverse filter, decomposes the speech signal into modulation components and adopts the algorithm of extracting resonance peaks by nonlinear processing method, and the resonance peak extraction algorithm based on multi-band energy separation is based on the FM-AM nonlinear generation model of speech signal, and uses a set of bandpass filters to separate the various resonance peak components, and then uses the energy separation algorithm to obtain the instantaneous frequency and instantaneous amplitude of each resonance peak. Before filtering, short-time Fourier transform is required to estimate the frequency and bandwidth of each resonance peak.
[0067] In this embodiment, the sound feature processing module includes a difference value analysis unit, an adjustment mode analysis unit, and a data packet connection adjustment unit;
[0068] The difference value analysis unit is used to compare each sound feature information with the audio information of the song to be sung, and determine the difference information between the sound feature information and the audio information of the singer to be sung, including the volume, timbre, and pronunciation strength;
[0069] The adjustment mode analysis unit determines the priority of the sound features of each dimension; for the standard singing data, the sound features of the lyrics are determined in turn according to the audio information of the lyrics from the sound features of the highest priority dimension; the sound features of each dimension are compared with the sound features of the corresponding dimension of the lyrics, the difference range between the sound features of the corresponding dimension is determined, and according to the difference amplitude, the sound features with the largest amplitude are adjusted first, until all adjustments are completed;
[0070] The data packet connection adjustment unit is used to make a second adjustment to the connection of adjacent data packets after the adjustment is completed, so as to ensure the continuity between the timbre and volume of each data packet.
[0071] In this embodiment, the sound output module includes a data packet splicing unit and a sound data playing unit;
[0072] The data packet splicing unit is used to connect the adjusted and connected data packets to restore the complete singing sound, maintain the continuity between the singing sounds, and avoid the phenomenon of mismatch;
[0073] The sound data playback unit is used to play the singing after the sound processing is completed.
[0074] Embodiment 2: A method for processing a vocal singing sound, comprising the following specific steps:
[0075] Step 1: The vocal singing sound is uniformly recorded by the sound collection unit and the sound recording unit, wherein the sound recording direction of the recording device can be adjusted in real time by the automatic orientation correction unit during the recording process;
[0076] Step 2: The recorded sound data is divided into several data packets for independent analysis, and marked respectively, such as: A00; A01; A02...Z98; Z99; the data packet includes feature information of N dimensions, where N is a multiple of 1; and the non-standard pronunciations in the divided data packets are captured and marked one by one;
[0077] Step 3: Analyze the volume, timbre, and vocalization status of each data packet independently, and also analyze the singer's breathing status and pronunciation continuity during the singing process;
[0078] Step 4: collecting the vocal features of the singer in the analysis process in step 3, wherein the special extraction includes fundamental frequency extraction and formant extraction;
[0079] Step 5: Compare the extracted sound features with the sample data to obtain difference data, and make targeted adjustments and records according to the range of the difference data;
[0080] Step 6: Make a second adjustment to the connection points of adjacent data packets after the adjustment is completed to ensure the continuity of the timbre and volume of each data packet;
[0081] Step 7: Restore the connected data packets to complete continuous sound data, and play or record after maintaining the continuity between the singing sounds.
[0082] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A sound processing system for vocal singing, characterized in that: It includes a sound input module, a sound data analysis module, a sound feature extraction module, a sound feature processing module, and a sound output module; The sound input module is used to operate the sound input of the singer during the vocal performance, and uses the singer's original sound output as the basic data model for later sound processing; The sound data analysis module is used to perform segmentation and truncation analysis on the original sound data model of the vocal singer, and divide the original sound data model into a number of data packets with equal time as the unit for independent analysis, and the data packets include feature information of N dimensions, where N is a multiple of 1; The sound feature extraction module is used to obtain each singing sound feature in each unit data packet and use it as the basic data node for processing. It is realized by calling the MATLAB script node in LabVIEW. The input speech signal is first pre-processed by noise reduction, and then its MFCC parameters are extracted and its VQ codebook is trained. When performing speech recognition, the Euclidean distance between the VQ codebook of the sample speech signal and the MFCC of the test speech signal is calculated to determine whether it is less than the set threshold. The sound feature processing module is used to perform corresponding adjustment operations on the singing voice of the vocal singer, compare each data packet segmented in the sound data analysis module with the audio information of the song to be sung, determine the difference amplitude between the sound feature information of each singer and the audio information of each lyrics of the song to be sung, and adjust the data packet of the singing voice accordingly according to the difference amplitude; The sound output module is used to integrate, reorganize and play the processed singing sound.
2. A sound processing system for vocal singing according to claim 1, characterized in that: The sound input module is composed of a sound collection unit, a sound recording unit, an automatic orientation correction unit, and a storage unit; The sound collection unit is used to collect various data of the singer's personal sound signal, including the parameters of sound wave direction, sound volume, and environmental status; The sound recording unit is used to collect the original sound data. The recording process is not limited by time, venue environment, or volume, and the original sound is recorded. The automatic orientation correction unit adjusts the orientation of the recording device according to the determination of the orientation of the sound wave center by the sound collection unit, so that the recording device faces the direction of the sound wave center. The storage unit is used to save the sound content of the mobile phone for use as technical data.
3. A sound processing system for vocal singing according to claim 1, characterized in that: The sound data analysis module includes volume analysis, timbre analysis, phonation analysis, breathing state analysis, and coherence analysis; Volume analysis is used to uniformly analyze and judge the volume of each equal amount of data packets, independently analyze the volume of each note, and store the analysis data; Timbre analysis is used to analyze and judge the trend of the singer's pronunciation tone of each note, and calculate the difference between the pronunciation tone and the optimal value; Vocal analysis, used to analyze the vocal state, vocal strength, and vocal continuity of the singer during the singing process; Breathing state analysis is used to analyze the breathing intervals of the singer during the singing process, which also includes the analysis of breathing frequency, breathing adjustment ability, transition between breathing and singing, and oxygen intake; Coherence analysis is used to analyze the degree of connection between each two adjacent sound data packets and the tone connection ability between each two adjacent notes; The sound data analysis module also includes a capturing unit for capturing non-standard pronunciations found during the sound data analysis process one by one.
4. The sound processing system for vocal singing according to claim 1, characterized in that: The sound feature extraction module is specifically used to perform frequency domain transformation processing on the chorus audio data to obtain chorus frequency domain data corresponding to the chorus audio data; based on the audio feature parameters of the singer, the solo frequency domain data corresponding to the audio feature parameters are extracted from the chorus frequency domain data.
5. The sound processing system for vocal singing according to claim 1, characterized in that: The sound feature extraction module includes fundamental frequency extraction and formant extraction; Pitch frequency extraction, a pitch detection method used for different speakers and different environmental requirements, divides the speech signal into speech frames of a certain length, and then calculates the average pitch period for each frame of speech data; Resonant peak extraction adopts the resonance peak extraction method based on inverse filter, decomposes the speech signal into modulation components and adopts the algorithm of extracting resonance peaks by nonlinear processing method, and the resonance peak extraction algorithm based on multi-band energy separation is based on the FM-AM nonlinear generation model of speech signal, and uses a set of bandpass filters to separate the various resonance peak components, and then uses the energy separation algorithm to obtain the instantaneous frequency and instantaneous amplitude of each resonance peak. Before filtering, short-time Fourier transform is required to estimate the frequency and bandwidth of each resonance peak.
6. The sound processing system for vocal singing according to claim 1, characterized in that: The sound feature processing module includes a difference value analysis unit, an adjustment mode analysis unit, and a data packet connection adjustment unit; The difference value analysis unit is used to compare each sound feature information with the audio information of the song to be sung, and determine the difference information between the sound feature information and the audio information of the singer to be sung, including the volume, timbre, and pronunciation strength; The adjustment mode analysis unit determines the priority of the sound features of each dimension; for the standard singing data, the sound features of the lyrics are determined in turn according to the audio information of the lyrics from the sound features of the highest priority dimension; the sound features of each dimension are compared with the sound features of the corresponding dimension of the lyrics, the difference range between the sound features of the corresponding dimension is determined, and according to the difference amplitude, the sound features with the largest amplitude are adjusted first, until all adjustments are completed; The data packet connection adjustment unit is used to make a second adjustment to the connection of adjacent data packets after the adjustment is completed, so as to ensure the continuity between the timbre and volume of each data packet.
7. The sound processing system for vocal singing according to claim 1, characterized in that: The sound output module includes a data packet splicing unit and a sound data playing unit; The data packet splicing unit is used to connect the adjusted and connected data packets to restore the complete singing sound, maintain the continuity between the singing sounds, and avoid the phenomenon of mismatch; The sound data playback unit is used to play the singing after the sound processing is completed.
8. A sound processing method for vocal singing according to any one of claims 1 to 7, characterized in that: The specific steps include: Step 1: The vocal singing sound is uniformly recorded by the sound collection unit and the sound recording unit, wherein the sound recording direction of the recording device can be adjusted in real time by the automatic orientation correction unit during the recording process; Step 2: The recorded sound data is divided into several data packets for independent analysis, and marked respectively, such as: A00; A01; A02...Z98; Z99; the data packet includes feature information of N dimensions, where N is a multiple of 1; and the non-standard pronunciations in the divided data packets are captured and marked one by one; Step 3: Analyze the volume, timbre, and vocalization status of each data packet independently, and also analyze the singer's breathing status and pronunciation continuity during the singing process; Step 4: collecting the vocal features of the singer in the analysis process in step 3, wherein the special extraction includes fundamental frequency extraction and formant extraction; Step 5: Compare the extracted sound features with the sample data to obtain difference data, and make targeted adjustments and records according to the range of the difference data; Step 6: Make a second adjustment to the connection points of adjacent data packets after the adjustment is completed to ensure the continuity of the timbre and volume of each data packet; Step 7: Restore the connected data packets to complete continuous sound data, and play or record after maintaining the continuity between the singing sounds.
Citation Information
Patent Citations
Method and device for beautifying sound
CN109410971A
