Vocal music pronunciation training system

Abstract

The invention discloses a vocal music pronunciation training system, and relates to the technical field of vocal music training, and the vocal music pronunciation training system comprises a sound collection module, a sound preprocessing module, a feature extraction module, a model construction module, and a standard pronunciation database module. The system has the advantages that the standard vocal music pronunciation data is pronunciated through the exclusive sound model of the trainee, so that the difference between the current vocal music pronunciation of the trainee and the standard vocal music pronunciation can be visually displayed, and the difference coefficient between the sound of the trainee and the standard pronunciation is analyzed through the comparative analysis module; according to the vocal music pronunciation training method, different types of vocal music pronunciation training schemes are formulated in a personalized manner according to the difference, and the personalized training scheme considers the individual difference between different trainers instead of using the same training scheme by all trainers traditionally, so that the training efficiency is improved. And the pertinence and effectiveness of training are further ensured.

Description

Technical Field

[0001] The invention relates to the technical field of vocal training, in particular to a vocal pronunciation training system. Background Art

[0002] In today's society, vocal pronunciation training has attracted more and more attention and importance. With the continuous improvement of people's pursuit of art and culture, both professional vocal practitioners and amateur music lovers hope to improve their vocal pronunciation level. Vocal pronunciation training has always been an important part of the music field.

[0003] Traditional vocal pronunciation training systems usually provide the same training program for all trainees, without considering individual differences. There is a lack of accurate coefficient of difference analysis, making it difficult to clearly determine the size of the gap and to develop personalized training programs accordingly. This results in insufficient training targeting and effectiveness, affecting the improvement of trainees. Often, multiple people share a set of programs, which may make it difficult for those with weak foundations to keep up, while those with a certain foundation may find it too simple. For this reason, we propose a vocal pronunciation training system. Summary of the invention

[0004] The purpose of the present invention is to provide a vocal pronunciation training system.

[0005] In order to solve the problems raised in the above background technology, the present invention provides the following technical solutions: a vocal pronunciation training system, the vocal pronunciation training system comprising a sound collection module, a sound preprocessing module, a feature extraction module, a model building module, a standard pronunciation database module, a comparative analysis module, an evaluation and diagnosis module, a personalized training program generation module and a training execution module;

[0006] Voice collection module: responsible for collecting the voices of trainees;

[0007] Sound preprocessing module: responsible for preprocessing the collected sound of the trainees;

[0008] Feature extraction module: responsible for extracting key features from preprocessed sound;

[0009] Model building module: responsible for building a unique voice model for trainees based on the feature data extracted by the feature extraction module;

[0010] Standard pronunciation database module: stores a large amount of standard music pronunciation data, covering excellent pronunciation examples of various music styles and types;

[0011] Comparative analysis module: responsible for connecting the model building module and the sound collection module, comparing the differences between the trainee's voice and the standard pronunciation in detail, and calculating the difference coefficient;

[0012] Evaluation and diagnosis module: responsible for evaluating and diagnosing the pronunciation status of trainees based on the calculation results of the comparative analysis module, and generating different training instructions;

[0013] Personalized training program generation module: responsible for generating personalized training programs for trainees based on the evaluation and diagnosis results;

[0014] Training execution module: responsible for assisting trainers to conduct actual training according to the training plan and providing real-time feedback and guidance.

[0015] As a further solution of the present invention: the sound collection module includes a sound feature collection unit and a vocal sound recording unit, the sound feature collection unit includes a NeumannU8 recording microphone, the voice of the trainee is recorded by the NeumannU8 recording microphone, the NeumannU8 recording microphone converts the mechanical energy of the sound wave into an electrical signal through its internal sensor when recording the voice of the trainee, the sound feature collection unit converts the electrical signal into a digital signal through the analog-to-digital converter (ADC) technology, which is convenient for subsequent digital processing and analysis. In the process of sound collection, the sound feature collection unit adopts a signal processing algorithm based on The adaptive noise reduction technology analyzes the characteristics of environmental noise in real time and filters it in a targeted manner to retain only the pure voice of the trainee. At the same time, the multi-microphone array technology is used to accurately locate the sound source and reduce noise interference. The automatic gain control (AGC) technology is used to automatically adjust the gain of the microphone according to the intensity of the sound to ensure the clarity of the sound collected at different volume levels. The vocal sound recording unit is used to record the pronunciation of the training vocal selected by the trainee. The vocal sound recording unit transmits the recorded vocal pronunciation data to the comparison and analysis module through the sound feature collection unit, and the sound feature collection unit transmits the collected trainee sound data to the sound preprocessing module.

[0016] As a further solution of the present invention: the sound preprocessing module is capable of receiving sound data transmitted by the sound acquisition module. The sound preprocessing module first performs spectrum analysis and noise feature extraction on the sound data to identify the frequency range and characteristics of the noise. For continuous background noise, a long-term average filtering method is adopted. For intermittent burst noise, an event-triggered noise reduction process is adopted. High-frequency noise can be removed while retaining sound details. After removing the high-frequency noise, the sound preprocessing module corrects the sound distortion through signal smoothing technology, so that the waveform of the sound signal is smooth and continuous. The sound preprocessing module transmits the preprocessed sound data to the feature extraction module.

[0017] As a further solution of the present invention: the feature extraction module includes a time-frequency analysis unit and a spectrum analysis unit. The time-frequency analysis unit decomposes the sound signal in two dimensions of time and frequency through short-time Fourier transform (STFT) to obtain the time-frequency characteristics of the sound, including the change of pitch over time and the distribution of frequency components. The spectrum analysis unit calculates the spectrum of the sound signal through the fast Fourier transform (FFT) algorithm to obtain the frequency characteristics of the sound, including the main frequency and harmonic characteristics. The spectrum analysis unit extracts the pitch characteristics of the sound signal through the cepstrum analysis algorithm, extracts the duration characteristics of the sound signal through the start and end time of the sound signal, extracts the rhythm characteristics of the sound signal through the periodicity of the sound signal, extracts the timbre characteristics of the sound signal through the spectrum shape and harmonic structure of the sound signal, and extracts the loudness characteristics of the sound signal through the amplitude of the sound signal. The feature extraction module will sort and organize the extracted sound features into the form of feature vectors for subsequent model construction and analysis.

[0018] As a further solution of the present invention: the model construction module selects a convolutional neural network (CNN) as a neural network architecture to construct an exclusive sound model, and uses the sound feature data of the trainee extracted by the feature extraction module as input data for training the exclusive sound model. During the training process, a back propagation algorithm is used to calculate the gradient, and the parameters of the network are updated through the Adagrad optimization algorithm, so that the model can gradually fit the sound features of the trainee. During the training process of the exclusive sound model, the model construction module uses data enhancement technology to randomly transform the original data to increase the diversity of the data, including adding noise and changing the pitch, so as to avoid overfitting of the model, and limits the complexity of the model through the L2 regularization technology to prevent the model from over-learning noise. The model construction module divides the sound features of the trainee into a training set, a verification set and a test set in a ratio of 2:2:1, trains on the training set, evaluates and adjusts the model on the verification set, and finally verifies the performance of the model on the test set, and determines the exclusive sound model with a pass rate of more than 99% on the test set as the final sound model.

[0019] As a further solution of the present invention: the standard pronunciation database module includes a standard pronunciation unit and an Internet unit, the standard pronunciation unit stores standard music pronunciation data for teaching, covering excellent pronunciation examples of various music styles and types, the Internet unit is used to connect to the Internet, which can improve the music search window, help trainers retrieve and download required pronunciation training music, and maintain the timeliness and advancement of the database, the standard pronunciation database module will transfer the vocal data trained by the trainers to the final version of the sound model, and form a standard vocal pronunciation with the same sound characteristics as the trainers through the final version of the sound model, and the standard pronunciation database module will transfer the standard vocal pronunciation data with the same sound characteristics as the trainers to the comparative analysis module.

[0020] As a further solution of the present invention: the comparative analysis module can receive the vocal pronunciation data recorded by the trainee and transmitted by the sound acquisition module and the standard vocal pronunciation data transmitted by the standard pronunciation database module, extract the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data through the fast Fourier transform (FFT), Mel-frequency cepstral coefficient (MFCC) feature extraction method and Librosa, PyAudio and Essentia audio processing library and tools, and calculate the difference coefficient k0 between the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data through a formula. The specific calculation formula of the difference coefficient k0 is as follows:

[0021]

[0022]

[0023] Where: k0 represents the difference coefficient between the vocal pronunciation data recorded by the trainer and the sound data of the standard vocal pronunciation data, S1 i and S0 i Respectively represent the frequency of each band of the recorded vocal pronunciation and the frequency of the standard vocal pronunciation band, L1 i and L0 i Respectively represent the loudness value of each band of the recorded vocal pronunciation and the loudness value of the standard vocal pronunciation band, C1 i and C0 i Respectively represent the duration of each band of the recorded vocal pronunciation and the duration of the standard vocal pronunciation band, B1 i and B0 i They respectively represent the interval duration between adjacent bands of recorded vocal pronunciation and the interval duration between adjacent bands of standard vocal pronunciation, and a1 and a0 respectively represent the number of bands of recorded vocal pronunciation and standard vocal pronunciation;

[0024] The comparison and analysis module transmits the difference coefficient data of the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data to the evaluation and diagnosis module and the personalized training program generation module.

[0025] As a further solution of the present invention: the evaluation and diagnosis module can receive the data transmitted by the comparison and analysis module, and perform classification evaluation according to the value of the difference coefficient k0:

[0026] When k0≤k, it means that the vocal pronunciation recorded by the trainee is slightly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a type A instruction to the personalized training program generation module;

[0027] When k0>k, it means that the vocal pronunciation recorded by the trainee is significantly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a Class B instruction to the personalized training program generation module.

[0028] As a further solution of the present invention: the personalized training program generation module is provided with a display screen, and the personalized training program generation module is capable of receiving instruction data transmitted by the evaluation and diagnosis module;

[0029] When the personalized training program generation module receives the Class A instruction, the personalized training program generation module displays the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data on the display screen, and marks the area where the difference between the standard vocal pronunciation and the recorded vocal pronunciation in each band frequency number, each band loudness value, each band utterance duration, and adjacent band interval duration exceeds 50% in red, marks the area where the difference is between 15% and 50% in yellow, and marks the area where the difference is less than 15% in green;

[0030] When the personalized training program generation module receives the Class B instruction, the personalized training program generation module generates a six-step training program on the display screen, and the training program is divided according to six different dimensions: pitch, duration, rhythm, loudness, vocalization position adjustment, and breath application skills;

[0031] Step 1: Perform demonstration exercises based on the vocal pronunciation recorded by the trainer with correct intonation;

[0032] Step 2: Based on step 1, practice the correct pronunciation duration;

[0033] Step 3: Based on step 2, practice the correct pronunciation rhythm;

[0034] Step 4: Based on step 3, practice the correct pronunciation pitch;

[0035] Step 5: Based on step 4, adjust the correct vocalization position and perform demonstration exercises;

[0036] Step 6: Based on step 5, practice the correct breathing techniques.

[0037] As a further solution of the present invention: the training execution module captures and analyzes the trainee's pronunciation in real time through real-time monitoring technology of audio and video. When the trainee's pronunciation deviates from the expected target, the module will give prompts and correction suggestions. The training execution module also has progress tracking and recording functions. It will record in detail the time, content, completion status and pronunciation performance data of each training of the trainee. By analyzing these data, the effect and progress of the training can be understood, problems can be discovered in time and the training plan can be adjusted.

[0038] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. The present invention provides highly personalized training programs for trainees, collects trainees' voices through a sound collection module, constructs a dedicated voice model for trainees after sound preprocessing and feature extraction, and pronounces standard vocal pronunciation data through the dedicated voice model of the trainees, so that the difference between the trainees' current vocal pronunciation and the standard vocal pronunciation can be intuitively displayed, and the difference coefficient between the trainees' voices and the standard pronunciation can be analyzed through a comparative analysis module, so as to clearly understand the size of the knowledge gap, and formulate different types of vocal pronunciation training programs according to the size of the difference. This personalized training program takes into account the individual differences between different trainees, rather than the traditional method of all trainees using the same training program, which further ensures the pertinence and effectiveness of the training;

[0040] 2. The present invention constructs a sound model dedicated to the trainee through a sound collection module, a sound preprocessing module, a feature extraction module and a model construction module, and imports the labeled vocal data into the trainee's sound model through a standard pronunciation database module, so that a standard vocal pronunciation with the same pitch as the trainee can be generated, so that the trainee can intuitively judge the correct pronunciation method under the same timbre and tone as himself, and then intuitively judge where his voice problems are, which can reduce unnecessary vocal learning and training time and greatly improve the vocal voice efficiency;

[0041] 3. The present invention can intuitively display the training plan and difference analysis results through the display screen of the personalized training plan generation module, so that the trainees can clearly understand their own problems and improvement directions. The real-time monitoring, prompting and correction suggestion functions of the training execution module, as well as the progress tracking and recording functions, enable the trainees to obtain feedback in time and form a good interaction with the system. This convenience and interactivity not only improves the enthusiasm and participation of the trainees, but also makes the training process more interesting and efficient, which helps the trainees maintain a good training state and improve the vocal pronunciation level more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the system flow in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0044] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] Please see attached Figure 1 The present invention provides a vocal pronunciation training system, which includes a sound collection module, a sound preprocessing module, a feature extraction module, a model building module, a standard pronunciation database module, a comparative analysis module, an evaluation and diagnosis module, a personalized training program generation module and a training execution module;

[0046] Voice collection module: responsible for collecting the voices of trainees;

[0047] Sound preprocessing module: responsible for preprocessing the collected sound of the trainees;

[0048] Feature extraction module: responsible for extracting key features from preprocessed sound;

[0049] Model building module: responsible for building a unique voice model for trainees based on the feature data extracted by the feature extraction module;

[0050] Standard pronunciation database module: stores a large amount of standard music pronunciation data, covering excellent pronunciation examples of various music styles and types;

[0051] Comparative analysis module: responsible for connecting the model building module and the sound collection module, comparing the differences between the trainee's voice and the standard pronunciation in detail, and calculating the difference coefficient;

[0052] Evaluation and diagnosis module: responsible for evaluating and diagnosing the pronunciation status of trainees based on the calculation results of the comparative analysis module, and generating different training instructions;

[0053] Personalized training program generation module: responsible for generating personalized training programs for trainees based on the evaluation and diagnosis results;

[0054] Training execution module: responsible for assisting trainers to conduct actual training according to the training plan and providing real-time feedback and guidance.

[0055] In one embodiment of the present invention: the sound collection module includes a sound feature collection unit and a vocal sound recording unit, the sound feature collection unit includes a NeumannU8 recording microphone, the NeumannU8 recording microphone is used to record the voice of the trainee, the NeumannU8 recording microphone converts the mechanical energy of the sound wave into an electrical signal through its internal sensor when recording the voice of the trainee, the sound feature collection unit converts the electrical signal into a digital signal through the analog-to-digital converter (ADC) technology, which is convenient for subsequent digital processing and analysis, and in the sound collection process, the sound feature collection unit adopts a signal processing algorithm based on the signal processing algorithm. Adaptive noise reduction technology analyzes the characteristics of environmental noise in real time and filters it in a targeted manner to retain only the pure voice of the trainees. At the same time, it adopts multi-microphone array technology to accurately locate the sound source and reduce noise interference. It adopts automatic gain control (AGC) technology to automatically adjust the gain of the microphone according to the intensity of the sound to ensure the clarity of the sound collected at different volume levels. The vocal sound recording unit is used to record the pronunciation of the trainee's selected training vocals. The vocal sound recording unit transmits the recorded vocal pronunciation data to the comparison and analysis module through the sound feature collection unit. The sound feature collection unit transmits the collected trainee sound data to the sound preprocessing module.

[0056] In one embodiment of the present invention: the sound preprocessing module is capable of receiving the sound data transmitted by the sound acquisition module. The sound preprocessing module first performs spectrum analysis and noise feature extraction on the sound data to identify the frequency range and characteristics of the noise. For continuous background noise, a long-term average filtering method is adopted. For intermittent burst noise, an event-triggered noise reduction process is adopted. High-frequency noise can be removed while retaining sound details. After removing the high-frequency noise, the sound preprocessing module corrects the sound distortion through signal smoothing technology, so that the waveform of the sound signal is smooth and continuous. The sound preprocessing module transmits the preprocessed sound data to the feature extraction module.

[0057] In one embodiment of the present invention: the feature extraction module includes a time-frequency analysis unit and a spectrum analysis unit. The time-frequency analysis unit decomposes the sound signal in two dimensions of time and frequency through short-time Fourier transform (STFT) to obtain the time-frequency characteristics of the sound, including the change of pitch over time and the distribution of frequency components. The spectrum analysis unit calculates the spectrum of the sound signal through the fast Fourier transform (FFT) algorithm to obtain the frequency characteristics of the sound, including the main frequency and harmonic characteristics. The spectrum analysis unit extracts the pitch characteristics of the sound signal through the cepstrum analysis algorithm, extracts the duration characteristics of the sound signal through the start and end time of the sound signal, extracts the rhythm characteristics of the sound signal through the periodicity of the sound signal, extracts the timbre characteristics of the sound signal through the spectrum shape and harmonic structure of the sound signal, and extracts the loudness characteristics of the sound signal through the amplitude of the sound signal. The feature extraction module will sort and organize the extracted sound features into the form of feature vectors for subsequent model construction and analysis.

[0058] In one embodiment of the present invention: the model building module selects a convolutional neural network (CNN) as the neural network architecture to build an exclusive sound model, and uses the sound feature data of the trainee extracted by the feature extraction module as input data for training the exclusive sound model. During the training process, the back propagation algorithm is used to calculate the gradient, and the network parameters are updated through the Adagrad optimization algorithm, so that the model can gradually fit the sound features of the trainee. During the training process of the exclusive sound model, the model building module uses data enhancement technology to randomly transform the original data to increase the diversity of the data, including adding noise and changing the pitch, so as to avoid overfitting of the model, and limits the complexity of the model through the L2 regularization technology to prevent the model from over-learning noise. The model building module divides the sound features of the trainee into a training set, a verification set and a test set in a ratio of 2:2:1, trains on the training set, evaluates and adjusts the model on the verification set, and finally verifies the performance of the model on the test set, and determines the exclusive sound model with a pass rate of more than 99% on the test set as the final sound model.

[0059] In one embodiment of the present invention: the standard pronunciation database module includes a standard pronunciation unit and an Internet unit. The standard pronunciation unit stores standard music pronunciation data for teaching, covering excellent pronunciation examples of various music styles and types. The Internet unit is used to connect to the Internet, which can improve the music search window, help trainers retrieve and download required pronunciation training music, and maintain the timeliness and advancement of the database. The standard pronunciation database module will transfer the vocal data trained by the trainers to the final version of the sound model, and form a standard vocal pronunciation with the same sound characteristics as the trainers through the final version of the sound model. The standard pronunciation database module will transfer the standard vocal pronunciation data with the same sound characteristics as the trainers to the comparative analysis module.

[0060] In one embodiment of the present invention: the comparative analysis module can receive the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data transmitted by the sound acquisition module, and extract the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data through the fast Fourier transform (FFT), Mel-frequency cepstral coefficient (MFCC) feature extraction method and Librosa, PyAudio and Essentia audio processing library and tools, and calculate the difference coefficient k0 between the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data through the formula. The specific calculation formula of the difference coefficient k0 is as follows:

[0061]

[0062]

[0063] Where: k0 represents the difference coefficient between the vocal pronunciation data recorded by the trainer and the sound data of the standard vocal pronunciation data, S1 i and S0 i Respectively represent the frequency of each band of the recorded vocal pronunciation and the frequency of the standard vocal pronunciation band, L1 i and L0 i Respectively represent the loudness value of each band of the recorded vocal pronunciation and the loudness value of the standard vocal pronunciation band, C1 i and C0 i Respectively represent the duration of each band of the recorded vocal pronunciation and the duration of the standard vocal pronunciation band, B1 i and B0 i They respectively represent the interval duration between adjacent bands of recorded vocal pronunciation and the interval duration between adjacent bands of standard vocal pronunciation, and a1 and a0 respectively represent the number of bands of recorded vocal pronunciation and standard vocal pronunciation;

[0064] The comparative analysis module transmits the difference coefficient data between the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data to the evaluation and diagnosis module and the personalized training program generation module.

[0065] In one embodiment of the present invention: the evaluation and diagnosis module can receive the data transmitted by the comparison and analysis module, and perform classification evaluation according to the value of the difference coefficient k0:

[0066] When k0≤k, it means that the vocal pronunciation recorded by the trainee is slightly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a type A instruction to the personalized training program generation module;

[0067] When k0>k, it means that the vocal pronunciation recorded by the trainee is significantly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a Class B instruction to the personalized training program generation module.

[0068] In one embodiment of the present invention: the personalized training program generation module is provided with a display screen, and the personalized training program generation module is capable of receiving instruction data transmitted by the evaluation and diagnosis module;

[0069] When the personalized training program generation module receives the Class A instruction, the personalized training program generation module displays the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data on the display screen, and marks the area where the difference between the standard vocal pronunciation and the recorded vocal pronunciation in each band frequency number, each band loudness value, each band utterance duration, and adjacent band interval duration exceeds 50% in red, marks the area where the difference is between 15% and 50% in yellow, and marks the area where the difference is less than 15% in green;

[0070] When the personalized training program generation module receives the Class B instruction, the personalized training program generation module generates a six-step training program on the display screen, and the training program is divided according to six different dimensions: pitch, duration, rhythm, loudness, vocalization position adjustment, and breath application skills;

[0071] Step 1: Perform demonstration exercises based on the vocal pronunciation recorded by the trainer with correct intonation;

[0072] Step 2: Based on step 1, practice the correct pronunciation duration;

[0073] Step 3: Based on step 2, practice the correct pronunciation rhythm;

[0074] Step 4: Based on step 3, practice the correct pronunciation pitch;

[0075] Step 5: Based on step 4, adjust the correct vocalization position and perform demonstration practice;

[0076] Step 6: Based on step 5, practice the correct breathing techniques.

[0077] In one embodiment of the present invention: the training execution module captures and analyzes the trainee's pronunciation in real time through real-time monitoring technology of audio and video. When the trainee's pronunciation deviates from the expected target, the module will give prompts and correction suggestions. The training execution module also has progress tracking and recording functions. It will record in detail the time, content, completion status and pronunciation performance data of each training of the trainee. By analyzing these data, the effect and progress of the training can be understood, problems can be discovered in time and the training plan can be adjusted.

[0078] Example 1, please refer to the attached Figure 1 The model building module takes the trainee’s voice feature data extracted by the feature extraction module as input and performs gradient calculation through the back propagation algorithm, which is the core mechanism of model learning and optimization. In each training iteration, the back propagation algorithm can back propagate the error signal according to the error between the model’s predicted output and the actual target, and calculate the gradient of each parameter. These gradient information indicate how the parameters should be adjusted to reduce the error, so that the model can gradually fit the trainee’s voice features.

[0079] Example 2, please refer to the attached Figure 1 In the process of extracting sound data, the comparative analysis module not only uses the classic feature extraction methods of Fast Fourier Transform (FFT) and Mel Frequency Cepstral Coefficient (MFCC), but also combines the advanced audio processing libraries and tools of Librosa, PyAud io and Essent ia to ensure that the most representative and distinguishing features are extracted from the sound signal. FFT can convert the sound signal from the time domain to the frequency domain, and clearly display the frequency components and energy distribution of the sound, while MFCC simulates the perception characteristics of the human auditory system to the sound, and pays more attention to the features related to speech recognition and audio analysis. Librosa library provides a series of powerful audio processing functions, which can easily perform audio reading, segmentation, resampling and feature extraction operations. PyAud io provides support for real-time audio input and output, so that the module can interact efficiently with external audio devices. As a comprehensive audio analysis framework, Essent ia contains a wealth of audio feature extraction algorithms and audio processing tools, which further improves the accuracy and comprehensiveness of comparative analysis.

[0080] Example 3, please refer to the attached Figure 1In the evaluation and diagnosis module, when the difference coefficient shows that the vocal pronunciation recorded by the trainee is slightly different from the standard pronunciation, the evaluation and diagnosis module will issue a Class A instruction to the personalized training program generation module, which means that the trainee has a good pronunciation foundation overall, but there may still be some local and subtle problems that need to be further optimized and improved. For example, there may be slight unnatural transitions in some specific pitch conversions, or the grasp of some complex rhythms may not be accurate enough. For these situations, the personalized training program generation module can formulate targeted reinforcement exercises to help trainees further improve the delicacy and expressiveness of pronunciation.

[0081] Specifically, the present invention provides highly personalized training programs for trainees, collects their voices with the help of a sound collection module, builds an exclusive sound model after sound preprocessing and feature extraction, and then sounds the standard vocal pronunciation data through the exclusive model, which can intuitively present the difference between the trainee's current vocal pronunciation and the standard vocal pronunciation, and obtains the difference coefficient through the comparison and analysis module, so as to clearly know the degree of the difference, and formulates vocal pronunciation training programs that vary from person to person accordingly, abandoning the traditional general training program, and effectively ensuring the pertinence and effectiveness of the training.

[0082] Specifically, the present invention utilizes a sound collection module, a sound preprocessing module, a feature extraction module, and a model construction module to create an exclusive sound model for trainees, and imports standard vocal data into it through a standard pronunciation database module to generate a standard vocal pronunciation with the same pitch as the trainees. This allows trainees to intuitively determine the correct pronunciation method under the same timbre and tone, quickly identify vocal problems, effectively reduce unnecessary vocal learning and training time, and greatly improve vocal sounding efficiency.

[0083] Specifically, the display screen equipped in the personalized training program generation module of the present invention can intuitively display the training program and difference analysis results, so that the trainees can clearly understand their own problems and improvement directions. The training execution module has the functions of real-time monitoring, prompting, correction suggestions, progress tracking and recording, so that the trainees can obtain feedback in time and interact well with the system. This convenience and interactivity can not only enhance the enthusiasm and participation of the trainees, but also make the training process more interesting and efficient, which helps the trainees maintain a good training state and accelerate the improvement of vocal pronunciation level.

[0084] Working principle:

[0085] First, the voice of the trainee is collected through the sound feature collection unit and the vocal recording unit in the sound collection module. The sound feature collection unit adopts a variety of technologies to ensure that pure and clear sound is collected. The collected sound is transmitted to the sound preprocessing module. This module first performs spectrum analysis and noise feature extraction, and then adopts different methods to remove noise and correct distortion. The preprocessed sound data is transmitted to the feature extraction module. The time-frequency analysis unit and spectrum analysis unit of this module extract the time-frequency, spectrum, pitch, duration, rhythm, timbre, loudness and other features of the sound through a variety of algorithms and technologies, and organize them into feature vector form. Then, the model construction module uses the convolutional neural network as the architecture and uses the extracted feature data to train the exclusive sound model. A variety of technologies are used in the training to avoid overfitting and limit the complexity of the model, and the training set, validation set and test set are divided to verify the model performance. The standard pronunciation database module stores a large number of standard The quasi-music pronunciation data can be connected to the Internet for update. The vocal data trained by the trainees is transmitted to the determined final version of the sound model to form a standard vocal pronunciation with the same characteristics. At the same time, the standard vocal pronunciation data is transmitted to the comparative analysis module. The comparative analysis module receives the vocal pronunciation data of the trainees and the standard, extracts the data and calculates the difference coefficient through relevant methods and tools, and transmits it to the evaluation and diagnosis module and the personalized training program generation module. The evaluation and diagnosis module classifies and evaluates the difference coefficient according to the numerical value, and issues different instructions to the personalized training program generation module. The personalized training program generation module displays the corresponding content on the display screen or generates a specific training program according to the instructions. Finally, the training execution module captures and analyzes the trainees' vocalization through audio and video real-time monitoring technology, gives prompts and correction suggestions, and also has progress tracking and recording functions, so as to understand the training effect and progress, and adjust the program in time. At this point, the entire workflow ends.

[0086] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0087] Some aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes ...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs ...), smart cards, and flash memory devices (e.g., cards, sticks, key drives ...).

[0088] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or a combination of any of the above mediums.

[0089] Similarly, it should be noted that in order to simplify the description disclosed in this application, so as to help understand one or more embodiments of the invention, in the above description of the embodiments of the present application, sometimes multiple features are merged into one embodiment, drawings or descriptions thereof. However, this disclosure method does not mean that the features required by the object of the present application are more than the features mentioned in the claims. In fact, the features of the embodiment are less than all the features of the single embodiment disclosed above. In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "approximately", "approximately" or "substantially" in some examples. Unless otherwise specified, "approximately", "approximately" or "substantially" indicate that the number allows a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining the digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, such numerical settings are as accurate as possible within the feasible range.

[0090] Although the present invention is disclosed as above in terms of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection defined by the claims of the present invention.

Claims

1. A vocal pronunciation training system, characterized in that: The vocal pronunciation training system includes a sound collection module, a sound preprocessing module, a feature extraction module, a model building module, a standard pronunciation database module, a comparison analysis module, an evaluation and diagnosis module, a personalized training program generation module and a training execution module; Voice collection module: responsible for collecting the voices of trainees; Sound preprocessing module: responsible for preprocessing the collected sound of the trainees; Feature extraction module: responsible for extracting key features from preprocessed sound; Model building module: responsible for building a unique voice model for trainees based on the feature data extracted by the feature extraction module; Standard pronunciation database module: stores a large amount of standard music pronunciation data, covering excellent pronunciation examples of various music styles and types; Comparative analysis module: responsible for connecting the model building module and the sound collection module, comparing the differences between the trainee's voice and the standard pronunciation in detail, and calculating the difference coefficient; Evaluation and diagnosis module: responsible for evaluating and diagnosing the pronunciation status of trainees based on the calculation results of the comparative analysis module, and generating different training instructions; Personalized training program generation module: responsible for generating personalized training programs for trainees based on the evaluation and diagnosis results; Training execution module: responsible for assisting trainers to conduct actual training according to the training plan and providing real-time feedback and guidance.

2. A vocal pronunciation training system according to claim 1, characterized in that: The sound collection module includes a sound feature collection unit and a vocal recording unit. The sound feature collection unit includes a NeumannU8 recording microphone. The voice of the trainee is recorded by the NeumannU8 recording microphone. When recording the voice of the trainee, the NeumannU8 recording microphone converts the mechanical energy of the sound wave into an electrical signal through its internal sensor. The sound feature collection unit converts the electrical signal into a digital signal through the analog-to-digital converter (ADC) technology to facilitate subsequent digital processing and analysis. During the sound collection process, the sound feature collection unit adopts adaptive noise reduction based on a signal processing algorithm. Technology is used to analyze the characteristics of environmental noise in real time and filter it in a targeted manner to retain only the pure voice of the trainee. At the same time, multi-microphone array technology is used to accurately locate the sound source and reduce noise interference. Automatic gain control (AGC) technology is used to automatically adjust the gain of the microphone according to the intensity of the sound to ensure the clarity of the sound collected at different volume levels. The vocal sound recording unit is used to record the pronunciation of the training vocal selected by the trainee. The vocal sound recording unit transmits the recorded vocal pronunciation data to the comparison and analysis module through the sound feature collection unit. The sound feature collection unit transmits the collected trainee sound data to the sound preprocessing module.

3. A vocal pronunciation training system according to claim 1, characterized in that: The sound preprocessing module can receive the sound data transmitted by the sound acquisition module. The sound preprocessing module first performs spectrum analysis and noise feature extraction on the sound data to identify the frequency range and characteristics of the noise. For continuous background noise, a long-term average filtering method is adopted. For intermittent burst noise, an event-triggered noise reduction process is adopted. It can remove high-frequency noise while retaining sound details. After removing the high-frequency noise, the sound preprocessing module corrects the sound distortion through signal smoothing technology, so that the waveform of the sound signal is smooth and continuous. The sound preprocessing module will transmit the preprocessed sound data to the feature extraction module.

4. A vocal pronunciation training system according to claim 1, characterized in that: The feature extraction module includes a time-frequency analysis unit and a spectrum analysis unit. The time-frequency analysis unit decomposes the sound signal in two dimensions of time and frequency through short-time Fourier transform (STFT) to obtain the time-frequency characteristics of the sound, including the change of pitch over time and the distribution of frequency components. The spectrum analysis unit calculates the spectrum of the sound signal through the fast Fourier transform (FFT) algorithm to obtain the frequency characteristics of the sound, including the main frequency and harmonic characteristics. The spectrum analysis unit extracts the pitch characteristics of the sound signal through the cepstrum analysis algorithm, extracts the duration characteristics of the sound signal through the start and end time of the sound signal, extracts the rhythm characteristics of the sound signal through the periodicity of the sound signal, extracts the timbre characteristics of the sound signal through the spectrum shape and harmonic structure of the sound signal, and extracts the loudness characteristics of the sound signal through the amplitude of the sound signal. The feature extraction module will sort and organize the extracted sound features into the form of feature vectors for subsequent model construction and analysis.

5. A vocal pronunciation training system according to claim 1, characterized in that: The model construction module selects a convolutional neural network (CNN) as a neural network architecture to construct an exclusive sound model, and uses the sound feature data of the trainee extracted by the feature extraction module as input data for training the exclusive sound model. During the training process, a back propagation algorithm is used to calculate the gradient, and the parameters of the network are updated through the Adagrad optimization algorithm, so that the model can gradually fit the sound features of the trainee. During the training process of the exclusive sound model, the model construction module uses data enhancement technology to randomly transform the original data to increase the diversity of the data, including adding noise and changing the pitch, so as to avoid overfitting of the model, and limits the complexity of the model through the L2 regularization technology to prevent the model from over-learning noise. The model construction module divides the sound features of the trainee into a training set, a verification set and a test set in a ratio of 2:2:1, trains on the training set, evaluates and adjusts the model on the verification set, and finally verifies the performance of the model on the test set, and determines the exclusive sound model with a pass rate of more than 99% on the test set as the final sound model.

6. A vocal pronunciation training system according to claim 1, characterized in that: The standard pronunciation database module includes a standard pronunciation unit and an Internet unit. The standard pronunciation unit stores standard music pronunciation data for teaching, covering excellent pronunciation examples of various music styles and types. The Internet unit is used to connect to the Internet, which can improve the music search window, help trainers retrieve and download required pronunciation training music, and maintain the timeliness and advancement of the database. The standard pronunciation database module will transfer the vocal data trained by the trainers to the final version of the sound model, and form a standard vocal pronunciation with the same sound characteristics as the trainers through the final version of the sound model. The standard pronunciation database module will transfer the standard vocal pronunciation data with the same sound characteristics as the trainers to the comparison and analysis module.

7. A vocal pronunciation training system according to claim 1, characterized in that: The comparative analysis module can receive the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data transmitted by the sound acquisition module, and extract the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data through the fast Fourier transform (FFT), Mel frequency cepstral coefficient (MFCC) feature extraction method and Librosa, PyAudio and Essentia audio processing library and tool, and calculate the difference coefficient k0 between the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data through the formula. The specific calculation formula of the difference coefficient k0 is as follows: Where: k0 represents the difference coefficient between the vocal pronunciation data recorded by the trainer and the sound data of the standard vocal pronunciation data, S1 i and S0 i Respectively represent the frequency of each band of the recorded vocal pronunciation and the frequency of the standard vocal pronunciation band, L1 i and L0 i Respectively represent the loudness value of each band of the recorded vocal pronunciation and the loudness value of the standard vocal pronunciation band, C1 i and C0 i Respectively represent the duration of each band of the recorded vocal pronunciation and the duration of the standard vocal pronunciation band, B1 i and B0 i They respectively represent the interval duration between adjacent bands of recorded vocal pronunciation and the interval duration between adjacent bands of standard vocal pronunciation, and a1 and a0 respectively represent the number of bands of recorded vocal pronunciation and standard vocal pronunciation; The comparison and analysis module transmits the difference coefficient data of the vocal pronunciation data recorded by the trainee and the sound data of the standard vocal pronunciation data to the evaluation and diagnosis module and the personalized training program generation module.

8. A vocal pronunciation training system according to claim 1, characterized in that: The evaluation and diagnosis module can receive the data transmitted by the comparison and analysis module, and perform classification evaluation according to the value of the difference coefficient k0: When k0≤k, it means that the vocal pronunciation recorded by the trainee is slightly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a type A instruction to the personalized training program generation module; When k0>k, it means that the vocal pronunciation recorded by the trainee is significantly different from the standard vocal pronunciation, and the evaluation and diagnosis module issues a Class B instruction to the personalized training program generation module.

9. A vocal pronunciation training system according to claim 1, characterized in that: The personalized training program generation module is provided with a display screen, and the personalized training program generation module is capable of receiving instruction data transmitted by the evaluation and diagnosis module; When the personalized training program generation module receives the Class A instruction, the personalized training program generation module displays the sound data of the vocal pronunciation data recorded by the trainee and the standard vocal pronunciation data on the display screen, and marks the area where the difference between the standard vocal pronunciation and the recorded vocal pronunciation in each band frequency number, each band loudness value, each band utterance duration, and adjacent band interval duration exceeds 50% in red, marks the area where the difference is between 15% and 50% in yellow, and marks the area where the difference is less than 15% in green; When the personalized training program generation module receives the Class B instruction, the personalized training program generation module generates a six-step training program on the display screen, and the training program is divided according to six different dimensions: pitch, duration, rhythm, loudness, vocalization position adjustment, and breath application skills; Step 1: Perform demonstration exercises based on the vocal pronunciation recorded by the trainer with correct intonation; Step 2: Based on step 1, practice the correct pronunciation duration; Step 3: Based on step 2, practice the correct pronunciation rhythm; Step 4: Based on step 3, practice the correct pronunciation pitch; Step 5: Based on step 4, adjust the correct vocalization position and perform demonstration exercises; Step 6: Based on step 5, practice the correct breathing techniques.

10. A vocal pronunciation training system according to claim 1, characterized in that: The training execution module uses real-time audio and video monitoring technology to capture and analyze the trainee's pronunciation in real time. When the trainee's pronunciation deviates from the expected target, the module will give prompts and correction suggestions. The training execution module also has progress tracking and recording functions, which will record the trainee's progress in detail. The time, content, completion and pronunciation performance data of each training session are analyzed. You can understand the effect and progress of training, identify problems in time and adjust the training plan.

Images