Sound model generation method and system for vocal sound production training

Abstract

The application relates to the technical field of voice recognition, in particular to a sound model generation method and system for vocal singing pronunciation training, which comprises the following steps: collecting sound signals of a trainer for a preset number of different contents, and collecting a preset number of sound signals for each content; for any modal component of any content and any sound signal, acquiring each contrast component of the any modal component, and acquiring a feature difference array of the any modal component; acquiring a difference feature value between the any modal component and each contrast component thereof, acquiring an influence feature value and a reconstruction weight of the any modal component, and then obtaining each reconstructed sound signal; obtaining a sound feature map of each sound signal of the trainer, training a sound model, and evaluating the quality of a new pronunciation of the trainer. The application aims to improve the processing effect of the collected sound signals and improve the precision of the constructed vocal singing pronunciation training sound model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a sound model generation method and system for vocal performance training. BACKGROUND

[0002] In the field of vocal learning, the accuracy of pronunciation is the core standard for measuring singing level. Building a precise vocal pronunciation training sound model can convert abstract vocal skills, such as pitch control, tone optimization, and breath stability, into quantifiable and analyzable objective indicators, providing real-time and scientific feedback for trainers. This not only breaks down the time and space constraints, allowing trainers to quickly identify pronunciation defects such as pitch deviation and abnormal volume fluctuations, but also generates personalized training programs based on the vocal characteristics of different trainers, significantly improving training efficiency and effectiveness.

[0003] However, in the actual construction process of the sound model, the collection of the trainer's sound signal is easily affected by various interference factors. For example, environmental steady-state noise such as device current sound can be superimposed on the effective sound signal in different frequency bands, causing distortion of the frequency spectrum characteristics, interfering with the fundamental frequency detection, and causing deviations in pitch extraction. At the same time, the trainer's own non- pronunciation signals, such as breathing sounds, swallowing sounds, and pronunciation errors, such as broken notes and air leaks, can form sudden interference, disrupting the continuity of the sound characteristics, resulting in deviations in the extracted key features such as pitch, tone, and volume from the actual pronunciation. Traditional preprocessing methods do not fully consider the differences in interference caused by different frequency bands during the collection of sound signals, which affects the accuracy of the sound model in judging the differences between standard pronunciation and trainer pronunciation, and reduces the accuracy of the construction of the sound model for vocal pronunciation training. SUMMARY

[0004] In view of the above, it is necessary to provide a sound model generation method and system for vocal pronunciation training. Compared with the traditional sound model generation method and system for vocal pronunciation training, the processing effect of the collected sound signal is improved, and the accuracy of the constructed vocal pronunciation training sound model is improved:

[0005] In a first aspect, the embodiments of the present application provide a sound model generation method for vocal pronunciation training, which comprises the following steps:

[0006] Collecting sound signals of a predetermined number of different contents from a trainer, wherein a predetermined number of sound signals of a predetermined duration are collected for each content;

[0007] Each modal component of each sound signal is acquired and numbered according to frequency magnitude. For any content and any modal component of any sound signal, modal components with the same number in each other sound signal are taken as reference components. The characteristic difference array of the modal component is obtained by the peak difference between the modal component and its reference components in the frequency domain. The difference feature value between the modal component and its reference components is obtained by the amplitude deviation and feature difference array deviation in the frequency domain. The difference feature value corresponding to the modal component is compared with the difference feature value corresponding to its reference components to obtain the influence feature value of the modal component. Then, the reconstruction weight of the modal component is obtained. Finally, all modal components of each sound signal are reconstructed to obtain each reconstructed sound signal.

[0008] Pitch, timbre, and volume are extracted from each reconstructed sound signal to obtain the sound feature map of each sound signal of the trainee. This map is used to train the sound model together with the standard sound feature map of each sound signal. Then, the sound feature map extracted from the new sound signal of the trainee is processed to evaluate the quality of the trainee's new pronunciation.

[0009] In one embodiment, the process of obtaining the feature difference array is as follows:

[0010] Obtain the fitting curve of all peak values ​​of each modal component in the frequency domain, calculate the mean of the difference values ​​of the ordinates under the same abscissa on the fitting curve between any modal component and each of its control components, and count the maximum value among the difference values ​​of the ordinates under the same abscissa on the fitting curve between any modal component and each of its control components.

[0011] The feature difference array of any modal component is formed by averaging the mean values ​​and the maximum values ​​between any modal component and all its control components.

[0012] In one embodiment, the process of obtaining the difference feature values ​​is as follows:

[0013] Calculate the difference in amplitude in the frequency domain between any modal component and each of its control components;

[0014] Calculate the difference in the feature difference array between any modal component and each of its control components;

[0015] The difference feature value can be further obtained by combining the gap and the difference amount.

[0016] In one embodiment, the difference feature value is the product of the gap and the difference amount.

[0017] In one embodiment, the process of obtaining the influencing feature value is as follows:

[0018] The DTW distance between the sequence of all differential feature values ​​corresponding to any modal component and the sequence of all differential feature values ​​corresponding to each of its control components is used as the feature difference coefficient between any modal component and its control components.

[0019] Obtain the calculation result of multiplying the feature difference coefficient and the difference feature value; obtain the influence feature value of any modal component through the calculation result of the multiplication.

[0020] In one embodiment, the influence feature value is the average of the calculated result of the multiplication between any modal component and all its control components.

[0021] In one embodiment, the expression for the reconstructed weights is:

[0022] ; The reconstruction weight represents the i-th modal component of the x-th sound signal; , represents the normalized values ​​of the influence eigenvalues ​​of the i-th and v-th modal components of the x-th sound signal, respectively; n represents the number of modal components of the x-th sound signal.

[0023] In one embodiment, the step of extracting pitch, timbre, and volume from each reconstructed sound signal to obtain a sound feature map of each sound signal of the trainee includes:

[0024] A fixed-length time window is preset. Within each time window, the autocorrelation function of the reconstructed sound signal is calculated. The reciprocal of the lag time when the autocorrelation function value is maximum is taken as the fundamental frequency. The pitch characteristics are determined by the fundamental frequency.

[0025] The sound quality characteristics are reflected by calculating the Mel frequency cepstral coefficients of the reconstructed sound signal within each time window.

[0026] The average amplitude of the reconstructed audio signal in the time domain within each time window is used as the volume feature within each time window;

[0027] For each reconstructed sound signal, the pitch, timbre, and volume features extracted from all time windows are arranged in chronological order to create a sound feature map corresponding to each reconstructed sound signal.

[0028] In one embodiment, the input to the training sound model is the sound feature map of the trainer's sound signal, and the target output is the standard sound feature map of the trainer's sound signal.

[0029] Secondly, embodiments of this application also provide a sound model generation system for vocal pronunciation training, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any of the above-described sound model generation methods for vocal pronunciation training.

[0030] This application has at least the following beneficial effects:

[0031] This application addresses the issue that, during the construction of sound models for vocal training, the actual acquired sound signals are often interfered with by steady-state environmental noise and sudden noise from the trainee, and these interferences exhibit significant differences in their impact across different frequency bands. Therefore, this application first acquires sound signals multiple times for the same training content. For sound signals with the same content, modal components are extracted, and the sound signals are decomposed according to frequency ranges. Differences between signals in different frequency ranges are compared and analyzed, fully considering the distribution variations of sound signals caused by interference within different frequency intervals. Subsequently, interference features of each sound signal are extracted within each frequency range. Based on these interference features, reasonable reconstruction weights are assigned to the modal components to optimize the quality of the reconstructed sound signals, thereby reducing the impact of steady-state environmental interference and multi-frequency interference caused by the trainee's own factors during sound signal acquisition. The beneficial effect lies in fully combining the interference features of different frequency bands under actual acquisition conditions to achieve accurate preprocessing of sound signals, improving the accuracy of feature extraction for different content sound signals, and thus enhancing the precision of the constructed vocal training sound model. Attached Figure Description

[0032] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating the steps of a sound model generation method for vocal pronunciation training provided in one embodiment of this application;

[0034] Figure 2 A schematic diagram illustrating the process of obtaining reconstructed weights;

[0035] Figure 3 A schematic diagram of the process for acquiring reconstructed sound signals. Detailed Implementation

[0036] In the description of the embodiments in this application, the words "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary," "or," and "for example" is intended to present the relevant concepts in a specific manner.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or".

[0038] It should also be noted that the terms "first" and "second" in this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0039] The following description, in conjunction with the accompanying drawings, details the specific scheme of the sound model generation method and system for vocal pronunciation training provided in this application.

[0040] Please see Figure 1 The diagram illustrates a flowchart of a sound model generation method for vocal pronunciation training according to an embodiment of this application. The method includes the following steps:

[0041] Step 1: Collect audio signals from the trainee for a preset number of different contents, where a preset number of audio signals of a preset duration are collected for each content.

[0042] Audio signal acquisition is a fundamental step in generating a sound model, and its quality directly affects the accuracy of subsequent processing. Specifically, a high-sensitivity condenser microphone is selected, with a sampling rate of 44.1kHz or higher and a bit depth of 16 bits to ensure complete capture of sound details. During the acquisition process, the trainee is guided to pronounce words according to preset training content, such as singing scales of different pitches or specific song excerpts. The duration of each acquisition session is determined based on the training content, ranging from 3 to 5 minutes. The same content is repeatedly acquired 3 to 5 times to reduce the impact of accidental factors during actual acquisition. In this application, the above process is used to acquire audio signals from the trainee for a preset number of different contents, with each content's audio signal acquisition data volume being 5 and the duration being 5 minutes.

[0043] In this embodiment, the preset quantity is 100. The preset quantity is preset by a person and the implementer can set it according to the actual situation. This application does not impose any special restrictions.

[0044] Step 2: Perform difference analysis on different audio signals of the same content in different frequency bands, and reconstruct each audio signal based on the difference analysis results.

[0045] After the above-mentioned data collection, the audio signals of the trainees during their practice were obtained. Although soundproofing materials were used to minimize environmental noise interference during the actual data collection process, the training was still affected by stable environmental noise and noise generated by the trainees themselves. This resulted in the collected audio signals potentially containing low-frequency uniformly distributed noise and high-frequency burst noise interference, affecting the accurate extraction of the trainees' audio signal features. Therefore, this application conducts a comparative analysis of each frequency range of the acquired audio signals from the trainees for different content. Based on the analysis results, the interference features of the audio signals for different content are extracted, and then the acquired audio signals are screened and preprocessed for optimization and adjustment to reduce the impact of interference from different noise sources on the quality of the acquired audio signals. The specific analysis and processing process is as follows:

[0046] Step 2.1: Obtain each modal component of each sound signal and number them according to frequency magnitude. For any content and any modal component of any sound signal, take the modal components with the same number in each other sound signal as each reference component. Obtain the feature difference array of any modal component by the peak value difference between the modal component and each reference component in the frequency domain. Obtain the difference feature value between the modal component and each reference component by the amplitude deviation and feature difference array deviation between the modal component and each reference component in the frequency domain.

[0047] First, considering the actual acquisition process of sound signals, the influence of different interference sources usually manifests as differences in interference effects across different frequency ranges. For example, stable environmental noise exhibits uniform amplitude fluctuations in the low-frequency range, while interference noise caused by the trainee exhibits single-frequency spikes in higher or lower frequency ranges. Based on the above analysis, each modal component of each sound signal is obtained and numbered according to its frequency magnitude. Simultaneously, Fourier transform is used to convert each modal component to the frequency domain. The Fourier transform is a well-known technique and will not be elaborated upon in this application.

[0048] In this embodiment, variational mode decomposition is used to obtain the modal components of each sound signal. The number of modal components of each sound signal is 5. Variational mode decomposition is a well-known technology and will not be described in detail in this application. As other implementation methods, based on the ability to obtain the modal components of each sound signal, the implementer may use other existing feasible technologies. This application does not impose any special restrictions.

[0049] Furthermore, by combining the signal characteristic differences of different audio signals in each content at each frequency range, the data change characteristics of the audio signals of each content caused by the differences in interference sources are analyzed. Taking the i-th modal component of the x-th audio signal of any content as an example, since the overall frequency range of the audio signals collected for the same content is roughly similar, and the output is arranged from low frequency to high frequency during the modal decomposition process, the modal components with the same number of different audio signals of the same content have similar frequency ranges. Therefore, the i-th modal component of each of the remaining audio signals of the aforementioned content is taken as the reference component of the i-th modal component of the x-th audio signal, and then the interference differences of different audio signals in the same frequency range are compared and analyzed.

[0050] Obtain the fitting curves of all peak values ​​for each modal component in the frequency domain. Calculate the mean of the differences in the ordinates of all the same horizontal axis on the fitting curves between the i-th modal component of the x-th sound signal and its control components. Calculate the maximum value among the differences in the ordinates of all the same horizontal axis on the fitting curves between the i-th modal component of the x-th sound signal and its control components. Combine the average of the mean values ​​and the average of the maximum values ​​of the i-th modal component of the x-th sound signal with a feature difference array for the i-th modal component of the x-th sound signal. The larger the average of the mean values ​​and the larger the average of the maximum values, the more significant the interference affecting the sound signal within the frequency range corresponding to the i-th modal component of the x-th sound signal, resulting in greater differences in data distribution.

[0051] In this embodiment, the AMPD (Automatic Multiscale-based Peak Detection) algorithm is used to obtain the peak value of each modal component in the frequency domain. The AMPD algorithm is a well-known technology and will not be described in detail in this application. As other implementation methods, based on the ability to obtain the peak value of each modal component in the frequency domain, implementers may use other existing technologies, such as peak and valley detection algorithms, extreme point detection algorithms, etc. This application does not impose any special restrictions.

[0052] In this embodiment, the least squares method is used to obtain the fitted curve. The least squares method is a well-known technique and will not be described in detail in this application. As other implementation methods, based on the ability to obtain the fitted curve, the implementer may use other existing techniques, such as local weighted regression, K-nearest neighbor regression, etc. This application does not impose any special restrictions.

[0053] In this embodiment, the difference between the vertical coordinates is the absolute value of the difference.

[0054] Furthermore, the difference in amplitude in the frequency domain between the i-th modal component of the x-th sound signal and its respective control components is denoted as the first characteristic value between the i-th modal component of the x-th sound signal and its respective control components. The calculation process for this difference is as follows: all amplitudes of each modal component in the frequency domain are arranged in ascending order of frequency to form an amplitude sequence for each modal component. The distance between the amplitude sequences of the i-th modal component of the x-th sound signal and its respective control components is calculated. In this embodiment, the distance between the amplitude sequences is the DTW (Dynamic Time Warping) distance. The calculation of the DTW distance is a well-known technique and will not be elaborated upon here. As other implementations, based on the ability to measure the degree of difference between elements in two amplitude sequences, the implementer may use other existing techniques, such as Euclidean distance, etc. This application does not impose any special limitations. The larger the first characteristic value, the greater the difference in the temporal variation of the data caused by the uniformity of noise distribution and the suddenness of noise within the corresponding frequency range of the i-th modal component of the x-th sound signal.

[0055] Simultaneously, the difference in the feature difference array between the i-th modal component of the x-th sound signal and its respective control components is denoted as the second feature value between the i-th modal component of the x-th sound signal and its respective control components. In this embodiment, the difference in the feature difference array is the Euclidean distance between the feature difference arrays. As another implementation, based on the ability to measure the degree of difference between elements at the same position in two feature difference arrays, the implementer may adopt other existing feasible technologies, and this application does not impose any special limitations. The larger the second feature value, the more significant the difference in data variation between different sound signals in different frequency ranges, based on the comprehensive analysis of steady-state environmental noise and sudden noise interference caused by the trainee during the actual acquisition process.

[0056] Furthermore, by using the first and second eigenvalues ​​between the i-th modal component of the x-th sound signal and each of its control components, the difference eigenvalues ​​between the i-th modal component of the x-th sound signal and each of its control components are obtained, expressed as:

[0057] In the formula, The feature value representing the difference between the i-th modal component of the x-th sound signal and its f-th control component; , Let represent the first eigenvalue and the second eigenvalue, respectively, between the i-th modal component of the x-th sound signal and its f-th control component.

[0058] It should be noted that the calculated... The larger the value, the more significant the differences in data distribution caused by environmental steady-state noise and sudden noise during data acquisition in different frequency bands.

[0059] The difference characteristic value between each modal component and each control component is calculated according to the method for calculating the difference characteristic value between the i-th modal component of the x-th sound signal and its f-th control component.

[0060] Step 2.2: Obtain the influence characteristic value of any modal component by comparing the difference characteristic value of any modal component with the difference characteristic values ​​of each control component.

[0061] By analyzing the differences in peak distribution and temporal variation of data distribution in different frequency bands of different audio signals of the same content, it is possible to identify the differences in data variation of different audio signals in similar frequency ranges. Compared with traditional methods, by considering the similarity of audio signal characteristics of the same content, it more effectively analyzes the impact of steady-state noise and sudden noise interference on different audio signals during the acquisition process.

[0062] Furthermore, since the degree of interference from different sources varies at different time periods during the actual acquisition process, the distribution of interference effects between different sound signals in each frequency range is different. Therefore, in order to fully compare the differences between sound signals acquired under the same content in different frequency bands, the difference feature values ​​between each modal component of each sound signal and all its control components are randomly sorted to form the feature difference sequence of each modal component of each sound signal. The DTW distance between the feature difference sequence of the i-th modal component of the x-th sound signal and its control components is used as the feature difference coefficient between the i-th modal component of the x-th sound signal and its control components.

[0063] Furthermore, by using the feature difference coefficients and difference feature values ​​between the i-th modal component of the x-th sound signal and its various control components, the influence feature value of the i-th modal component of the x-th sound signal is obtained, expressed as:

[0064] In the formula, The eigenvalue represents the influence characteristic value of the i-th modal component of the x-th sound signal; F represents the number of control components of the i-th modal component of the x-th sound signal. , Let represent the difference eigenvalue and eigenvalue coefficient between the i-th modal component of the x-th sound signal and its f-th control component, respectively.

[0065] It should be noted that the calculated... The larger the value, the more significant the impact of steady-state environmental noise and sudden noise interference on the x-th sound signal at each frequency range.

[0066] Calculate the influence characteristic value of each modal component of each sound signal according to the method for calculating the influence characteristic value of the i-th modal component of the x-th sound signal.

[0067] Step 2.3: Obtain the reconstruction weight of any modal component, and then reconstruct all modal components of each sound signal to obtain each reconstructed sound signal.

[0068] Furthermore, based on the interference feature analysis results, reconstruction weights for different modal components are assigned to avoid significant differences in signal characteristics due to interference during the acquisition process. The reconstruction weights for each modal component of the x-th sound signal are obtained using the influence feature values ​​of all modal components of the x-th sound signal, expressed as:

[0069] ; The reconstruction weight represents the i-th modal component of the x-th sound signal; , represents the normalized values ​​of the influence eigenvalues ​​of the i-th and v-th modal components of the x-th sound signal, respectively; n represents the number of modal components of the x-th sound signal.

[0070] In this embodiment, the Sigmoid function is used to obtain the normalized value of the influence feature value of the modal component. The Sigmoid function is a well-known technique and will not be described in detail in this application.

[0071] It should be noted that the modal component most affected by interference has a smaller reconstruction weight. A schematic diagram of the reconstruction weight acquisition process is shown below. Figure 2 As shown.

[0072] Furthermore, a Gaussian filter is used to denoise each modal component of each sound signal, and the denoised modal components of each sound signal are then reconstructed using determined reconstruction weights to obtain the reconstructed sound signals. The Gaussian filter is a well-known technique and will not be described in detail here. The Gaussian filter is merely one embodiment of this application; implementers may use other existing feasible techniques, and this application does not impose any special limitations. A schematic diagram of the process for obtaining the reconstructed sound signal is shown below. Figure 3 As shown.

[0073] Step 3: Extract pitch, timbre, and volume from each reconstructed sound signal to obtain the sound feature map of each sound signal of the trainee. This map is used to train the sound model together with the standard sound feature map of each sound signal. Then, the sound feature map extracted from the new sound signal of the trainee is processed to evaluate the quality of the trainee's new pronunciation.

[0074] After obtaining the reconstructed sound signals, key features, including pitch, timbre, and volume, are extracted from each reconstructed sound signal. A fixed-length time window is preset for segment-by-segment analysis of each reconstructed sound signal. The YIN algorithm is used for pitch extraction. The extraction process is as follows: within each time window, the autocorrelation function of the reconstructed sound signal is calculated, and the reciprocal of the lag time when the autocorrelation function value is maximum is taken as the fundamental frequency. The pitch is then determined using the fundamental frequency. The calculation process of the autocorrelation function and the method for determining the pitch using the fundamental frequency are well-known techniques and will not be elaborated upon here. Sound quality analysis is achieved by extracting spectral features. The sound quality analysis process involves using a short-time Fourier transform to analyze the reconstructed sound signals within each time window. The audio signal is converted into a spectrum, and the characteristics such as harmonic distribution and formants in the spectrum are analyzed. Among them, the position and intensity of the formants are important indicators reflecting the sound quality. Specifically, in this embodiment, the sound quality characteristics are reflected by calculating the Mel frequency cepstral coefficients of the reconstructed audio signal in each time window. The calculation of the Mel frequency cepstral coefficients is a well-known technique and will not be described in detail in this application. The volume is measured by calculating the amplitude of the reconstructed audio signal. Specifically, the average amplitude of the reconstructed audio signal in the time domain in each time window is used as the volume value in each time window to reflect the volume characteristics of the reconstructed audio signal.

[0075] In this embodiment, the length of the time window is 12ms. While the length of the time window is within the range of [10ms, 20ms], the implementer can set the length of the time window according to the actual situation.

[0076] The extracted pitch, timbre, and volume features are integrated in chronological order to form a two-dimensional sound feature map. Specifically, for each reconstructed sound signal, the pitch, timbre, and volume features extracted within all time windows are arranged in chronological order to create a two-dimensional sound feature map. This results in a sound feature map for each reconstructed sound signal, where the horizontal axis represents time and the vertical axis represents the numerical value of each feature. Different colors are used to distinguish different features in the sound feature map, allowing for a visual representation of the changes in the trainee's voice characteristics at different times.

[0077] Based on the extracted vocal feature maps of the trainees' voice signals, a personalized voice model is constructed using deep learning algorithms combined with standard pronunciation data. First, a large amount of standard pronunciation data is collected, derived from professional singers' performances, covering different vocal genres and vocal ranges. This collected standard pronunciation data is then processed using the same methods as the trainees' voice signals to obtain standard vocal feature maps for each trainee's voice signal. Vocal genres include bel canto, folk, and pop.

[0078] Furthermore, considering that Convolutional Neural Networks (CNNs) are suitable for processing two-dimensional features such as the spectrum, this embodiment uses a CNN model to train the vocal model of the trainee. Specifically, the acoustic feature map of the trainee's voice signal is used as input, and the standard acoustic feature map of the trainee's voice signal is used as the target output. The input and output vocal types are the same. The CNN model is trained using the cross-entropy loss function and the Adam optimization algorithm. After multiple iterations of training, a personalized vocal model is obtained. The specific model training process is well-known to those skilled in the art and will not be elaborated upon here. The goal of the CNN model is to learn the mapping relationship between the acoustic feature map of the trainee's voice signal and the standard acoustic feature map, thereby identifying deviations in the trainee's pronunciation.

[0079] Collect new audio signals from the trainee for a predetermined number of different contents as described in step 1, and use the same processing method as the trainee's audio signals to obtain the audio feature map of the trainee's new audio signals. Using the trained audio model, compare the audio feature map of the trainee's new audio signals with the standard feature map to identify deviations in the trainee's new audio signals, i.e., the deviation values ​​of each feature, such as pitch deviation (Hz), volume deviation (dB), and formant position deviation. The difference analysis is performed from multiple dimensions, including pitch accuracy, timbre similarity, and volume stability. Based on the difference analysis results, a detailed pronunciation deviation feedback report is generated.

[0080] Specifically, the feedback report is presented in an intuitive way. For example, pitch deviation is displayed as a curve graph, comparing the trainee's pitch trajectory with the standard pitch. For tone quality deviation, abnormal positions of the formants are pointed out and adjustment suggestions are given. Volume deviation is displayed as a bar chart showing volume differences at different times. At the same time, combined with professional vocal training knowledge, the deviations are transformed into specific improvement methods, such as "the pitch is too low, it is recommended to appropriately increase the tension of the vocal cords" and "the formants are too forward, try opening the throat and increasing chest resonance," to help trainees optimize their pronunciation techniques in a targeted manner.

[0081] Based on the same inventive concept as the above methods, this application also provides a sound model generation system for vocal pronunciation training, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described sound model generation methods for vocal pronunciation training.

[0082] In summary, this application considers that during the construction of a sound model for vocal training, the actual acquired sound signals are often interfered with by steady-state environmental noise and sudden noise from the trainee, and these interferences have significantly different effects across different frequency bands. Therefore, this application first acquires sound signals multiple times for the same training content. For sound signals with the same content, modal components are extracted, and the sound signals are decomposed according to frequency ranges. The differences between signals in different frequency ranges are compared and analyzed, fully considering the distribution changes of sound signals caused by interference in different frequency intervals. Subsequently, the interference features of each sound signal in each frequency range are extracted, and reasonable reconstruction weights are assigned to the modal components based on the interference features to optimize the quality of the reconstructed sound signal, thereby reducing the impact of steady-state environmental interference and multi-frequency interference caused by the trainee's own factors during sound signal acquisition. Its beneficial effect lies in fully combining the interference features of different frequency bands under actual acquisition conditions to achieve accurate preprocessing of sound signals, improve the accuracy of feature extraction for sound signals of different content, and thus improve the accuracy of the constructed vocal training sound model.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from its essential characteristics. Therefore, the embodiments described above should be considered exemplary and non-limiting in all respects.

Claims

1. A method for generating sound models for vocal pronunciation training, characterized in that, The method includes the following steps: The system collects a preset number of audio signals of different content from the trainee, with each content consisting of a preset number of audio signals of a preset duration. Each modal component of each sound signal is acquired and numbered according to frequency magnitude. For any content and any modal component of any sound signal, the modal components with the same number in each other sound signal are taken as control components. The fitting curves of all peaks of each modal component in the frequency domain are obtained. The mean of the difference values ​​of the ordinates under the same horizontal axis on the fitting curves between the given modal component and each of its control components is calculated. The maximum value of the difference values ​​of the ordinates under the same horizontal axis on the fitting curves between the given modal component and each of its control components is calculated. The feature difference array of any modal component is formed by averaging the mean and averaging the maximum value between any modal component and all its control components. Calculate the difference in amplitude in the frequency domain between any modal component and each of its control components; Calculate the difference in the feature difference array between any modal component and each of its control components; determine the difference feature value between any modal component and each of its control components based on the difference and the difference amount; and take the DTW distance between the sequence of all difference feature values ​​corresponding to any modal component and the sequence of all difference feature values ​​corresponding to each of its control components as the feature difference coefficient between any modal component and each of its control components. The calculation result of multiplying the feature difference coefficient and the difference feature value is obtained; the influence feature value of any modal component is obtained through the calculation result of the multiplication, and then the reconstruction weight of any modal component is obtained. Then, all modal components of each sound signal are reconstructed to obtain each reconstructed sound signal. Pitch, timbre, and volume are extracted from each reconstructed sound signal to obtain the sound feature map of each sound signal of the trainee. This map is used to train the sound model together with the standard sound feature map of each sound signal. Then, the sound feature map extracted from the new sound signal of the trainee is processed to evaluate the quality of the trainee's new pronunciation.

2. The sound model generation method for vocal pronunciation training as described in claim 1, characterized in that, The difference feature value is the product of the gap and the difference amount.

3. The sound model generation method for vocal pronunciation training as described in claim 1, characterized in that, The influence characteristic value is the average of the calculation results of the multiplication between any modal component and all its control components.

4. The sound model generation method for vocal pronunciation training as described in claim 1, characterized in that, The expression for the reconstructed weights is: ; The reconstruction weight represents the i-th modal component of the x-th sound signal; , represents the normalized values ​​of the influence eigenvalues ​​of the i-th and v-th modal components of the x-th sound signal, respectively; n represents the number of modal components of the x-th sound signal.

5. The sound model generation method for vocal pronunciation training as described in claim 1, characterized in that, The process of extracting pitch, timbre, and volume from each reconstructed sound signal to obtain the sound feature map of each sound signal from the trainee includes: A fixed-length time window is preset. Within each time window, the autocorrelation function of the reconstructed sound signal is calculated. The reciprocal of the lag time when the autocorrelation function value is maximum is taken as the fundamental frequency. The pitch characteristics are determined by the fundamental frequency. The sound quality characteristics are reflected by calculating the Mel frequency cepstral coefficients of the reconstructed sound signal within each time window. The average amplitude of the reconstructed audio signal in the time domain within each time window is used as the volume feature within each time window; For each reconstructed sound signal, the pitch, timbre, and volume features extracted from all time windows are arranged in chronological order to create a sound feature map corresponding to each reconstructed sound signal.

6. The sound model generation method for vocal pronunciation training as described in claim 1, characterized in that, The input to the training sound model is the sound feature map of the trainer's sound signal, and the target output is the standard sound feature map of the trainer's sound signal.

7. A sound model generation system for vocal pronunciation training, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the sound model generation method for vocal pronunciation training as described in any one of claims 1-6.

Images