Music teaching method and system based on vocal music live classroom

Abstract

The invention discloses a music teaching method and system based on a vocal music live classroom, and relates to the technical field of music teaching, and the method comprises the steps: respectively collecting audio signals of a teacher end and a student end in the vocal music live classroom, carrying out the preprocessing, and generating frame-level acoustic feature vectors of teachers and students; based on frame-level acoustic feature vectors of teachers and students, generating a mapping feature vector, category probability and an adversarial sample set through mapping, domain discrimination and a CTC model, constructing a total loss function in combination with a loss function value of the model, and performing iterative optimization, gradient fixed-point rounding and feedback of model parameters; according to the invention, through constructing the total loss function, introducing the gradient inversion layer and the gradient fixed-point rounding mechanism, combining the three-state HMM state path and calculating the pronunciation quality logarithm score, the robustness and precision of student pronunciation evaluation in a vocal music live broadcast teaching scene are significantly improved, and thus a high-resolution teaching feedback basis is provided for teachers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of music teaching, in particular to a music teaching method and system based on a vocal live classroom. BACKGROUND

[0002] In recent years, the field of online music education has developed rapidly with the deep integration of Internet technology and multimedia technology. As the core form, the vocal live classroom has gradually become one of the mainstream modes of vocal teaching. The technical evolution in this field presents the characteristics of multi-disciplinary cross-fusion. On the one hand, the real-time transmission of high-definition audio and video is realized by relying on 5G communication technology, breaking through the time and space limitations of traditional vocal teaching, and making high-quality teaching resources widely shared. On the other hand, the introduction of artificial intelligence technology promotes the development of vocal teaching towards intelligence. At present, the mainstream technical solutions mostly use mel-frequency cepstral coefficients or mel features to represent acoustic features, and combine hidden Markov models or deep neural networks to build basic evaluation models, trying to simulate the subjective scoring process of professional teachers through technical means to improve teaching efficiency and standardization.

[0003] However, the existing music teaching method based on the vocal live classroom lacks the discrimination and correction mechanism for domain differences (natural differences between teachers and students in equipment quality, pronunciation conditions, speech signal collection environment, etc., which leads to evaluation deviation when directly comparing acoustic features), and is difficult to adapt to the non-stationary characteristics of acoustic features. Moreover, the model parameter quantization process lacks a fixed-point optimization mechanism for speech tasks, resulting in computational redundancy and precision loss. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a music teaching method and system based on a vocal live classroom, which solves the problems of the existing music teaching method based on a vocal live classroom, such as lack of discrimination and correction mechanism for domain differences, difficulty in adapting to non-stationary characteristics of acoustic features, and lack of fixed-point optimization mechanism for speech tasks.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a music teaching method based on a vocal live classroom, which includes,

[0008] Respectively collecting audio signals of the teacher end and the student end in the vocal live classroom for preprocessing to generate teacher and student frame-level acoustic feature vectors;

[0009] Based on the teacher, the student frame-level acoustic feature vector is mapped, the domain is discriminated, and the CTC model is generated, the mapping feature vector, the category probability and the anti-sample set are generated, the total loss function is constructed combined with the loss function value of the model, and the model parameter is iteratively optimized, the gradient is fixed point rounded and fed back;

[0010] Based on the teacher frame-level acoustic feature vector, the phoneme sequence is extracted, and the three-state HMM unit is configured to form the state path, then the phoneme posterior probability distribution vector is obtained using the teacher acoustic model, the optimal state path and the pronunciation quality logarithmic score are calculated combined with the state path, and then the phoneme pronunciation evaluation is performed, and the database is applied for storage.

[0011] The three-state HMM unit comprises a starting state, a continuous state and an ending state of the phoneme.

[0012] As a preferred scheme of the music teaching method based on the vocal live classroom of the application, wherein: after the mapping, the domain discrimination and the CTC model are generated, the mapping feature vector, the category probability and the anti-sample set are generated, and the total loss function is constructed combined with the loss function value of the model, the mapping model is constructed using the full connection neural network architecture, the student frame-level acoustic feature vector is used as the input of the mapping model, and the forward propagation is performed to generate the mapping feature vector.

[0013] The domain discrimination model is constructed using the multi-layer perception architecture, the mapping feature vector is converted into probability through the domain discrimination model to generate the category probability, including the teacher domain and the student domain.

[0014] The mapping feature vector is converted into probability through the domain discrimination model to generate the category probability.

[0015] The binary classification cross-entropy loss is defined as the frame-level domain loss function.

[0016] The pre-trained CTC model is loaded in the Kaldi tool chain, the student frame-level acoustic feature vector and the teacher frame-level acoustic feature vector are respectively input into the CTC model, the pseudo label sequence is output, the greedy decoding technology is used to decode the pseudo label sequence to generate the character-level label of the student and the teacher, including the letter and blank category.

[0017] Based on the character-level label, the frame position is divided, and the corresponding features of each frame of each category of characters are randomly sampled to form an anti-sample set, including the frame-level acoustic feature vectors in the teacher and student domains.

[0018] Based on the anti-sample set, the mapping feature vector is generated again through the mapping model, and the category probability is solved using the domain discrimination model, and the character-level domain loss function is constructed combined with the binary classification cross-entropy loss.

[0019] Based on all the loss function values, the total loss function is constructed.

[0020] As a preferred scheme of the music teaching method based on the live vocal classroom of the present application, wherein: the iterative optimization of the model parameters, the gradient fixed-point rounding and the feedback pointer are inserted into the gradient inversion layer between the mapping model and the domain discrimination model, the Adam optimizer is used to iteratively optimize each model parameter in the mapping model and the domain discrimination model, and at each parameter iteration, the gradient of each model loss function value is calculated using the stochastic gradient descent method, the mapping feature vector of the forward propagation is executed by the gradient inversion layer, and when the backward propagation is executed, a negative constant is set using the empirical rule, and then the gradient of the returned loss function value is multiplied by the negative constant to generate a direction-changing gradient for propagation, then the rounding probability function is constructed to calculate the rounding probability of each model parameter, and then a random number uniformly distributed is generated by applying a random pseudo-number generator, the random number is compared with the rounding probability, if the random number is less than or equal to the rounding probability, the gradient is rounded to the upper limit of the fixed-point value, otherwise it is rounded to the lower limit of the fixed-point value, and based on the comparison result, the fixed-point value belonging to is extracted to update each model parameter;

[0021] In the parameter updating process, the convergence condition and the convergence speed limit of the total loss function are set using the PL inequality, and until the loss converges, the optimal parameters of the mapping model and the domain discrimination model are output, including the weight matrix and the bias vector of each model;

[0022] Based on the optimal parameters, the parameter values of the mapping model and the domain discrimination model are reset, and the student frame-level acoustic feature vector is re-input into the mapping model to output the student best mapping vector, and the best mapping vector of each frame is spliced to generate a vector sequence.

[0023] As a preferred scheme of the music teaching method based on the live vocal classroom of the present application, wherein: after the phoneme sequence is extracted and the three-state HMM unit is configured to form a state path, the teacher acoustic model is used to obtain the phoneme posterior probability distribution vector, which is based on the teacher structured acoustic matrix, extracts the feature segments belonging to the task of singing along and the segment to be scored, and constructs the target acoustic reference feature vector, which contains the corresponding text lyrics content and the teacher demonstration rhythm template;

[0024] The corresponding text lyrics content in the target acoustic reference feature vector is converted using Grapheme-to-Phoneme to obtain a phoneme sequence, and after the Kaldi tool is used to configure a three-state HMM unit for each phoneme in the phoneme sequence, the HMM units are spliced in the order of phonemes to form a complete state path;

[0025] The three-state HMM unit refers to the initial state, the continuous state and the end state of the phoneme;

[0026] The teacher acoustic model is constructed using a deep neural network architecture, the student optimal mapping vector in the vector sequence is input into the final teacher acoustic model, and a phoneme probability distribution vector of each frame is output, which is defined as a phoneme posterior probability vector.

[0027] As a preferred scheme of the music teaching method based on the live vocal music classroom according to the present application, the optimal state path and the pronunciation quality logarithmic score are calculated based on the combined state path, the phoneme posterior probability in the phoneme posterior probability vector is used as the observation probability of the corresponding state, the state transition rule is set to only allow self-loop transition and forward transition, the transition probability is set using the Delphi method and is initialized;

[0028] Based on the observation probability and the transition probability, the maximum path probability of each state is calculated using a recursive formula, and at the last frame, the state corresponding to the maximum path probability is selected as the termination state, the reverse frame-by-frame backtracking is performed from the termination state, the complete optimal state path is generated, all frames belonging to the state corresponding to the current phoneme in the optimal state path are traversed, the minimum frame index is taken as the starting frame, and the maximum frame index is taken as the ending frame;

[0029] Based on the starting frame and the ending frame, the cumulative posterior probability of the phoneme is obtained by multiplying all the posterior probability values in the interval between the starting frame and the ending frame, the posterior probability product of the current phoneme in the frame interval is obtained, all the phonemes in the frame interval are traversed one by one, and the cumulative posterior probability product is performed, the posterior probability product of each phoneme is combined to generate all possible phoneme sets, the maximum posterior probability product is selected from all possible phoneme sets, the pronunciation quality logarithmic score of each phoneme is calculated based on the current phoneme posterior probability product;

[0030] The division threshold is set, the pronunciation quality logarithmic score is compared with the division threshold, when the pronunciation quality logarithmic score is greater than or equal to the division threshold, the current phoneme is marked as standard pronunciation, otherwise the phoneme is marked as non-standard;

[0031] The comparison result is used as the basis for the final music teaching result.

[0032] As a preferred scheme of the music teaching method based on the live vocal music classroom according to the present application, the application database is used for storage, the comparison result is stored using the database, and the student ID is added to the comparison result after storage.

[0033] As a preferred scheme of the music teaching method based on the live vocal classroom of the present application, wherein: the generation of the teacher and student frame-level acoustic feature vector is to use the Hamming window to frame the audio signal, form the windowed signal of each frame of audio signal, use the fast Fourier transform to convert the frequency domain, get the frequency spectrum, and use the power spectrum density formula to solve the power spectrum density of the frequency spectrum, then apply the Mel filter to cover the frequency domain in the form of triangle, combine the power spectrum density to generate the weighted energy of each Mel filter, perform logarithmic operation to get the Mel feature, use the symmetric window to perform sliding first-order difference to get the first-order difference value, perform second-order difference to get the second-order difference value, and then splice the Mel feature, the first-order difference value and the second-order difference value to generate the teacher or student frame-level acoustic feature vector.

[0034] In a second aspect, the present application provides a music teaching system based on a live vocal classroom, comprising,

[0035] The comprehensive processing module is used for collecting and pre-processing the audio signals of the teacher end and the student end to generate the teacher and student frame-level acoustic feature vectors.

[0036] The construction rounding module is used for generating the mapping feature vector, the category probability and the set of adversarial samples, constructing the total loss function, and performing parameter iterative optimization, gradient fixed-point rounding and feedback of the model.

[0037] The evaluation storage module is used for extracting the phoneme sequence, configuring the three-state HMM unit to form the state path, obtaining the phoneme posterior probability distribution vector, calculating the optimal state path and the logarithmic score of pronunciation quality, and then performing phoneme pronunciation evaluation and storage by using the database.

[0038] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any step of the music teaching method based on the live vocal classroom of the first aspect of the present application is realized.

[0039] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any step of the music teaching method based on the live vocal classroom of the first aspect of the present application is realized.

[0040] The present application has the following advantages: by constructing the total loss function, introducing the gradient inversion layer and the gradient fixed-point rounding mechanism, and combining the three-state HMM state path and the calculation of the logarithmic score of pronunciation quality, the robustness and precision of the pronunciation evaluation of the student in the live vocal teaching scene are significantly improved, thereby providing high-resolution teaching feedback basis for the teacher. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.

[0042] Fig. 1 The flow chart of the music teaching method based on the live singing classroom in Embodiment 1.

[0043] Fig. 2 The structural diagram of the music teaching system based on the live singing classroom in Embodiment 1.

[0044] Fig. 3 The flow chart of the phoneme pronunciation evaluation in Embodiment 1. DETAILED DESCRIPTION

[0045] In order to make the above objectives, features and advantages of the present application more apparent and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0046] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other ways different from those described herein without departing from the scope of the present application, and those skilled in the art can make similar generalizations without departing from the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0047] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0048] Embodiment 1, with reference to Figs. 1-3 , the first embodiment of the present application provides a music teaching method based on a live singing classroom, comprising the following steps:

[0049] S1, respectively collecting the audio signals of the teacher end and the student end in the live singing classroom for preprocessing to generate teacher and student frame-level acoustic feature vectors;

[0050] Specifically, the audio signals of the teacher end and the student end in the live singing classroom are collected respectively for preprocessing to generate teacher and student frame-level acoustic feature vectors, that is, the audio signals of the actual singing voice in the live singing classroom of the teacher are obtained by using a microphone for denoising and normalization;

[0051] The audio signal is framed using a Hamming window to form a windowed signal of each frame of the audio signal, frequency domain conversion is performed using a fast Fourier transform to obtain a spectrum, and a power spectral density formula is used to solve the power spectral density of the spectrum, a Mel filter is applied to cover the frequency domain in a triangular form, and the power spectral density is weighted to generate the weighted energy of each Mel filter for logarithmic operation to obtain Mel features;

[0052] Based on the Mel features, a symmetric window is used for sliding first-order difference to obtain a first-order difference value for second-order difference to obtain a second-order difference value, and based on the Mel features, the first-order difference value and the second-order difference value are spliced to generate a teacher frame-level acoustic feature vector;

[0053] Synchronously, the face image and the visible action of the mouth pronunciation in the live broadcast process of the teacher are collected by the camera to generate a video frame stream for H.264 format encoding to generate a compressed video stream;

[0054] Based on the teacher frame-level acoustic feature vector and the compressed video stream, the MPEG-DASH protocol is used for synchronous packaging, the course structure is manually annotated by the teacher end during the live broadcast, and the course structure annotation is embedded in the compressed video stream in the synchronous packaging in real time, and then the synchronous packaging is transmitted to the student end through the 5G network for music teaching;

[0055] The course structure annotation includes the start and end time points of each teaching action and the segment function category;

[0056] The segment function category includes singing demonstration, singing task, skill explanation and to-be-scored segment;

[0057] Based on the student end, the real-time voice input signal of the student in the music teaching process is obtained using a microphone, and then denoising and normalization are performed, and then the real-time voice input signal is framed, frequency domain converted, sliding differentiated and spliced using a Hamming window to generate a student frame-level acoustic feature vector, including Mel features, first-order difference values and second-order difference values.

[0058] By constructing a unified frame-level acoustic feature processing flow, high-precision feature extraction and dynamic modeling of teacher and student audio are realized, and the accuracy of voice analysis is enhanced. Combined with video encoding and MPEG-DASH protocol, the synchronous transmission and structured annotation of audio and video data in the live classroom are ensured, and the teaching interactivity and real-time feedback capability are improved. Through the standardized feature format, subsequent automatic scoring, pronunciation alignment and other functions are supported, which promotes the transformation of vocal music teaching from experience-driven to data-driven. Secondly, denoising and normalization provide a high-precision data basis for subsequent use.

[0059] S2. Based on the frame-level acoustic feature vectors of teachers and students, the mapped feature vectors, class probabilities and adversarial example sets are generated through mapping, domain discrimination and CTC model. The total loss function is constructed by combining the loss function value of the model, and the model parameters are iteratively optimized, gradient fixed-point rounding and feedback are performed.

[0060] Specifically, after generating mapped feature vectors, class probabilities, and adversarial example sets through mapping, domain discrimination, and CTC models, a total loss function is constructed by combining the model's loss function value. A fully connected neural network architecture is used to build the mapping model, taking the trainee's frame-level acoustic feature vectors as input and performing forward propagation to generate mapped feature vectors. The formula is as follows:

[0061]

[0062]

[0063] In the formula, In the feature mapping model, the first... The output vector of the layer, Indicates the first The layer's BatchNormalization function, Represents the ReLU activation function. Indicates the first The weight matrix of the layer, In the feature mapping model, the first... The output vector of the layer, Indicates the first Layer bias vector, Indicates time frame The mapping feature vector at that time;

[0064] A domain discrimination model is constructed using a multilayer perceptron architecture. The mapping feature vector is then probabilistically transformed using the domain discrimination model to generate class probabilities, including teacher domain and student domain.

[0065] The probability transformation of the mapped feature vector using the domain discriminant model to generate class probabilities is given by the following formula:

[0066]

[0067] In the formula, Indicates time frame Time The class probability of each mapped feature vector. This represents the Softmax function. The second-layer weight matrix of the domain discriminant model is represented. Represents the ReLU activation function. The first-layer weight matrix of the domain discriminant model is represented. The first-layer bias vector of the domain-discriminative model. The second-layer bias vector of the domain discriminant model;

[0068] The binary classification cross-entropy loss is defined as a frame-level domain loss function, and the formula is:

[0069]

[0070] In the formula, This represents the frame-level domain loss function value. Indicates the total number of frames. Indicates time frame Time The true domain labels of each mapped feature vector Represents logarithmic operations;

[0071] Load the pre-trained CTC model into the Kaldi toolchain, define the standard CTC loss function, and use the Adam optimizer to iteratively optimize the CTC model parameters until the loss converges, and output the trained CTC model.

[0072] The student frame-level acoustic feature vector and the teacher frame-level acoustic feature vector are respectively input into the CTC model, and the output pseudo-label sequence is used. Greedy decoding technology is used to decode the pseudo-label sequence to generate character-level labels for the student and the teacher, including letter and blank categories (26 letters, plus blank category, for a total of 27 categories).

[0073] Frame positions are divided based on character-level labels, and the corresponding features of one frame in each character class are randomly sampled to form an adversarial sample set, including frame-level acoustic feature vectors in the teacher and student domains.

[0074] Based on the adversarial example set, a new mapping feature vector is generated using a mapping model, and the class probability is solved using a domain discriminant model. Combined with binary classification cross-entropy loss, a character-level domain loss function is constructed, with the following formula:

[0075]

[0076] In the formula, This represents the character-level field loss function value. Indicates the total number of character categories. Represents the set of adversarial examples Chinese characters The corresponding number The class probability of each mapped feature vector. Represents logarithmic operations. Character The corresponding number a real domain label of the mapping feature vector;

[0077] Based on all the loss function values, a total loss function is constructed, and the formula is:

[0078]

[0079] In the formula, indicates a total loss function value, indicates a loss function value, indicates a weight of a frame-level domain loss function value (using network search setting), indicates a weight of a character-level domain loss function value (using network search setting).

[0080] By using a fully connected neural network containing BatchNormalization and ReLU activation function, the application can effectively improve the nonlinear expression ability of acoustic features, and at the same time, enhance the adaptability of the model in the multi-speaker (teacher and student) scene. The construction of the mapping model not only provides high-dimensional embedding for the speech recognition task, but also lays the foundation for cross-domain feature unification. By integrating the domain discrimination mechanism into the speech teaching system, the model is supervised by using cross-entropy loss in the training process, guiding the mapping model to generate feature representations that cannot be distinguished by the domain discriminator, achieving "field confrontation". This mechanism significantly alleviates the problem of speech distribution deviation caused by the teacher and student in terms of pronunciation conditions, device performance, etc., and is more helpful to achieve more fair and accurate evaluation. The CTC model does not rely on the accurate alignment between frames and labels, and by maximizing the path probability of the label sequence, it can automatically adapt to the variable-length audio input. By loading the Kaldi pre-trained model and using the greedy decoding strategy to generate 27 types of character labels (including blank), the application can quickly obtain trainable pseudo-labels in the teaching process, improving the overall real-time feedback capability. By pseudo-label division and character-level frame sampling, the application constructs an adversarial sample set covering teachers and students. This mechanism realizes the comparative learning of cross-domain samples on the same phonemes, so that the application can capture more stable speech unit features and improve the detection capability of incorrect pronunciation in the music teaching of the music teaching. By constructing a total loss function by integrating the CTC loss, the frame-level domain loss and the character-level domain loss, and setting the weight coefficients for different loss terms, the application realizes the dual constraints of accuracy and robustness in the model training process. The multi-objective optimization strategy makes the application not only guarantee the recognition performance, but also realize the alignment of speech style;

[0081] Therefore, the application organically integrates end-to-end speech recognition, domain adaptation and adversarial training mechanism, and realizes a cross-domain pronunciation evaluation method based on live scenes in the field of vocal music teaching, taking into account real-time, accuracy and universality.

[0082] Furthermore, iterative optimization of model parameters, gradient fixed-point rounding, and feedback are performed. A gradient inversion layer is inserted between the mapping model and the domain discrimination model. The Adam optimizer is used to iteratively optimize each model parameter in the mapping model and the domain discrimination model respectively. During each parameter iteration, the gradient of the loss function value of each model is calculated using stochastic gradient descent. The gradient inversion layer performs an identity transformation on the forward propagation mapping feature vector (i.e., without modifying the data, it is directly passed). During backpropagation, a negative constant is set using empirical rules. The gradient of the returned loss function value is multiplied by the negative constant to generate a gradient with a changed direction. After propagation, a rounding probability function is constructed to calculate the rounding probability of each model parameter. A random pseudo-number generator is then used to generate a uniformly distributed random number. The random number is compared with the rounding probability. If the random number is less than or equal to the rounding probability, the gradient is rounded to the upper bound point value; otherwise, it is rounded to the lower bound point value. Based on the comparison result, the corresponding fixed-point value is extracted to update each model parameter.

[0083] The rounding probability function is constructed to calculate the rounding probability of each parameter, and the formula is as follows:

[0084]

[0085] In the formula, Indicates the first The rounding probability of the gradient of each parameter. This means limiting the probability to Functions between This represents the vertex baseline value rounded down. Represents the smallest unit of resolution at a fixed point (e.g.) (This indicates 5-bit precision). Represents a symbolic function. This represents the bias factor (which can be set through experiments and relevant knowledge).

[0086] The gradient is then rounded to the upper bound point value, as shown in the formula:

[0087]

[0088] In the formula, This indicates the upper bound value for rounding. This represents the floor operator;

[0089] Otherwise, the value is rounded to the lower bound point, and the formula is as follows:

[0090]

[0091] In the formula, This indicates the lower bound value for rounding. This represents the floor operator;

[0092] The formula for updating each model parameter by extracting the corresponding fixed-point value is as follows:

[0093]

[0094]

[0095] In the formula, This indicates the fixed-point value (upper bounding point value or lower bounding point value) to be rounded. Indicates belonging to, Indicates the first Update the first iteration during the round of iteration Updated values ​​for each parameter. Indicates the first Parameter vector during round iteration The Middle The values ​​of the parameters, Represents set symbols;

[0096] During the parameter update process, the PL inequality is used to set the convergence condition and convergence rate limit of the total loss function, and after the loss converges, the optimal parameters of the mapping model and the domain discrimination model are output, including the weight matrix and bias vector of each model.

[0097] The convergence condition for the total loss function is set using the PL inequality, and the formula is as follows:

[0098]

[0099] In the formula, Represents the PL coefficient. Indicates through parameter vector The calculated total loss function value, This represents the target value of the total loss function (which can be set through experiments and relevant knowledge). Describes the Euclidean norm. Represents the parameter vector The gradient vector;

[0100] The convergence rate limit of the total loss function is set using the PL inequality, and the formula is as follows:

[0101]

[0102] In the formula, Represents the expectation operator. Indicates the first During round iteration, the parameter vector is used The calculated total loss function value, Indicates the learning rate. Indicates the convergence factor. Indicates the first wheel iteration, representing the initial iteration time through the parameter vector the total loss function value calculated;

[0103] Based on the optimal parameters, the parameter values of the mapping model and the domain discrimination model are reset, and the student frame-level acoustic feature vector is re-input into the mapping model to output the student optimal mapping vector, and the optimal mapping vector of each frame is spliced to generate a vector sequence.

[0104] In the traditional method, there are non-semantic differences such as collection equipment, context, and environmental noise between the teacher and the student audio, and if it is directly used for pronunciation scoring, it is easy to produce deviation. The present application controls the gradient back propagation direction through GRL, so that the present application can learn "field-independent" speech embedding representation, effectively solving the scoring domain offset problem. By constructing a rounding probability function + random pseudorandom generator + upper and lower bound point selection mechanism, a weak random disturbance is introduced in the parameter update process, which has the ability of regularization and anti-overfitting. Especially in vocal music teaching, the pronunciation features have strong individual differences and contingency, and this mechanism can improve the generalization performance of the present application when migrating between different students. The convergence termination condition and speed upper bound are set by using the PL inequality, forming a dynamic convergence control process based on the gradient norm and the difference between the target loss. The introduction of this mechanism provides a mathematical level of robustness for the optimization process of the present application, especially in the face of high-dimensional acoustic feature space and small sample live scenes, it can still ensure that the training process is efficient, stable and interpretable.

[0105] Therefore, the present application is especially suitable for vocal music live classroom teaching scenes with high real-time requirements and limited computing resources, and through fine modeling and optimization control, the intelligent level of the music teaching system is significantly improved.

[0106] S3, based on the teacher frame-level acoustic feature vector, extracting the phoneme sequence, and configuring a three-state HMM unit to form a state path, using the teacher acoustic model to obtain the phoneme posterior probability distribution vector, combining the state path to calculate the optimal state path and the pronunciation quality logarithmic score, and then performing phoneme pronunciation evaluation, and applying the database for storage;

[0107] Specifically, after extracting the phoneme sequence and configuring a three-state HMM unit to form a state path, using the teacher acoustic model to obtain the phoneme posterior probability distribution vector, based on the teacher structured acoustic matrix, extracting the feature segments belonging to the karaoke task and the segment to be scored, constructing the target acoustic reference feature vector, including the corresponding text lyrics content, the teacher demonstration rhythm template (the duration of each syllable);

[0108] The corresponding text lyric content to which the target acoustic reference feature vector belongs is converted using Grapheme-to-Phoneme to obtain a phoneme sequence, and after a three-state HMM unit is configured for each phoneme in the phoneme sequence using a Kaldi tool, the HMM units are spliced in the order of phonemes to form a complete state path;

[0109] The three-state HMM unit refers to the initial state, the continuous state and the end state of the phoneme;

[0110] A teacher acoustic model is constructed using a deep neural network architecture, a cross-entropy loss function is defined as a loss function, and the parameter combination in the teacher acoustic model is iteratively optimized using an Adam optimizer, and when the loss value of the loss function no longer obviously decreases, the iteration is stopped, and the final teacher acoustic model is output;

[0111] The student best mapping vector in the vector sequence is input into the final teacher acoustic model, and the phoneme probability distribution vector of each frame is output, which is defined as the phoneme posterior probability vector.

[0112] By extracting features in the singing task and the to-be-scored segment and introducing a teacher demonstration rhythm template, the present application can obtain a time-aligned structured acoustic reference as a "gold standard" for subsequent evaluation. This can standardize the subjective singing style into quantifiable time-frequency features, which helps to realize synchronous evaluation of the rhythm dimension and the sound quality dimension, thereby enhancing the multidimensionality of teaching feedback. The mapping of the lyric content to the phoneme layer is realized through G2P, and an HMM unit is constructed for each phoneme through a Kaldi tool to realize dynamic modeling of the speech state. This deep fusion of natural language processing technology and acoustic modeling constructs a high-fitting phoneme path, which can realize fine-grained analysis of the pronunciation process of each phoneme, avoiding the extensive evaluation mode of traditional scoring based on complete sentences or complete words. Through the trained teacher acoustic model, the student's acoustic feature mapping vector is converted into the posterior probability distribution of the corresponding phoneme, which is then used to accurately judge the accuracy of pronunciation. The present application does not need to use the same speech acquisition conditions for the teacher and the student, but determines through a unified acoustic model reference, has good migration adaptability, and improves the evaluation consistency across devices and environments.

[0113] Further, the optimal state path and the pronunciation quality logarithmic score are calculated in combination with the state path, and then the phoneme pronunciation evaluation is performed. Each phoneme posterior probability in the phoneme posterior probability vector is taken as the observation probability of the corresponding state, and the state transition rule is set to only allow self-loop transition and forward transition (i.e., the state can only be transferred to itself or the next state wherein , The transition probability is set and initialized using the Delphi method, and the total number of states is represented by N.

[0114] Based on the observation probability and the transition probability, the maximum path probability of each state is calculated using a recursive formula. At the last frame, the state corresponding to the maximum path probability is selected as the termination state. From the termination state, the complete optimal state path is generated by backtracking frame by frame. Then, all frames belonging to the current phoneme corresponding state in the optimal state path are traversed, and the minimum frame index is taken as the starting frame and the maximum frame index as the ending frame.

[0115] Based on the starting frame and the ending frame, the cumulative posterior probability of the current phoneme is obtained by multiplying all the posterior probability values in the interval between the starting frame and the ending frame. The posterior probability product of each phoneme is obtained by multiplying the cumulative posterior probability of the current phoneme and performing the cumulative posterior probability product of all phonemes in the frame interval. The complete optimal state path is generated by backtracking frame by frame. Then, all frames belonging to the current phoneme corresponding state in the optimal state path are traversed, and the minimum frame index is taken as the starting frame and the maximum frame index as the ending frame.

[0116]

[0117] In the formula, represents the log score of the pronunciation quality of the phoneme , represents the natural logarithm function, represents the maximum value operation, represents the phoneme , represents all possible phoneme sets, represents the posterior probability product of the phoneme in the alignment feature sequence from the given starting frame to the ending frame , represents the posterior probability product of the phoneme in the alignment feature sequence from the given starting frame to the ending frame .

[0118] The alignment feature sequence can be directly generated by backtracking the optimal state path.

[0119] The division threshold is set using domain knowledge and teacher experience. The log score of the pronunciation quality is compared with the division threshold. If the log score of the pronunciation quality is greater than or equal to the division threshold, the current phoneme is marked as standard pronunciation. Otherwise, the phoneme is marked as non-standard.

[0120] ​​​The comparison result is taken as a final music teaching result basis.

[0121] By setting the "self-looping" and "forward-moving" state transition rules, the optimal state path is derived by using the recursive algorithm, effectively simulating the time extension characteristics (such as holding notes and sentence breaks) in actual vocal performance. This simplified but efficient state transition mechanism avoids excessive invalid path interference, improving the path decoding efficiency and accuracy. The backtracking strategy ensures that the boundaries of the evaluation area are determined by the true optimal state, enhancing the accuracy of the judgment of the start and end points of the performance. The intelligibility of phoneme-level performance not only depends on the accuracy of a single frame, but also relies on the continuous expression within the entire time period. By multiplying all the posterior probabilities between the start frame and the end frame, the invention can consider the performance of the phoneme within the entire interval, enhance the time domain consistency of the score, and avoid the deviation of the score from the true performance caused by an abnormal value in a certain frame. By comparing the product of the posterior probability of the current phoneme and all possible phonemes with the maximum probability value as a reference item, the scoring deviation caused by label errors can be effectively suppressed. This relative scoring mechanism not only has a normalization effect, but also is more sensitive to small differences after taking the logarithm, especially suitable for distinguishing easily confused phonemes such as "s" and "sh" in fine teaching scenarios. By combining teacher experience and knowledge in the field of vocal teaching to set the score division threshold, the invention can automatically judge whether each phoneme meets the teaching requirements. This method has flexibility and professionalism, which can adapt to the differences of students at different levels, and also facilitates teachers to implement differentiated teaching strategies based on the scoring results, effectively improving the relevance and operability of teaching feedback.

[0122] Further, the application database is used to store the comparison result, and the student ID is added to the comparison result after storage.

[0123] By associating and storing the comparison result with the student ID, the invention can accumulate the performance data of each student in different classes and different tasks for a long time, so that the teaching strategy can be dynamically adjusted according to the historical performance in the future.

[0124] The embodiment also provides a music teaching system based on a vocal live classroom, which comprises:

[0125] The comprehensive processing module is used to collect and preprocess the audio signals of the teacher end and the student end, and generate teacher and student frame-level acoustic feature vectors.

[0126] The rounding module is used to generate a mapping feature vector, a category probability, and an adversarial sample set, construct a total loss function, and perform parameter iterative optimization, gradient fixed-point rounding, and feedback of the model.

[0127] The evaluation storage module is used to extract the phoneme sequence, configure a three-state HMM unit to form a state path, obtain a phoneme posterior probability distribution vector, calculate an optimal state path and a pronunciation quality logarithmic score, and then perform phoneme pronunciation evaluation and storage by using a database.

[0128] The embodiment also provides a computer device suitable for the music teaching method based on the vocal music live classroom, which comprises a memory and a processor.

[0129] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0130] The embodiment also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to implement the music teaching method based on the vocal music live classroom. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0131] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A music teaching method based on a vocal music live classroom, characterized in that: The application relates to a voice live classroom teacher and student end audio signal pre-processing method and device. Based on the teacher and student frame-level acoustic feature vectors, mapping feature vectors, category probabilities and an adversarial sample set are generated through mapping, domain discrimination and a CTC model, a total loss function is constructed by combining the loss function value of the model, and model parameter iterative optimization, gradient fixed-point rounding and feedback are performed. Based on the teacher frame-level acoustic feature vectors, phoneme sequences are extracted, a three-state HMM unit is configured to form a state path, a phoneme posterior probability distribution vector is obtained using a teacher acoustic model, the optimal state path and pronunciation quality logarithmic score are calculated by combining the state path, phoneme pronunciation evaluation is performed, and a database is used for storage. The three-state HMM unit comprises a phoneme start state, a phoneme duration state and a phoneme end state. The mapping feature vector, the category probability and the adversarial sample set are generated through the mapping, the domain discrimination and the CTC model, and the total loss function is constructed by combining the loss function value of the model.

2. The vocal live lesson based music teaching method according to claim 1, wherein: The domain discrimination model is constructed using a multi-layer perception architecture, the mapping feature vector is converted into a probability through the domain discrimination model, and the category probability is generated, including a teacher domain and a student domain. The mapping feature vector is converted into a probability through the domain discrimination model, and the category probability is generated. The binary classification cross-entropy loss is defined as a frame-level domain loss function. The pre-trained CTC model is loaded in the Kaldi tool chain, the student frame-level acoustic feature vector and the teacher frame-level acoustic feature vector are respectively input into the CTC model, a pseudo-label sequence is output, the pseudo-label sequence is decoded using a greedy decoding technology, and character-level labels of the student and the teacher are generated, including letters and blank categories. Based on the character-level labels, frame positions are divided, and corresponding features of one frame in each category of characters are randomly sampled to form an adversarial sample set, including the frame-level acoustic feature vectors in the teacher and student domains. Based on the adversarial sample set, the mapping feature vector is regenerated through the mapping model, the category probability is solved using the domain discrimination model, the character-level domain loss function is constructed by combining the binary classification cross-entropy loss, and the total loss function is constructed based on all the loss function values. ​ ​ 3. The vocal-based live lesson classroom music teaching method of claim 2, wherein: The iterative optimization of the model parameters, the gradient fixed-point rounding, and the feedback are as follows: a gradient reversal layer is inserted between the mapping model and the domain discrimination model, the Adam optimizer is used to iteratively optimize each model parameter in the mapping model and the domain discrimination model, and at each parameter iteration, the gradient of each model loss function value is calculated using the stochastic gradient descent method, the gradient reversal layer performs an identity transformation on the forward-propagated mapping feature vector, and when back-propagating, a negative constant is set using the empirical rule, the gradient of the returned loss function value is multiplied by the negative constant to generate a direction-changing gradient, which is propagated to construct a rounding probability function to calculate the rounding probability of each model parameter, and a random number uniformly distributed is generated by a random pseudo-number generator, which is compared with the rounding probability, if the random number is less than or equal to the rounding probability, the gradient is rounded to the upper bound fixed-point value, otherwise, it is rounded to the lower bound fixed-point value, and based on the comparison result, the corresponding fixed-point value is extracted to update each model parameter; In the parameter updating process, the convergence condition and the convergence speed limit of the total loss function are set using the PL inequality, and after the loss converges, the optimal parameters of the mapping model and the domain discrimination model are output, including the weight matrix and the bias vector of each model; Based on the optimal parameters, the parameter values of the mapping model and the domain discrimination model are reset, and the student frame-level acoustic feature vector is re-input into the mapping model to output the student best mapping vector, and the best mapping vectors of each frame are spliced to generate a vector sequence.

4. The vocal-based live classroom music teaching method of claim 3, wherein: After the phoneme sequence is extracted and the three-state HMM unit is configured to form a state path, the teacher acoustic model is used to obtain the phoneme posterior probability distribution vector, which is based on the teacher structured acoustic matrix, extracts the feature segments belonging to the karaoke task and the segment to be scored, and constructs the target acoustic reference feature vector, which includes the corresponding text lyrics content and the teacher demonstration rhythm template; The corresponding text lyrics content in the target acoustic reference feature vector is converted into a phoneme sequence using Grapheme-to-Phoneme, and after the Kaldi tool is used to configure a three-state HMM unit for each phoneme in the phoneme sequence, the HMM units are spliced in the order of phonemes to form a complete state path; The three-state HMM unit refers to the initial state, the persistent state, and the end state of the phoneme; The teacher acoustic model is constructed using a deep neural network architecture, the student best mapping vector in the vector sequence is input into the final teacher acoustic model, and the phoneme probability distribution vector of each frame is output, which is defined as the phoneme posterior probability vector.

5. The vocal-based live classroom music teaching method of claim 4, wherein: The optimal state path and the pronunciation quality logarithmic score are calculated based on the state path, and the phoneme pronunciation evaluation is performed, which is that each phoneme posterior probability in the phoneme posterior probability vector is used as the observation probability of the corresponding state, the state transition rule is set to only allow self-loop transition and forward transition, and the transition probability is set and initialized using the Delphi method; Based on the observation probability and the transition probability, the maximum path probability of each state is calculated by using a recursive formula, and at the last frame, the state corresponding to the maximum path probability is selected as the termination state. From the termination state, the complete optimal state path is generated by backtracking frame by frame. Then, all frames belonging to the current phoneme corresponding state in the optimal state path are traversed, and the minimum frame index is taken as the starting frame and the maximum frame index as the ending frame. Based on the starting frame and the ending frame, the cumulative posterior probability of the phoneme is obtained by multiplying all the posterior probability values in the interval between the starting frame and the ending frame. The posterior probability product of each phoneme is obtained by multiplying the cumulative posterior probability of the phoneme. All possible phoneme sets are generated by combining the posterior probability products of all phonemes in the frame interval. The maximum posterior probability product is selected from all possible phoneme sets. The pronunciation quality logarithmic score of each phoneme is calculated by combining the current phoneme posterior probability product. A division threshold is set to compare the pronunciation quality logarithmic score with the division threshold. If the pronunciation quality logarithmic score is greater than or equal to the division threshold, the current phoneme is marked as standard pronunciation. Otherwise, the phoneme is marked as non-standard. The comparison result is used as the final result of the music teaching.

6. The vocal-based live classroom music teaching method of claim 5, wherein: The application database is used to store the comparison result and add the student ID to the comparison result after storage.

7. The vocal-based live lesson classroom music teaching method of claim 6, wherein: The teacher and student frame-level acoustic feature vectors are generated by using a Hamming window to frame the audio signal, forming a windowed signal for each frame of audio signal. Fast Fourier transform is used for frequency domain conversion to obtain a frequency spectrum. The power spectral density of the frequency spectrum is solved using the power spectral density formula. A Mel filter is applied to cover the frequency domain in a triangular form. The power spectral density is weighted to generate the weighted energy of each Mel filter. The Mel feature is obtained by taking the logarithm of the weighted energy. A symmetric window is used for sliding first-order difference to obtain a first-order difference value. The second-order difference value is obtained by taking the second-order difference of the first-order difference value. The teacher or student frame-level acoustic feature vector is generated based on the Mel feature, the first-order difference value, and the second-order difference value.

8. A music teaching system based on a live vocal classroom, based on the music teaching method based on a live vocal classroom according to any one of claims 1-7, characterized in that: The comprehensive processing module is used to collect and preprocess the audio signals of the teacher and student to generate the teacher and student frame-level acoustic feature vectors. The rounding module is used to generate the mapping feature vector, the class probability, and the adversarial sample set. The total loss function is constructed, and the parameter iteration optimization, gradient fixed-point rounding, and feedback of the model are performed. The evaluation storage module is used to extract the phoneme sequence, configure the three-state HMM unit to form the state path, obtain the phoneme posterior probability distribution vector, calculate the optimal state path and the pronunciation quality logarithmic score, perform the phoneme pronunciation evaluation, and store the result in the database. The processor executes the computer program to implement the steps of the music teaching method based on the vocal live classroom of any one of claims 1-7. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The computer program is executed by the processor to implement the steps of the music teaching method based on the vocal live classroom of any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: ​

Images