Vocal music training vowel pronunciation quality evaluation method based on auditory and visual spatio-temporal feature fusion

Abstract

The invention provides a vocal music training vowel pronunciation quality evaluation method based on auditory and visual spatial-temporal feature fusion, and the method comprises the steps: collecting vowel pronunciation audio signals and corresponding videos of a singer, and constructing a multi-modal data set; generating a fractional order Mel spectrogram for the audio signal through short-time fractional order Fourier transform of an adaptive order; extracting time sequence features and spatial features of the fractional order Mel spectrogram, and fusing the time sequence features and the spatial features through a gating mechanism to generate audio spatio-temporal features; face visual features in the video are extracted and fused with the audio spatio-temporal features through a cross attention mechanism, and the cross attention mechanism is integrated with a periodic modeling network; the fused features are input into a classifier, a dynamic weight multi-mode cosine loss function training model is adopted, the dynamic weight multi-mode cosine loss function dynamically adjusts the sample weight through a confusion matrix, and the weight is increased for the samples with classification errors based on the historical frequency mistaken division times of the samples; and outputting a pronunciation quality evaluation result.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary technical field of digital signal processing, computer vision, artificial intelligence and deep learning, and multimodal information fusion, and specifically relates to a method for evaluating the pronunciation quality of vowels in vocal training by fusing auditory and visual spatiotemporal features. Background Art

[0002] With the rapid development of digital music technology, the field of music education is undergoing profound changes. Traditional music teaching methods rely on direct instruction from experienced teachers. While effective, this approach also presents challenges such as high cost, low efficiency, and geographical limitations. In contrast, AI-assisted teaching not only improves teaching efficiency and quality, but also provides students with a more personalized and interactive learning experience.

[0003] Currently, the application of AI technology in the music field is primarily to assist in music creation. However, its application in music education is lacking. Pronunciation training is the most fundamental part of music teaching. In music classrooms, teachers and students typically teach one-on-one, but due to limited teaching resources, it is difficult for teachers to provide comprehensive instruction to every student. Based on this, we have combined deep learning technology with vocal education, aiming to guide students to provide timely feedback and corrections on their pronunciation during vocal training.

[0004] Although there is currently a lack of evaluation models specifically for basic pronunciation of professional voices, we can divide vocal pronunciation into two parts. One part is the evaluation of audio information, that is, classifying the input audio information according to audio features such as the pitch, timbre, intonation, resonance peak, and resonance area of ​​the pronunciation. The other aspect can consider the evaluation of facial features during pronunciation, and classify them according to a series of facial feature information such as mouth shape and facial expression. Therefore, this task can be regarded as an audio-visual multimodal classification task.

[0005] When evaluating audio information, traditional feature extraction methods often focus on three core elements: pitch, timbre, and intonation. While this approach can reflect a singer's basic skill level to a certain extent, its limitations are becoming increasingly apparent. First, due to physiological differences between men and women, women generally have a higher natural vocal range than men. Furthermore, within the same gender, there are also distinct characteristics between mid-range and high-range voices. Therefore, relying solely on the traditional three-element evaluation method is difficult to comprehensively and accurately assess pronunciation quality. Furthermore, the issue of resonance area cannot be ignored. High-quality vocal pronunciation should contain a high proportion of head cavity resonance, which not only enhances the roundness and penetration of the voice but is also a key factor in improving the overall auditory experience. Therefore, in light of these issues, it is difficult to use a unified standard to evaluate pronunciation quality. Common music evaluation models rely primarily on neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and recurrent gated units (GRUs). These methods offer advantages in speech signal processing, including automatic feature extraction, strong modeling capabilities, and adaptability to varying timescales. They have significantly improved performance in tasks such as speech recognition, speech synthesis, and speech sentiment analysis. However, these methods still face challenges that need to be addressed.

[0006] (1) Most existing evaluation models focus on technical features such as pitch and rhythm, while ignoring the important dimension of emotional expression. However, in singing performances, emotional expression can often significantly affect the audience's feelings. Therefore, how to effectively capture and evaluate emotional expression has become an urgent problem to be solved.

[0007] (2) Every singer has his or her own unique voice and personal style, and existing evaluation systems may not be able to adapt well to such individual differences.

[0008] High-quality labeled data is crucial for training effective machine learning models. However, in the field of singing pronunciation quality assessment, obtaining large amounts of high-quality labeled data is a challenge, which limits the generalization ability and accuracy of the model. Summary of the Invention

[0009] In response to the shortcomings and deficiencies of existing technologies, the present invention provides a method and system for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, and spatiotemporal features. This system uses multimodal deep learning technology to address issues such as reliance on subjective experience, neglect of emotional expression, difficulty in quantifying individual differences, and data imbalance in traditional vocal teaching. The core innovative design of this solution is as follows:

[0010] 1. Fractional-order Mel acoustic feature extraction technology

[0011] A short-time fractional Fourier transform with adaptive order is designed to dynamically optimize the fractional-order domain conversion parameters (0-1.0 order), breaking through the limitations of the time-frequency positioning capabilities of traditional spectrograms and more accurately capturing the time-frequency energy distribution characteristics of vowel pronunciation.

[0012] Combined with the Mel-scale filter bank to compress frequency domain information, it simulates the auditory characteristics of the human ear (low-frequency sensitivity is higher than high-frequency sensitivity), making the feature expression more in line with the human pronunciation quality judgment criteria.

[0013] 2. Multimodal spatiotemporal feature collaborative fusion mechanism

[0014] 1. Audio spatiotemporal feature fusion:

[0015] A gated weighted fusion mechanism is used to dynamically balance the temporal dependency features extracted by LSTM and the spatial spectrum features extracted by CNN to solve the feature redundancy problem caused by traditional splicing.

[0016] 2. Cross-modal fusion of audio and visual

[0017] Innovative design of Fourier-enhanced Cross-Attention Network (FAN):

[0018] Integrate the periodic feature modeling component (cosine / sine transform) and the transient feature preservation component (nonlinear activation) in the hidden layer to simultaneously capture the periodicity of the acoustic signal and the transient changes in pronunciation;

[0019] Audio and video feature alignment is achieved through a bidirectional cross-attention mechanism, and the scaling factor adjusts the attention gradient distribution to reduce modal difference interference.

[0020] 3. Dynamic Weight Multimodal Loss Optimization Mechanism

[0021] Propose the dynamic weight multimodal cosine loss function (DWMMCL):

[0022] Based on the confusion matrix, the historical misclassification frequency of samples is counted, and the weight of high-frequency misclassification samples is dynamically increased to improve the model's ability to focus on difficult samples;

[0023] Audio and visual modal features are fused through cosine similarity to construct a hyperspherical feature space and coordinate multimodal optimization conflicts.

[0024] 4. Closed-loop acoustic training feedback system

[0025] Build an end-to-end evaluation-feedback execution chain:

[0026] Input is limited to Chinese and Latin core vowels (a, o, e, i, u), covering typical vocal training scenarios;

[0027] Output quality assessment results and match preset acoustic correction plans (such as insufficient resonance → head cavity resonance enhancement training) to form a closed teaching loop.

[0028] This solution, for the first time, deeply integrates fractional-order signal processing, periodic modeling networks, and a dynamic weight loss mechanism, breaking through the limitations of traditional vocal assessment models that ignore emotional expression and individual differences. By integrating multimodal auditory and visual features with closed-loop feedback, it provides an efficient, objective, and intelligent auxiliary tool for vocal instruction, driving technological innovation in music education.

[0029] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0030] A method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory and visual spatiotemporal features, comprising:

[0031] Collect the singer's vowel pronunciation audio signals and corresponding videos to construct a multimodal dataset annotated with pitch, timbre, rhythm, resonance, facial expression and lip shape quality grades;

[0032] Generating a fractional Mel spectrogram by performing adaptive-order short-time fractional Fourier transform on the audio signal;

[0033] Extracting the temporal features and spatial features of the fractional-order Mel spectrogram, and fusing them to generate audio spatiotemporal features through a gating mechanism;

[0034] Extracting facial visual features from the video and fusing them with the audio spatiotemporal features through a cross-attention mechanism, wherein the cross-attention mechanism integrates a periodic modeling network; the periodic modeling network simultaneously integrates a periodic feature modeling component and a transient feature preservation component in a hidden layer;

[0035] The fused features are input into the classifier, and the model is trained using a dynamic weighted multimodal cosine loss function. The dynamic weighted multimodal cosine loss function dynamically adjusts the sample weights through a confusion matrix, increasing the weight of misclassified samples based on their historical misclassification frequency.

[0036] Output pronunciation quality assessment results.

[0037] Furthermore, the periodic modeling network is a Fourier analysis network, whose hidden layer output is generated by splicing cosine transform components, sine transform components and nonlinear activation components. The periodic characteristics of the acoustic signal are modeled by the cosine transform components and sine transform components, and the time domain non-stationary characteristics are retained by the nonlinear activation components.

[0038] Furthermore, in the dynamic weighted multimodal cosine loss function, the weight increment of the misclassified sample is proportional to the frequency with which it is misclassified into the predicted category.

[0039] Furthermore, the dynamic weighted multimodal cosine loss function fuses audio and visual features through cosine similarity, and accumulates the number of misclassifications; is updated at the end of each training cycle and stored in a confusion matrix; and the growth rate of the weight increment is controlled by an adjustment factor.

[0040] Furthermore, the confusion matrix is ​​updated according to the training cycle to store the historical records of each sample being misclassified; wherein the update operation is performed at the end of each training cycle, and the historical records include the cumulative number of times the true category of the sample is misclassified as the predicted category.

[0041] Furthermore, the optimization method of the adaptive order is:

[0042] Initialize the fractional order to a learnable parameter in the interval [0, 1.0];

[0043] The order value is dynamically adjusted during training via backpropagation.

[0044] Furthermore, the gating mechanism dynamically balances temporal features and spatial features through trainable weights.

[0045] Furthermore, the cross-attention mechanism generates cross-modal joint features through bidirectional query-key-value interaction, where the query vector originates from one modality and the key-value vector originates from the other modality; and balances the attention score gradients through a scaling factor d to prevent softmax saturation.

[0046] Furthermore, the vowel types are a, o, e, i, and u.

[0047] And, a vocal training pronunciation quality assessment system, comprising:

[0048] (a) a multimodal data acquisition device, comprising a microphone array and a high-definition camera, for collecting the singer's vowel pronunciation audio signals and facial video stream in real time;

[0049] (b) A fractional-order signal processor configured as:

[0050] performing an adaptive-order short-time fractional Fourier transform on the audio signal;

[0051] Compress frequency domain information through Mel-scale filter banks;

[0052] Output time-frequency feature map to feature fusion gateway;

[0053] (c) A spatiotemporal feature fusion engine, which includes a parallel LSTM timing processing unit and a CNN convolutional processing unit, generating an audio spatiotemporal feature vector through a gated weighted circuit;

[0054] (d) Cross-modal decision module, including:

[0055] ResNet visual feature extractor to parse facial action units;

[0056] Fourier-enhanced cross-attention kernel to fuse audio and visual features;

[0057] Dynamic weight classifier, which updates sample weights based on the confusion matrix;

[0058] (e) Feedback execution terminal receives the quality assessment results and matches the preset acoustic correction scheme based on the assessment results.

[0059] And, a computer device includes a memory, a processor and a computer program stored in the memory, and the processor implements the above method when executing the computer program.

[0060] A non-transitory computer-readable storage medium stores a computer program, which implements the method described above when executed by a processor.

[0061] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0062] 1. The ability to express acoustic features is significantly enhanced

[0063] Through adaptive fractional-order Mel acoustic feature extraction technology, we break through the time-frequency analysis limitations of traditional spectrograms, achieve more accurate pronunciation resonance peak and energy distribution analysis, and solve the problems of physiological differences between men and women and resonance area evaluation.

[0064] By combining the human hearing characteristics to optimize feature expression, quality assessment can be more in line with human subjective judgment standards.

[0065] 2. Deep optimization of multimodal fusion effects

[0066] The gating mechanism dynamically balances the temporal and spatial characteristics of audio (time dependency + spectrum space), avoiding the information redundancy of traditional feature splicing;

[0067] The Fourier-enhanced cross-attention network simultaneously captures acoustic periodicity and transient changes in pronunciation, significantly improving the ability to model the correlation between emotional expression and lip shape.

[0068] 3. Effectively improve training generalization performance

[0069] The dynamic weight loss mechanism uses misclassification frequency statistics to focus on difficult sample training, solving the problems of scarcity and category imbalance of professional vocal data;

[0070] Multimodal cosine similarity is coordinated and optimized to construct a more discriminative feature space distribution.

[0071] 4. Improved efficiency of the evaluation-feedback closed loop

[0072] Limit the core vowel types (a, o, e, i, u) to cover typical vocal training scenarios and improve the system's pertinence;

[0073] The quality results are matched with preset acoustic correction plans (such as resonance enhancement training) to form a feasible teaching intervention closed loop.

[0074] 5. Enhanced technical robustness

[0075] The confusion matrix updates the misclassification history according to the training cycle, ensuring that dynamic weight adjustments respond to model state changes in real time;

[0076] Adaptive optimization of fractional order can avoid manual parameter adjustment deviation and adapt to the fluctuation of acoustic characteristics of different speakers.

[0077] Overall, it achieved:

[0078] Improved objectivity: Quantitative evaluation replaces subjective experience judgment, eliminating the impact of insufficient teaching staff and geographical restrictions;

[0079] Efficiency optimization: End-to-end automated processing shortens the training feedback cycle and enables immediate correction;

[0080] Universal expansion: The multimodal collaborative mechanism can be transferred to other pronunciation training scenarios (such as language learning and pathological speech rehabilitation). BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0082] Figure 1 Mel mapping diagram for Mel scale conversion.

[0083] Figure 2 Schematic diagram of a triangular Mel filter bank.

[0084] Figure 3 Schematic diagram of the fractional-order Mel spectrogram generation method.

[0085] Figure 4 This is a structural diagram of a Fourier analysis network (FAN) according to an embodiment of the present invention.

[0086] Figure 5 This is a structural diagram of the cross-attention module according to an embodiment of the present invention.

[0087] Figure 6 This is a diagram of the prediction process of evaluating the quality of voice training pronunciation according to an embodiment of the present invention.

[0088] Figure 7 Schematic diagram of the training loss entropy and accuracy curve of an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are given for detailed description:

[0090] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs.

[0091] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0092] The traditional deep learning model focuses on technical features and cannot effectively capture the emotional information in the voice. There are individual differences among singers, different singers may have different evaluation indicators, and there is a lack of professional data in the field of singing voice training and pronunciation. The embodiment of the present invention proposes to convert the original speech signal into a fractional-order Mel spectrogram for feature extraction. The original speech signal is usually a high-dimensional time domain signal containing a large number of sampling points. Directly processing these high-dimensional data will lead to a sharp increase in computational complexity and very high requirements for computing resources. The fractional-order Mel spectrogram converts the time domain signal into a time-frequency domain representation through time-frequency conversion, reducing the dimension of the data and making subsequent processing more efficient. In addition, it can more intuitively display the frequency components and time domain changes of the speech signal, which helps to extract more representative and robust features. The traditional CNN network mainly focuses on global features, while vocal training and pronunciation also need to consider temporal dependencies. The present invention proposes a module that extracts temporal relationships and global features of the speech signal respectively using LSTM and CNN, and performs feature fusion through the Gate_Fusion mechanism. In response to the problem that traditional models lack emotional expression, the present invention proposes a multimodal singing voice training pronunciation evaluation model, which introduces the video modality of the singer singing, extracts the facial information therein and uses the cross-attention mechanism to fuse the module with the extracted audio features.

[0093] Due to the wide variety of songs and the varying lengths of time, it's difficult to make a unified assessment. Instead, we focus on the vocal quality of the five vowels: a, o, e, i, and u. Due to the different vocal conditions between men and women, the evaluation criteria also differ. Firstly, the audio signals of the five vowels a, o, e, i, and u from male and female singers were collected respectively, and the corresponding quality levels were marked according to the intonation, pitch, timbre, rhythm and other indicators of the pronunciation. At the same time, the videos of the singers singing were collected, and the videos were divided into corresponding quality levels according to the indicators such as facial expressions and pronunciation mouth shape to construct a sample data set; secondly, the singing audio signals were converted into Mel spectrograms, fractional spectrograms and Hilbert spectrograms based on short-time fractional Fourier transform; then, a deep feature extraction network of fractional spectrograms based on LSTM and CNN was constructed, and the extracted series of deep features of fractional spectrograms were input into the Gate_Fusion module for feature fusion, and ResNet was used for facial visual feature extraction. The extracted audio features were fused with video features through the cross-attention mechanism, and the fused features were superimposed on the audio features and visual features. Finally, they were input into the classifier for classification and evaluation of the quality of singing pronunciation. The method of the present invention not only quantitatively evaluates the vocal quality of the singer, abandons subjective feelings and evaluates the vocal quality more objectively, but also incorporates the singer's emotional expression to make the classification results more in line with human judgment standards, thereby improving the efficiency and reliability of vocal quality assessment. It can be applied to auxiliary teaching of singing voice to improve the quality and efficiency of singing teaching.

[0094] The following is a detailed introduction to the technical solutions provided by the embodiments of the present invention:

[0095] 1. Singing audio collection and dataset construction

[0096] This example constructs a dataset of vowel vocalizations from teachers and students at a university conservatory of music. A total of 50 teachers and students of varying ages and genders were selected for the collection. To ensure model robustness, the same dynamic microphone and video recorder were used to capture vowel vocalizations from each subject at three different time periods: morning, noon, and evening. The audio and video clips were maintained in WAV format, and the videos in MP4 format. A total of 3,000 audio and video clips of the five vowels a, o, e, i, and u were collected. Experts from the university conservatory of music were hired to assess the audio and video quality based on metrics such as intonation, pitch, timbre, rhythm, resonance, facial expression, and articulation. The data were then labeled with a quality grade. Finally, the dataset was constructed and divided into training, validation, and test sets in a ratio of 8:1:1 (2,400:300:300).

[0097] 2. Audio feature extraction based on fractional Mel spectrogram

[0098] Spectrogram is a common form of visualization. It can transform audio signals into two-dimensional images including time, frequency, and intensity through short-time Fourier transform (STFT), making audio information more intuitive. Short-time fractional Fourier transform is a generalization of short-time Fourier transform. It not only retains the properties and characteristics of Fourier transform, but also can transform the signal into a domain between time domain and frequency domain, solving the problem that Fourier transform cannot locate time and frequency, and can obtain the time information of a certain frequency. In addition, research has found that the human ear can hear frequencies of 20~20000Hz, and the human perception of sound waves is not linear. The sensitivity to low-frequency signals is higher than that to high-frequency signals. When the frequency is above 1000Hz, the ratio between human hearing and frequency changes from proportional to logarithmic. Compared with ordinary frequency cepstrum coefficients, Mel frequency can focus on analyzing the auditory characteristics of the human ear and better distinguish people's judgment of pronunciation quality. The concept of Mel scale was first proposed in 1937, which converts the frequency in Hz into a signal in Mel. The conversion formula is shown in formula (1):

[0099] (1)

[0100] Where f is the actual frequency in Hz, Mel is the transformed Mel scale, and its Mel mapping relationship is as follows: Figure 1 shown.

[0101] In order to achieve the frequency conversion Mel scale in formula (1), the currently commonly used method is to use the Mel scale triangular filter to perform the conversion. People mainly use equal-area triangular filters for speech recognition. The center frequency of each triangular filter is Not the same, , M is the number of triangular filters, usually 40. Figure 2 It can be seen that the spacing distance of the filters becomes smaller as the center frequency value decreases, and becomes larger as the center frequency value increases.

[0102] Let the triangular filter be , the frequency response of each filter is defined as:

[0103] (2)

[0104] The fractional-order Mel spectrogram is based on the spectrogram by adding short-time fractional-order Fourier transform and Mel scaling. The specific generation method is as follows: Figure 3 shown.

[0105] 3. Improve cross-modal fusion mechanism

[0106] Cross-modal fusion is implemented based on the attention mechanism in Transformer. Since MLP cannot effectively model periodic signals, it can only achieve marginal performance in the training domain. In order to effectively analyze the periodic features in audio signals, this embodiment replaces the MLP layer in Transformer with (FAN). FAN naturally integrates periodicity into the structure and calculation process of FAN by introducing the Fourier principle. The definition of FAN layer is as follows: Figure 4 shown.

[0107] The FAN layer adds cosine and sine components to the original MLP layer, and FAN follows two principles: (1) The ability to represent Fourier coefficients is positively correlated with its depth, and (2) The output of any hidden layer can be used to model periodicity using Fourier series through subsequent layers. The first principle uses its depth to enhance the ability to model periodicity, and the second principle ensures that the functions of the intermediate layers of FAN can be used for periodicity modeling. The neural network should be expressed as f s (x), can be expressed as , and must satisfy the formula .

[0108]

[0109]

[0110]

[0111] And in order to meet the above two principles, the FAN middle layer needs to apply f in With f out The definition of the FAN layer is as follows: As shown, and is a learnable parameter, Represents the activation function.

[0112]

[0113] The attention module first performs layer normalization on the input facial features and audio spatiotemporal features, and then generates vectors q (Query), k (Key), and v (Value) respectively through linear transformation. The dot product of q and k is then calculated to obtain the attention score, and normalized by the softmax function. Dropout regularization is then applied, and the obtained attention score is used to perform weighted summation on v, and the result is added back to the original Xq through a residual connection. The periodic features are then captured through a FAN layer, and then passed through a feedforward neural network containing nonlinear activations, and the captured periodic features are again added to Xq through a residual connection, and the processed feature representation is finally output. The cross-attention module structure of this embodiment is as follows: Figure 5 shown.

[0114] The attention mechanism is based on the dot product similarity of attention in Transformer, and is specifically defined as follows As shown. For different mode characteristics Φ a With Ф v , use the learned weights to calculate q(Query) and k(Key) to get the scaled dot product similarity A.

[0115]

[0116] The Softmax function promotes competition in the attention matrix, highlighting the more important attributes of each modality, thereby providing the importance score of each key for each query, that is, the representation of modality a relative to modality b. Through this calculation, an attention vector that can represent more relevant attributes of modality a is obtained. The attention vector of the video modality can be obtained .

[0117] This method does not directly fuse the features of the two modalities. Instead, it identifies the most relevant attributes of each modality relative to the other based on similarity scores, ensuring that each modality contributes equally to the final prediction and guiding the model to learn features that are unrelated or highly consistent between the modalities. This method allows for the sharing of information between modalities while eliminating the need for strong interdependence between features of different modalities. It can well adapt to the modal differences between audio and video and better perceive periodic changes in audio and video features.

[0118] 4. Dynamic Weight Multimodal Cosine Loss Function

[0119] Multi-Modal Cosine Loss (MMCosine) is a method designed to solve the imbalanced optimization problem of single modality encoders in audio-visual fine-grained learning. The core idea of ​​the MMCosine loss function is to construct a hyperspherical feature space by performing L2 normalization on features and weights. In traditional deep learning frameworks, features are usually measured by Euclidean distance or dot product operations to measure similarity. However, in multimodal data scenarios, this approach may lead to large intra-class variability and reduce the discriminability of feature distribution. Therefore, MMCosine introduces cosine similarity calculation, so that the learned embedding features can be distributed on a modality-specific hypersphere, thereby maximizing the discriminability of L2 distance or cosine similarity between inter-modal features and weight vectors. By removing the radial changes in weights and features, MMCosine can reduce the logit differences between single modalities and coordinate the optimization process. The functional form of MMCosine is as follows: shown.

[0120]

[0121] Although MMCosine effectively addresses the issues in multimodal learning caused by unbalanced optimization of a single modality encoder by introducing cosine similarity calculation, there is still room for improvement when dealing with complex and unbalanced datasets. To further enhance the model's attention to difficult-to-classify samples and better coordinate feature representations between different modalities, this example proposes Dynamic Weight Multi-Modal Cosine Loss (DWMMCL) based on this.

[0122] DWMMCL retains the core advantages of MMCosine and adds the ability to dynamically adjust sample weights. This approach not only reduces intra-class variability and improves the discriminability of feature distributions, but also optimizes the overall performance of the model by increasing attention to difficult-to-classify samples. Specifically, DWMMCL incorporates the following improvements:

[0123] (1) Angle constraint and L2 normalization: The audio and visual feature vectors are L2 normalized so that these features are distributed on a specific hypersphere, and cosine similarity is used as the classification basis.

[0124] (2) Dynamic weight mechanism: Dynamically adjust the weight of each sample based on the confusion matrix during training, so as to more effectively handle the imbalanced data problem.

[0125] Assume there are audio vector and video vector f a With f b, and the corresponding weight vector w a and w b First, perform L2 feature normalization on the audio and video vectors, as shown in the formula shown.

[0126]

[0127] Next, calculate the cosine value of the angle between each sample and each category weight vector, as shown in the formula shown.

[0128]

[0129] The cosine similarity scores of the audio and visual features of each sample are added to get the final logit score, and multiplied by the scaling parameter s as shown in the formula shown.

[0130]

[0131] And based on the training results of the previous cycle, dynamic weights are generated , as shown in the formula shown.

[0132]

[0133] Among them, γ is a regulating factor used to control the growth rate of dynamic weights. Indicates that the true label is The samples are mistakenly classified as The number of times, c is the total number of categories.

[0134] The final defined DWMMCL loss function is as follows: As shown, N is the number of samples and n is the total number of categories.

[0135] .

[0136] 5. Specific implementation process and steps

[0137] The prediction process of the singing voice training pronunciation quality assessment in this embodiment is as follows: Figure 6As shown in the figure, first, audio signals and videos of the singer practicing the five vowels a, o, e, i, and u are collected, and the corresponding quality levels are labeled based on the singer's pronunciation, pitch, timbre, rhythm, resonance, facial expression, and mouth shape. The data is then divided into training, validation, and test sets in an 8:1:1 ratio. The original audio signal is converted into an optimal-order fractional spectrogram, and the video signal is passed through a face extraction network to extract facial information from the original video image. For the preprocessed audio and facial information, it is necessary to extract deeper features through the feature extraction network again. Two parallel LSTM networks and CNN networks are used to extract the time features and spatial features of the fractional-order spectrogram respectively. The Gate_Fusion module is used to fuse the extracted time features and spatial features to obtain the audio time-space fusion features. The ResNet network is used to extract the deep features of the facial information. In order to reduce the modal difference between audio features and video features, the cross-modal attention mechanism is used to extract the joint feature information of the two modalities, and the extracted joint feature information is added to the deep audio features and video features. The final audio features and video features are spliced ​​and input into the classifier to obtain the voice training pronunciation quality level results.

[0138] The following is a more detailed introduction to the implementation plan:

[0139] 1. Setting up the experimental environment and constructing the dataset

[0140] This experiment uses an RTX 4080 (16GB) GPU and a 12 vCPU Intel(R) Xeon(R) Platinum 8255CCPU @ 2.50GHz processor. The Python programming language is used, and the Cuda version used is 11.7. The deep learning framework used is Pytorch1.7.0.

[0141] The present invention constructs a dataset of vowel vocalizations from teachers and students from a university's music department. A total of 50 teachers and students of different ages and genders are selected as the collection subjects. To ensure model robustness, the same dynamic microphone and camera are used to capture audio and video of the five vowels a, o, e, i, and u during vocal training for different subjects in three time periods: morning, noon, and evening. The collected audio is maintained in WAV format, and a total of 3,000 audio and video clips of the five vowels are collected. Professors from a university's music school are hired to label the corresponding quality levels based on vocalization indicators such as pitch, timbre, rhythm, resonance, facial expression, and lip shape. This data is then divided into training, validation, and test sets in a ratio of 8:1:1, that is, 2400:300:300.

[0142] 2. Fractional Mel Spectrogram Generation

[0143] Fractional Mel spectrograms address the problem that traditional spectrograms can only reflect the time-frequency characteristics of audio signals from a single perspective. However, the most relevant time-frequency characteristics vary for different tasks. This embodiment sets the order of the fractional Mel spectrogram to 0-1.0 and adaptively selects the fractional order by setting the parameters to learnable parameters. Experimental results show that adaptively adjusting the short-time fractional Fourier transform parameters can achieve optimal experimental results. The specific experimental results are shown in Table 1.

[0144] Table 1 Experimental results of fractional Mel spectrogram

[0145] Order Accuracy Order Accuracy 0 86.34 0.6 85.17 0.1 86.18 0.7 86.84 0.2 88.71 0.8 86.94 0.3 81.58 0.9 86.75 0.4 84.21 1.0 77.63 0.5 88.82 Adaptive order 90.13

[0146] 3. Model Training

[0147] The present invention adopts the dynamic weight multimodal cosine loss function and training accuracy as indicators to judge the model training effect. The calculation formula of training accuracy is shown in formula (20), where TP is the number of samples correctly predicted as positive, TN is the number of samples correctly predicted as negative, FP is the number of samples incorrectly predicted as positive, and FN is the number of samples incorrectly predicted as negative. The loss and accuracy curves of model training are shown in Figure 20. Figure 7 shown.

[0148] (20)

[0149] In order to further evaluate the performance of the model, this paper also uses precision (PRE), recall rate (REC) and F1 score as evaluation indicators. The calculation formulas of each indicator are shown in formulas (21)(22)(23).

[0150] (twenty one)

[0151] (twenty two)

[0152] (twenty three)

[0153] In order to verify the effectiveness of the method of this embodiment, the designed ablation experiments are shown in Table 2. Among them, ① is the method using a single-order fractional-order spectrogram, feature splicing fusion, and cross-loss entropy function. The experimental accuracy of this method is 89.27%, the recall rate is 93.25%, and the F1 score is 90.23%. ② is the method using fractional-order Mel spectrogram, feature splicing fusion, and cross-loss entropy function. The experimental accuracy of this method is 91.41%, the recall rate is 94.24%, and the F1 score is 92.27%. ③ is the method using fractional-order Mel spectrogram, improved c cross-modal attention fusion, and cross-loss entropy function. The experimental accuracy of this method is 92.82%, the recall rate is 95.36%, and the F1 score is 93.57%. ④ is the method using adaptive fractional-order spectrogram, cross-modal attention fusion, and dynamic cosine loss entropy function. The experimental accuracy of this method is 94.71%, the recall rate is 94.18%, and the F1 score is 94.34%.

[0154] Table 2 Ablation experiment results

[0155] index Accuracy Recall F1 ① 89.27 93.25 90.23 ② 91.41 94.24 92.27 ③ 92.82 95.36 93.57 ④ 94.81 94.18 94.34

[0156] In this experiment, a series of ablation experiments were conducted to verify the impact of different technical components on model performance. First, the effects of using a single-order fractional-order spectrogram (method ①) and a fractional-order Mel spectrogram (method ②) were compared. The results showed that the method using the fractional-order Mel spectrogram achieved 91.41% precision, 94.24% recall, and 92.27% F1 score, respectively, compared to 89.27%, 93.25%, and 90.23% of method ①, all of which were significant improvements. This shows that the adaptive fractional-order spectrogram can more effectively extract feature information from speech signals and improve classification performance.

[0157] Secondly, in terms of feature fusion strategy, we replaced simple feature concatenation (method ②) with a cross-modal attention mechanism (method ③). The results showed that this adjustment further improved model performance, achieving 92.82% accuracy, 95.36% recall, and 93.57% F1 score. This shows that the cross-modal attention mechanism can more accurately capture and integrate key information in multimodal data, enhancing the model's focus on important features, thereby improving overall performance.

[0158] Finally, this example examines the impact of the choice of loss function on model performance. Replacing the cross-entropy loss function with the dynamic cosine loss entropy function (method 4) resulted in a slight decrease in recall to 94.18%, but the precision and F1 score increased to 94.71% and 94.34%, respectively. This suggests that the dynamic cosine loss entropy function may be more suitable for the current data distribution, helping to optimize classification boundaries and thus improve classification accuracy.

[0159] Based on the above analysis, adaptive fractional-order spectrogram, cross-modal attention fusion and dynamic cosine loss entropy function have enhanced the overall performance of the model to varying degrees, demonstrating the effectiveness of these methods in improving the pronunciation quality evaluation task.

[0160] In summary, the solution provided by the embodiment of the present invention converts the speech signal into a fractional-order Mel spectrogram that can better reflect the time-frequency characteristics of the audio signal and is more in line with the human ear's auditory effect. The xLSTM model and the CNN model are used to extract feature spectrogram information respectively, and the Gate_Fusion mechanism is used to fuse dual audio features, enriching the representation of the temporal relationship and frequency characteristics of audio features. The MCTNN and ResNet networks are used to deeply extract the video face features. Through the multimodal feature fusion learning of audio features and video features, the cross-attention mechanism is used to extract the dependency relationship between audio features and video features. The visual face features are used to supplement the emotional features and lip shape features missing in the audio features, thereby achieving effective expression of emotional information in vocal pronunciation and making the model more in line with the manual judgment standard. The method of the present invention can quantitatively evaluate the vocal quality of the singer, combines the emotional expression during singing, makes the evaluation criteria more in line with human subjective feelings, improves the efficiency and reliability of vocal quality assessment, and can be applied to auxiliary teaching of vocal training to improve the quality and efficiency of vocal teaching.

[0161] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0162] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0165] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of a vocal training vowel pronunciation quality assessment method that integrates auditory, visual, temporal and spatial features under the inspiration of the present invention. All equal changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features, characterized in that: include: Collect the singer's vowel pronunciation audio signals and corresponding videos to construct a multimodal dataset annotated with pitch, timbre, rhythm, resonance, facial expression and lip shape quality grades; Generating a fractional Mel spectrogram by performing adaptive-order short-time fractional Fourier transform on the audio signal; Extracting the temporal features and spatial features of the fractional-order Mel spectrogram, and fusing them to generate audio spatiotemporal features through a gating mechanism; Extracting facial visual features from the video and fusing them with the audio spatiotemporal features through a cross-attention mechanism, wherein the cross-attention mechanism integrates a periodic modeling network; the periodic modeling network simultaneously integrates a periodic feature modeling component and a transient feature preservation component in a hidden layer; The fused features are input into the classifier, and the model is trained using a dynamic weighted multimodal cosine loss function. The dynamic weighted multimodal cosine loss function dynamically adjusts the sample weights through a confusion matrix, increasing the weight of misclassified samples based on their historical misclassification frequency. Output pronunciation quality assessment results.

2. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory and visual temporal and spatial features according to claim 1, characterized in that: The periodic modeling network is a Fourier analysis network, whose hidden layer output is generated by splicing cosine transform components, sine transform components and nonlinear activation components. The periodic characteristics of the acoustic signal are modeled by the cosine transform components and sine transform components, and the time domain non-stationary characteristics are retained by the nonlinear activation components.

3. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: In the dynamic weighted multimodal cosine loss function, the weight increment of the misclassified sample is proportional to the frequency with which it is misclassified into the predicted category.

4. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, wherein: The dynamic weighted multimodal cosine loss function fuses audio and visual features through cosine similarity, accumulating the number of misclassifications; is updated at the end of each training cycle and stored in a confusion matrix; and the growth rate of the weight increment is controlled by an adjustment factor.

5. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: The confusion matrix is ​​updated according to the training cycle and stores the historical records of each sample being misclassified; wherein the update operation is performed at the end of each training cycle, and the historical records include the cumulative number of times the true category of the sample is misclassified as the predicted category.

6. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: The optimization method of the adaptive order is: Initialize the fractional order to a learnable parameter in the interval [0, 1.0]; The order value is dynamically adjusted during training via backpropagation.

7. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: The gating mechanism dynamically balances temporal and spatial features through trainable weights.

8. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: The cross-attention mechanism generates cross-modal joint features through bidirectional query-key-value interaction, where the query vector originates from one modality and the key-value vector originates from the other modality; and balances the attention score gradients through a scaling factor d to prevent softmax saturation.

9. The method for evaluating the pronunciation quality of vowels in vocal training by integrating auditory, visual, temporal and spatial features according to claim 1, characterized in that: The vowel types are a, o, e, i, and u.

10. A vocal training pronunciation quality assessment system, characterized in that: include: (a) a multimodal data acquisition device, comprising a microphone array and a high-definition camera, for collecting the singer's vowel pronunciation audio signals and facial video stream in real time; (b) A fractional-order signal processor configured as: performing an adaptive-order short-time fractional Fourier transform on the audio signal; Compress frequency domain information through Mel-scale filter banks; Output time-frequency feature map to feature fusion gateway; (c) A spatiotemporal feature fusion engine, which includes a parallel LSTM timing processing unit and a CNN convolutional processing unit, generating an audio spatiotemporal feature vector through a gated weighted circuit; (d) Cross-modal decision module, including: ResNet visual feature extractor to parse facial action units; Fourier-enhanced cross-attention kernel to fuse audio and visual features; Dynamic weight classifier, which updates sample weights based on the confusion matrix; (e) Feedback execution terminal receives the quality assessment results and matches the preset acoustic correction scheme based on the assessment results.

Images