Vocal music training method and system based on deep learning
By collecting vocal training data through multi-source sensors and analyzing it using deep learning models, real-time feedback information and personalized training plans are generated, which solves the problems of geographical limitations and low efficiency in traditional vocal training and realizes personalized teaching and efficient learning.
Patent Information
- Application Number
- CN202510976331.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-15
- Publication Date
- 2025-10-17
AI Technical Summary
Traditional vocal training has problems such as geographical limitations, low learning efficiency and difficulty in achieving personalized teaching.
Multi-source sensors are used to collect vocal training data, and deep learning models are used to analyze audio, facial expressions, and physiological signals to generate real-time feedback information, dynamically adjust training difficulty, and generate personalized training plans.
It achieves precise and personalized feedback, improves learning efficiency, dynamically adjusts training difficulty, forms a virtuous circle, and promotes the continuous improvement of vocal training effects.
Smart Images
Figure CN120808734A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of vocal training, in particular to a vocal training method and system based on deep learning. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, its application in the field of vocal learning is increasingly widespread, bringing revolutionary changes to traditional vocal teaching. Traditional vocal training usually relies on on-site guidance by teachers, and learners mainly rely on feedback from teachers to adjust in terms of pitch, rhythm, emotional expression, etc. This approach has regional limitations, low learning efficiency, and difficulty in achieving personalized teaching. SUMMARY
[0003] To solve the above technical problems, the present application provides a vocal training method and system based on deep learning. The present application can effectively solve the problems of regional limitations, low learning efficiency, and difficulty in achieving personalized teaching. The specific scheme is as follows:
[0004] On the one hand, the present application provides a vocal training method based on deep learning, comprising:
[0005] S1, collecting vocal training data through multi-source sensors, the vocal training data including audio signals, facial expression images, and physiological signals;
[0006] S2, preprocessing the vocal training data, extracting acoustic features, expression features, and physiological features, and constructing a multi-modal feature vector;
[0007] S3, inputting the multi-modal feature vector into a pre-trained deep learning model, the model analyzing acoustic features through a convolutional neural network, analyzing physiological feature time series changes through a time recurrent network, and fusing multi-modal feature vectors through a cross-modal attention mechanism;
[0008] S4, generating real-time feedback information based on the output of the deep learning model, including pitch correction suggestions, breathing rhythm guidance, and emotional expression intensity scores;
[0009] S5, dynamically adjusting the training difficulty according to the real-time feedback information, generating a personalized training plan, and updating the parameters of the deep learning model.
[0010] Further, the audio signal in step S1 includes a singing fundamental frequency, a harmonic component, and a dynamic range, wherein the singing fundamental frequency is the basic frequency of the sound, the harmonic component is an integer multiple of the singing fundamental frequency, and the dynamic range represents the amplitude of the change in sound intensity;
[0011] The facial expression image captures muscle micro-movements through an infrared camera and extracts muscle unit activation intensity from the Facial Action Coding System.
[0012] The physiological signals include heart rate variability and galvanic skin response, which are collected in real time by a wearable device.
[0013] Further, the deep learning model training method in step S3 comprises:
[0014] A labeled data set is constructed, which includes score labels of professional vocal music teachers, emotional state labels and physiological indicator thresholds;
[0015] A multi-task learning framework is adopted to simultaneously optimize pitch prediction, emotion classification and physiological feature anomaly detection tasks.
[0016] Further, step S4 comprises:
[0017] A virtual spectrogram is superimposed in the visualization interface to mark the pitch deviation area;
[0018] Real-time beat prompts are provided through bone conduction earphones;
[0019] A three-dimensional virtual tutor animation is generated to simulate mouth shape and breathing action demonstration.
[0020] Further, step S5 comprises:
[0021] If the pitch deviation rate of continuous singing for three times is lower than the threshold value, the target vocal range is increased and the ornamentation complexity is increased;
[0022] If physiological feature anomalies are detected, the system automatically switches to a relaxation training mode and reduces the accompaniment speed.
[0023] Further, the pitch deviation rate is a key indicator for measuring the singing accuracy of learners, which is calculated by comparing the difference between the actual singing pitch and the standard pitch, and the calculation formula is:
[0024]
[0025] In the formula, f 实际 represents the actual singing pitch, and f 标准 represents the standard pitch.
[0026] On the other hand, the present application provides a vocal music training system based on deep learning, comprising:
[0027] A data acquisition module is used to acquire vocal music training data, including a high-sensitivity microphone array, an infrared camera and a wearable physiological sensor;
[0028] A feature extraction module is used to calculate Mel frequency cepstral coefficients (MFCC), facial muscle movement unit intensity and heart rate variability index;
[0029] A deep learning analysis module is configured to analyze the input multi-modal feature vector, and includes a multi-modal fusion neural network and an adaptive parameter updating unit.
[0030] A feedback execution module is integrated with a visual interface, an audio synthesizer and a training plan generator.
[0031] Further, the deep learning analysis module adopts a hierarchical architecture.
[0032] The bottom layer processes acoustic features through a one-dimensional convolutional neural network.
[0033] The middle layer models the time sequence dependence of physiological signals through a bidirectional time recurrent network.
[0034] The top layer fuses acoustic features, expression features and physiological features through an attention weight matrix to output multi-dimensional evaluation results.
[0035] The present application has the beneficial effect of introducing physiological signal threshold interference and enhanced feedback into a deep learning model, and generating feedback information in real time, which can achieve precise personalized feedback. At the same time, the training difficulty is dynamically adjusted to generate a personalized training plan, and the feedback information is used to update the parameters of the deep learning model, so that the model can continuously learn and adapt to the characteristics and training needs of different learners, further improve the accuracy of evaluation and feedback, improve learning efficiency, form a virtuous cycle, and continuously improve the effect of vocal training. BRIEF DESCRIPTION OF DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description.
[0037] Figure 1 A flowchart of a vocal training method based on deep learning according to the present application;
[0038] Figure 2 A module schematic diagram of a vocal training system based on deep learning according to the present application. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme of the present application will be described clearly and completely below in combination with specific embodiments and drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of protection.
[0040] In recent years, the application of deep learning technology in vocal training has gradually emerged. Convolutional neural network (CNN) has been widely used in acoustic audio field due to its powerful feature extraction capability, which can effectively analyze audio signals and extract acoustic features such as pitch and timbre. Time recurrent network such as long short-term memory network (LSTM) is good at processing sequence data and can be used to analyze the timing changes of physiological signals such as breathing rhythm. The development of multi-source sensor technology provides convenience for the collection of vocal training data, which can simultaneously collect multi-modal data such as audio signals, facial expression images and physiological signals. Through the fusion analysis of these data, the vocal performance of learners can be more comprehensively evaluated.
[0041] Please refer to Figure 1 The present application provides a vocal training method based on deep learning, comprising:
[0042] S1, collecting vocal training data through multi-source sensors, the vocal training data including audio signals, facial expression images and physiological signals.
[0043] Further, the audio signals include singing fundamental frequency, harmonic components and dynamic range; the singing fundamental frequency is the basic frequency of sound, which determines the pitch; the harmonic components are integer multiples of the fundamental frequency, which enrich the timbre of sound; the dynamic range reflects the change amplitude of sound intensity, which embodies the intensity change and emotional fluctuation in singing. These audio signal features are captured in real time during the vocal training process through audio acquisition devices.
[0044] Further, the facial expression images capture the muscle micro-movement through infrared cameras and extract the muscle unit activation intensity of the facial action coding system (FACS). The infrared camera captures the slight movement of the learner's facial muscles, based on the facial action coding system (FACS), which decomposes the facial muscle movement into different muscle units and quantifies the activation intensity of each muscle unit.
[0045] Further, the physiological signals include heart rate variability (HRV) and galvanic skin response (GSR), which are collected in real time through wearable physiological sensors. The heart rate variability (HRV) reflects the regulation function of the autonomic nervous system on the heart, which can reflect the psychological state stability of the learner during singing; the galvanic skin response (GSR) is mainly related to the sympathetic nervous activity, which can reflect the emotional arousal level. Wearable physiological sensors such as smart bracelets, smart watches, etc. have built-in corresponding biological sensors, which can collect physiological signals in real time and continuously, and send data to the deep learning model through wireless transmission technology, so as to monitor and analyze the physiological state of the learner in real time.
[0046] S2, pre-processing the vocal training data, extracting acoustic features, expression features and physiological features, and constructing multi-modal feature vectors.
[0047] Specifically, the vocal training data is preprocessed and feature extracted, acoustic features, expression features and physiological features are extracted, and a multi-modal feature vector is constructed.
[0048] S3, input the multi-modal feature vector into a pre-trained deep learning model, the model analyzes the acoustic features through a convolutional neural network (CNN), analyzes the physiological feature time sequence changes through a time recurrent network (LSTM), and fuses the multi-modal feature vector through a cross-modal attention mechanism.
[0049] Further, the deep learning model is trained in the following manner:
[0050] An annotated data set is constructed, including score labels of professional vocal teachers, emotional state labels and physiological index thresholds;
[0051] A multi-task learning framework is used to simultaneously optimize the pitch prediction task, the emotion classification task and the physiological feature anomaly detection task.
[0052] Further, the score label includes pitch, rhythm, emotional expression, etc., which can directly reflect the quality of singing; the emotional state label is determined by analyzing the emotional style of the sung works and the performance of the learner during singing; the physiological index threshold is obtained by analyzing the physiological data of a large number of professional vocalists and learners, and is used to determine whether the physiological features are within the normal range, for example, setting the normal range threshold of heart rate variability (HRV), when the learner's HRV exceeds this range, it may indicate that the learner's psychological state is unstable, affecting the singing performance.
[0053] In the present application, the pitch prediction task accurately determines whether the learner's singing pitch is accurate by analyzing acoustic features; the emotion classification task identifies the emotional state conveyed by singing by combining acoustic features and expression features; the physiological feature anomaly detection task mainly relies on physiological features to timely detect physiological abnormalities of learners during singing. During the training process, the model optimizes the joint loss function of multiple tasks, and each task cooperates and supplements each other.
[0054] S4, based on the output of the deep learning model, real-time feedback information is generated, including pitch correction suggestions, breathing rhythm guidance and emotional expression intensity scores.
[0055] Further, the real-time feedback information can be presented in the following ways: superimposing a virtual spectrogram in a visual interface, marking the pitch deviation area; providing real-time beat prompts through bone conduction earphones; generating a three-dimensional virtual tutor animation to simulate mouth shape and breathing action demonstration.
[0056] Further, the visualization interface is a window for the learner to interact with the deep learning model. The virtual spectrogram visually presents the audio frequency spectrum information of the singing, showing the intensity distribution and changes of different frequency components. On the spectrogram, the areas of pitch deviation are marked by predetermined color annotations or line marks, such as when the learner's pitch is too high, the corresponding frequency spectrum area is marked with a red line; when the pitch is too low, it is marked with a blue line.
[0057] The bone conduction earphone is an audio device that directly transmits sound signals to the inner ear through vibration of the skull. In vocal training, the bone conduction earphone provides real-time beat cues to the learner, such as stable beat sounds or rhythm cues, so that the learner can adjust his or her singing speed and rhythm according to the beat cues.
[0058] The three-dimensional virtual tutor animation uses computer graphics technology to create a highly realistic virtual tutor image. The virtual tutor can simulate the mouth movements and breathing movements of a real vocal teacher, and according to the learner's singing situation, it can display the standard mouth shape and breathing method in real time.
[0059] S5, dynamically adjusting the training difficulty according to the real-time feedback information, generating a personalized training plan, and updating the parameters of the deep learning model.
[0060] Further, the rules for dynamically adjusting the training difficulty include:
[0061] If the pitch deviation rate of the learner's singing for three consecutive times is lower than the threshold, the target vocal range is increased and the ornamentation complexity is increased.
[0062] If an abnormal physiological feature is detected (such as a sudden drop in HRV), the system automatically switches to a relaxation training mode and reduces the accompaniment speed.
[0063] Wherein, the pitch deviation rate is a key indicator to measure the accuracy of the learner's singing pitch, which is calculated by comparing the difference between the actual singing pitch and the standard pitch. The calculation formula is as follows:
[0064]
[0065] When the learner's pitch deviation rate for three consecutive times is lower than the set threshold, it means that he or she has certain stability in pitch, and at this time the deep learning model will automatically increase the target vocal range, such as from the current tenor vocal range to the tenor-high vocal range, to increase the challenge and pertinence of the training, and further improve the learner's vocal level.
[0066] During the singing process, if the physiological characteristics of the learner are detected to have abnormal conditions, such as a sudden drop in heart rate variability (HRV), which may mean that the learner is in a state of excessive tension or anxiety, affecting his singing performance and training effect. At this time, the deep learning model will automatically switch to a relaxation training mode, playing soothing music or guiding deep breathing and other relaxation exercises, while reducing the accompaniment speed to reduce the psychological pressure and singing difficulty of the learner, so that he can gradually recover to a more relaxed state and continue effective vocal training.
[0067] Please refer to Figure 2 The present application provides a vocal training system based on deep learning, comprising:
[0068] A data acquisition module for acquiring vocal training data through multi-modal sensors, including a high-sensitivity microphone array, an infrared camera, and a wearable physiological sensor;
[0069] A feature extraction module for calculating Mel-frequency cepstral coefficients (MFCC), facial muscle movement unit intensity, and heart rate variability index, and constructing a multi-modal feature vector;
[0070] A deep learning analysis module for analyzing the input multi-modal feature vector, including a multi-modal fusion neural network and an adaptive parameter update unit;
[0071] A feedback execution module integrating a visualization interface, an audio synthesizer, and a training plan generator;
[0072] Among them, the present system realizes the real-time evaluation and teaching intervention of vocal performance through an end-to-end pipeline.
[0073] Further, the deep learning analysis module adopts a hierarchical architecture:
[0074] The bottom layer processes acoustic features through one-dimensional CNN;
[0075] The middle layer models the temporal dependence of physiological signals through bidirectional LSTM;
[0076] The top layer fuses acoustic, expression, and physiological features through an attention weight matrix to output multi-dimensional evaluation results.
[0077] Further, the feedback execution module supports augmented reality (AR) display, including:
[0078] Superimposing a virtual vocal cord vibration simulation graph in a real environment;
[0079] Positioning the pitch deviation direction through spatial audio technology;
[0080] According to the learning progress, dynamically rendering the difficulty level identifier.
[0081] The beneficial effects of the present application are: the vocal training data of the learner is comprehensively collected by the multi-source sensor, and the multi-modal feature vector is extracted and input into the deep learning model, and the physiological signal threshold interference and enhanced feedback are introduced into the deep learning model, the analysis of acoustic characteristics by CNN can accurately judge the pitch problem; the analysis of the time sequence change of physiological signals by LSTM can monitor the breathing rhythm in real time, guide the learner to adjust the breathing, and make it match the singing demand; the analysis of the facial expression image can evaluate the emotional expression intensity, help the learner better understand the song emotion and express it, and realize accurate personalized feedback.
[0082] Secondly, the feedback information is generated based on the real-time output of the deep learning model, the learner can understand the problems and make adjustments in the singing process, and the real-time interactivity of the training is enhanced. At the same time, the training difficulty is dynamically adjusted, and the personalized training plan is generated, such as increasing the corresponding exercise content for the weak links of the learner, avoiding the problem of fixed difficulty and lack of pertinence in traditional training, so that the learner gradually improves the vocal skills under the training intensity suitable for himself. The feedback information is used to update the parameters of the deep learning model, so that the model can continuously learn and adapt to the characteristics and training needs of different learners, further improve the accuracy of evaluation and feedback, form a virtuous cycle, and promote the continuous improvement of the effect of vocal training.
[0083] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Based on the embodiments of the present application, those skilled in the art can make formal modifications to the technical solutions described in the following embodiments or replace some technical features under the premise of not making creative labor, and all other embodiments obtained are within the protection scope of the present application.
Claims
1. A vocal training method based on deep learning, comprising the following steps: S1. Collecting vocal training data through multi-source sensors, wherein the vocal training data includes audio signals, facial expression images, and physiological signals; S2. Preprocessing the vocal training data to extract acoustic features, expression features, and physiological features to construct a multimodal feature vector; S3. Input the multimodal feature vector into the deep learning model. The deep learning model analyzes the acoustic features through a convolutional neural network, analyzes the temporal changes of physiological features through a time recursive network, and fuses the multimodal feature vectors through a cross-modal attention mechanism. S4. Generating real-time feedback information based on the output of the deep learning model, wherein the real-time feedback information includes pitch correction suggestions, breathing rhythm guidance, and emotional expression intensity scores; S5. Dynamically adjust the training difficulty according to the real-time feedback information, generate a personalized training plan, and update the parameters of the deep learning model.
2. A vocal training method based on deep learning according to claim 1, characterized in that, In step S1, the audio signal is captured in real time by an audio acquisition device, including the fundamental frequency of the singing, harmonic components, and dynamic range; wherein the fundamental frequency of the singing is the basic frequency of the sound, the harmonic components are integer multiple frequencies of the fundamental frequency of the singing, and the dynamic range represents the amplitude of change in the strength of the sound; The facial expression image is captured by an infrared camera to capture muscle micro-movements, and the muscle unit activation intensity of the facial action coding system is extracted; The physiological signals are collected in real time through wearable physiological sensors, including heart rate variability and skin electrical response.
3. A vocal training method based on deep learning according to claim 1, characterized in that, The deep learning model training method in step S3 includes: Constructing a labeled dataset, wherein the labeled dataset includes score labels, emotional state labels, and physiological indicator thresholds of professional vocal teachers; A multi-task learning framework is used to simultaneously optimize the pitch prediction task, emotion classification task, and physiological feature anomaly detection task.
4. A vocal training method based on deep learning according to claim 1, characterized in that, Generating real-time feedback information in step S4 includes: Overlay a virtual spectrogram in the visualization interface and mark the pitch deviation area; Provides real-time beat cues via bone conduction headphones; Generate 3D virtual tutor animations to simulate lip movements and breathing demonstrations.
5. A vocal training method based on deep learning according to claim 1, characterized in that, The dynamic adjustment of the training difficulty in step S5 includes: If the pitch deviation rate of three consecutive performances is lower than the threshold, the target range will be increased and the complexity of the ornaments will be increased; If abnormal physiological characteristics are detected, it will automatically switch to relaxation training mode and reduce the accompaniment speed.
6. A vocal training method based on deep learning according to claim 5, characterized in that, The pitch deviation rate is a key indicator for measuring the accuracy of a learner's singing pitch. It is calculated by comparing the difference between the actual singing pitch and the standard pitch. The calculation formula is: Where, f 实际 is the actual singing pitch, f 标准 Indicates standard pitch.
7. A vocal training system based on deep learning, characterized in that: include: The data acquisition module is used to collect vocal training data and includes a high-sensitivity microphone array, an infrared camera, and wearable physiological sensors; Feature extraction module, used to calculate Mel-frequency cepstral coefficients (MFCC), facial muscle motor unit strength and heart rate variability index, and construct multimodal feature vectors; Deep learning analysis module, used to analyze the input multimodal feature vector, including multimodal fusion neural network and adaptive parameter update unit; Feedback execution module, integrating visual interface, audio synthesizer and training plan generator.
8. A vocal training system based on deep learning according to claim 7, characterized in that: The deep learning analysis module adopts a hierarchical architecture: The bottom layer processes acoustic features through a one-dimensional convolutional neural network; The middle layer models the temporal dependency of physiological signals through a bidirectional time recurrent network; The top layer fuses acoustic features, facial features, and physiological features through the attention weight matrix to output multi-dimensional evaluation results.