Vocalization position determination method and device, and vocal music practice system
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2024-12-02
- Publication Date
- 2026-06-05
Smart Images
Figure CN119811173B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, and vocal practice system for determining vocal position. Background Technology
[0002] Vocal placement refers to the location where airflow is obstructed and sound is produced during phonation; it is also the dominant resonance location. Vocal placement can be further subdivided into different resonance areas, such as the head cavity, nasal cavity, pharynx, oral cavity, laryngeal cavity, and chest cavity. Vocal placement offers a degree of flexibility and adjustability. When singing or speaking, the vocal placement can be adjusted as needed to achieve different timbre and tone.
[0003] Currently, the method relies heavily on the subjective judgment of experienced individuals to determine the vocal placement. This involves observing the pitch, timbre, and facial muscles of the vocalist to guess the vocal placement. However, this method is highly dependent on personal experience and subjective feelings, which can easily lead to inaccuracies in the determined vocal placement. Summary of the Invention
[0004] This invention provides a method, apparatus, and vocal practice system for determining vocal position, in order to overcome the deficiencies in the prior art.
[0005] This invention provides a method for determining the sound emission location, comprising the following steps:
[0006] Acquire the speech and physiological signals of the speaker;
[0007] The speech signal is used to locate the sound source and determine the first sound source location;
[0008] Based on the vocalization location determination model, the second vocalization location is determined by applying the speech signal and the physiological signal.
[0009] The voice position of the person making the voice is determined based on the first voice position and the second voice position.
[0010] According to a method for determining vocalization location provided by the present invention, the step of determining a second vocalization location by applying the speech signal and the physiological signal includes:
[0011] Speech features are extracted from the speech signal;
[0012] Physiological features are extracted from the physiological signals;
[0013] The second vocalization location is determined based on the spoken features and the physiological features.
[0014] According to a method for determining vocalization location provided by the present invention, determining the second vocalization location based on the speech features and the physiological features includes:
[0015] The speech features and the physiological features are spliced together;
[0016] The second sound source location is determined based on the spliced features.
[0017] According to the method for determining the location of a sound emission provided by the present invention, the speech signal is acquired by a microphone array;
[0018] The step of locating the sound source of the speech signal and determining the first sound source location includes:
[0019] Based on the signal differences between the microphones in the microphone array, the voice signals collected by each microphone are enhanced to obtain the target signal of each microphone;
[0020] Based on the target signals from each microphone, the sound source is located to determine the first sound-emitting position.
[0021] According to a method for determining the vocal position provided by the present invention, the vocal position determination model is selected from a model library based on the attribute characteristics of the vocal person, and the model library stores vocal position determination models corresponding to different attribute characteristics.
[0022] According to a method for determining the location of a sound emission provided by the present invention, the physiological signal includes body activity signal and / or temperature distribution signal.
[0023] According to a method for determining the vocal position provided by the present invention, after determining the vocal position of the vocal person, the method further includes:
[0024] The speech signal is subjected to pitch detection to obtain pitch detection results;
[0025] Based on the pitch detection results and the vocal position of the speaker, the vocal accuracy of the speaker is determined.
[0026] The present invention also provides a sound emission location determination device, comprising the following modules:
[0027] The acquisition unit is used to acquire the speech signal and physiological signal of the speaker;
[0028] A positioning unit is used to locate the sound source of the speech signal and determine the first sound source location;
[0029] The prediction unit is used to determine the second vocalization position based on the vocalization position determination model, by applying the speech signal and the physiological signal.
[0030] The determining unit is used to determine the voice position of the person making the voice based on the first voice position and the second voice position.
[0031] The present invention also provides a vocal practice system, comprising:
[0032] Microphone array, camera, infrared sensor, and the sound source location determination device as described above;
[0033] The microphone array is used to collect the voice signal, the camera is used to collect the body activity signal in the physiological signal, and the infrared sensor is used to collect the temperature distribution signal in the physiological signal.
[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sound emission location determination method as described above.
[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sound location determination method as described above.
[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sound location determination method as described above.
[0037] The vocal position determination method, device, and vocal practice system provided by this invention can accurately detect a few special vocal positions by locating the sound source of the speech signal. At the same time, the vocal position determination model can accurately detect the vocal position of new vocalists based on speech. Thus, by combining the first vocal position and the second vocal position, the vocal position of the vocalist can be further accurately obtained. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating the method for determining the sound emission location provided by the present invention.
[0040] Figure 2 This is a flowchart illustrating an implementation of step 130 in the sound emission location determination method provided by the present invention.
[0041] Figure 3 This is a flowchart illustrating an implementation of step 133 in the sound emission location determination method provided by the present invention.
[0042] Figure 4 This is a flowchart illustrating an implementation of step 120 in the sound emission location determination method provided by the present invention.
[0043] Figure 5 This is a flowchart illustrating another method for determining the sound emission location provided by the present invention.
[0044] Figure 6 This is a schematic diagram of the sound emission position determination device provided by the present invention.
[0045] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0047] In vocal training, the teacher's intuition is crucial for determining a student's vocal placement and accuracy. However, this reliance on auditory perception makes the assessment heavily influenced by the teacher's personal experience and subjective feelings, leading to subjective evaluations. Different teachers may use varying evaluation criteria and preferences, resulting in drastically different feedback for the same student. Furthermore, individual physiological differences (such as cavity size and shape) lead to diverse vocal techniques and outcomes. Current vocal training strategies often follow standardized frameworks and models, failing to accurately address each student's individual needs and limiting optimal teaching effectiveness. The specialized terminology and theoretical concepts used in vocal instruction are often complex and abstract, posing a challenge for beginners. The lack of intuitive and concrete explanations and demonstrations makes it difficult for students to effectively identify and correct specific vocal problems. Moreover, hiring experienced vocal teachers incurs significant financial costs, placing a substantial burden on students with limited resources. At the same time, vocal music teachers are relatively scarce, and the fixed teaching time and location limit the accessibility and flexibility of vocal music training services, making it difficult to meet the diverse needs of students.
[0048] In response, this invention provides a method for determining vocal position. By integrating multiple sensors (sound sensor, visual sensor, and thermal imaging sensor), it captures and records the student's vocal process in real time and performs objective and quantitative analysis. This helps reduce the bias caused by the subjective judgment of vocal teachers and provides vocal learners with more engaging, convenient, and personalized training.
[0049] Figure 1 This is a flowchart illustrating the sound emission location determination method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.
[0050] Step 110: Obtain the speech signal and physiological signal of the person making the sound.
[0051] Here, the person making the sound is the one whose voice is being produced. A speech signal is the sound signal emitted by the person making the sound, which can be produced by the vibration of the vocal cords and the resonance of the vocal cavities (i.e., the head cavity, nasal cavity, pharynx, oral cavity, larynx, chest cavity, etc.). Speech signals are typically acquired using a microphone or other acoustic sensors.
[0052] Physiological signals refer to bodily signals directly related to the vocalization process. These signals reflect physiological activities during vocalization, such as vocal cord movement, muscle activity, and airflow generation. Physiological signals can include body activity signals and temperature distribution signals. Body activity signals, which can be acquired via a camera, characterize changes in specific parts of the vocalizer's body during vocalization, including oral cavity expansion or contraction and muscle changes. Temperature distribution signals, which can be acquired via infrared sensors, characterize the temperature distribution in different parts of the vocalizer's body during vocalization. For example, when the vocalization location is the larynx, the contraction of laryngeal muscles during vocalization leads to an increase in temperature, while capillary blood flow accelerates.
[0053] Step 120: Locate the sound source of the speech signal and determine the first sound source location.
[0054] Specifically, speech signals possess unique spectral characteristics, which are influenced by the vocal body (such as the human vocal cords, larynx, nasal cavity, and oral cavity) and resonating cavities. By locating the sound source, the spectral characteristics of the speech signal can be analyzed, thereby inferring the characteristics of the vocal body and the type of resonating cavity.
[0055] For example, in a multi-microphone array, the arrival time of the speech signal at different microphones will have varying degrees of delay, known as the time difference of arrival. By analyzing the time difference of arrival, the direction of the speech signal's source can be determined. Furthermore, by combining this with the acoustic characteristics of the resonant cavity, the specific location of the resonant cavity, i.e., the first sound-emitting position, can be deduced. For instance, a resonant cavity has specific resonant frequencies. When the frequency of the speech signal matches these resonant frequencies, resonance occurs, amplifying the speech signal. Through sound source localization, the resonant frequencies of the speech signal can be analyzed to determine the type of resonant cavity, thereby locating the first sound-emitting position.
[0056] Before locating the sound source of the speech signal, the speech signal can be preprocessed, such as denoising, filtering, and enhancement, to improve the clarity and accuracy of the speech signal.
[0057] Step 130: Based on the vocalization location determination model, the second vocalization location is determined by applying speech signals and physiological signals.
[0058] Specifically, speech signals are used to acoustically characterize the features of the sound production process. For example, the fundamental frequency is the frequency of vocal cord vibration, reflecting the speaker's pitch. Formants are produced by the filtering effect of the vocal tract shape on the sound; different vocal tract shapes produce different formant characteristics, thus determining the timbre of the sound.
[0059] Physiological signals are used to characterize the features of the vocalization process from a physiological perspective. For example, during vocalization, the state of the vocal cords, diaphragm, lips, muscles, etc. may change, and the changes corresponding to different vocalization positions are different, that is, the physiological signals corresponding to different vocalization positions are different.
[0060] It is evident that speech signals and physiological signals each have their own emphasis when reflecting the vocalization process. Combining them can complement each other and improve the accuracy of vocalization location determination.
[0061] Before determining the second vocalization location, a vocalization location determination model can be constructed based on a large amount of training data (including sample speech signals, sample physiological signals, and sample vocalization location labels). This model can be a machine learning model (such as support vector machine, neural network, etc.) or a deep learning model (such as convolutional neural network, recurrent neural network, etc.).
[0062] Step 140: Determine the vocal position of the speaker based on the first vocal position and the second vocal position.
[0063] Specifically, the first sound source location is determined after the sound signal is located. That is, the first sound source location mainly relies on the acoustic characteristics of the speech signal and the corresponding acoustic standards for location. However, acoustic information may be affected by factors such as environmental noise and propagation medium, which may lead to errors in the location results. In addition, each person's physiological structural characteristics (such as vocal tract length, resonant cavity shape, etc.) are different, which may limit the applicability of acoustic standards among individuals.
[0064] Furthermore, the second vocal position is determined using a vocal position determination model, applying both speech and physiological signals. Since the physiological signals contain the physiological structural characteristics of the speaker, combining these signals with the speech signals allows for the accurate determination of the second vocal position, taking into account acoustic characteristics and individual differences among the speakers.
[0065] Furthermore, the vocal placement determination model can adapt to individual differences through training and learning. In other words, the model possesses learning capabilities, recognizing and learning new acoustic characteristics and individual variations, thus continuously adapting to new changes and accurately determining the second vocal placement. Compared to the first vocal placement, which is limited by acoustic standards, the vocal placement determination model can dynamically predict the vocal placement of different individuals.
[0066] Furthermore, the vocal position determination model can be pre-trained using sample speech signals, sample physiological signals, and sample vocal position labels. However, since the information obtained from sample speech signals, sample physiological signals, and sample vocal position labels is generalized feature information, it is impossible to summarize the feature information corresponding to a few special vocal positions. Therefore, the vocal position determination model cannot accurately detect a few special vocal positions. On the other hand, acoustic standards can include standards corresponding to a few special vocal positions, and thus, a few special vocal positions can be accurately detected based on acoustic standards.
[0067] The method for determining the vocal position provided in this invention can accurately detect a few special vocal positions by locating the sound source of the speech signal. At the same time, the vocal position determination model can accurately detect the vocal position of new vocalists based on speech. Thus, by combining the first vocal position and the second vocal position, the vocal position of the vocalist can be obtained more accurately.
[0068] Based on the above embodiments, Figure 2 This is a flowchart illustrating an implementation method for step 130 of the sound emission location determination method provided by the present invention, as shown below. Figure 2 As shown, step 130 uses speech and physiological signals to determine the second vocalization location, including:
[0069] Step 131: Extract speech features from the speech signal.
[0070] Specifically, speech features are parameters extracted from speech signals that reflect the acoustic characteristics of the speaker. Speech features can include fundamental frequency, formants, spectral characteristics, cepstral characteristics, etc. Optionally, the speech signal can be converted to the Bark domain, and speech features can be extracted from the converted speech signal. When extracting speech features from a speech signal, features such as Linear Predictive Coding (LPC), Mel-Frequency Cepstral Coefficients (MFCC), and Perceptual Linear Predictive (PLP) features can be used.
[0071] Step 132: Extract physiological features from physiological signals.
[0072] Specifically, physiological characteristics refer to parameters extracted from physiological signals that reflect the physiological state of the vocalist. Physiological characteristics can include physiological parameters such as respiratory rate, respiratory depth, and muscle tone. Respiratory rate can be monitored using a respiratory sensor; muscle tone can be monitored using a muscle activity sensor.
[0073] Step 133: Determine the second vocalization position based on speech and physiological features.
[0074] Specifically, speech features and physiological features reflect information about the vocalization process from acoustic and physiological perspectives, respectively. When determining the second vocalization location, a multi-feature fusion method can be employed, which integrates speech and physiological features. Utilizing their correlation and complementarity improves the accuracy and reliability of the analysis, allowing for a comprehensive consideration of the influence of various factors during vocalization, thus more accurately determining the second vocalization location.
[0075] Based on any of the above embodiments Figure 3 This is a flowchart illustrating an implementation method for step 133 of the sound emission location determination method provided by the present invention, as shown below. Figure 3 As shown, step 133 determines the second vocalization location based on speech features and physiological features, including:
[0076] Step 1331: Segmenting speech features and physiological features;
[0077] Step 1332: Determine the second sound source position based on the spliced features.
[0078] Specifically, speech features and physiological features reflect information about the vocalization process from the acoustic and physiological levels, respectively. After splicing speech features and physiological features, the spliced features include acoustic and physiological information from the vocalization process.
[0079] Considering that different features may have different dimensions and value ranges, speech features and physiological features can be standardized before splicing to ensure that they have the same weight and influence in the spliced feature vector.
[0080] In addition, when splicing speech features and physiological features, they can be spliced in a certain order (such as splicing physiological features after speech features) to obtain spliced features, and the second phonation position can be determined based on the acoustic and physiological information contained in the spliced features.
[0081] Based on any of the above embodiments, the speech signal is acquired by a microphone array, which is an array of multiple microphones used to acquire speech signals from different directions.
[0082] Figure 4 This is a flowchart illustrating an implementation method for step 120 of the sound emission location determination method provided by the present invention, as shown below. Figure 4 As shown, step 120 involves locating the sound source of the speech signal and determining the first sound source location, including:
[0083] Step 121: Based on the signal differences between the microphones in the microphone array, enhance the voice signals collected by each microphone to obtain the target signals of each microphone;
[0084] Step 122: Based on the target signals from each microphone, perform sound source localization to determine the first sound source location.
[0085] Specifically, a microphone array consists of multiple microphones arranged in a specific geometric structure, capable of simultaneously acquiring speech signals from different directions. Due to the varying distances between the microphones and the sound source, as well as the influence of environmental noise and reflections, the signals acquired by each microphone will differ, i.e., signal differences. These signal differences can include time differences (the time it takes for sound waves to reach different microphones), phase differences (the phase shift of sound waves at different microphones), and intensity differences (the different signal intensities produced by sound waves at different microphones).
[0086] Based on the aforementioned signal differences, signal processing techniques can be used to enhance the speech signals acquired by each microphone. Enhancement methods typically include noise reduction, echo cancellation, and signal filtering, aiming to improve the quality and recognizability of the speech signal, resulting in a target signal with a higher signal-to-noise ratio and clarity.
[0087] Furthermore, during sound generation, different sound-generating locations produce sound waves with different frequencies and characteristics. These sound waves are affected by environmental factors during propagation, such as reflection and diffraction. However, because microphone arrays can capture these subtle differences in sound waves, the sound-generating location can be determined by analyzing the differences between the target signals.
[0088] For example, throat sounds typically have higher frequencies and shorter wavelengths, while chest sounds have lower frequencies and longer wavelengths. Different vocal positions produce different characteristics such as time differences, phase differences, and intensity differences during sound wave propagation. By measuring these characteristics and combining them with the geometry of the microphone array and signal processing techniques, the initial vocal position can be accurately determined.
[0089] Based on any of the above embodiments, the vocal position determination model is selected from the model library based on the attribute characteristics of the vocal person. The model library stores vocal position determination models corresponding to different attribute characteristics.
[0090] Specifically, attribute features refer to factors that can describe or distinguish the individual characteristics of different voice actors. Attribute features may include age, gender, height, weight, etc. Considering that voice actors with different attribute features may produce different physiological and speech signals even when using the same vocal position, this embodiment of the invention matches the corresponding vocal position determination model based on the voice actor's attribute features. This enables personalized matching for voice actors with different attribute features, avoiding the problem of errors in positioning results caused by ignoring individual differences.
[0091] Optionally, a pre-trained model can be obtained based on a large number of sample speech signals, sample physiological signals, and sample vocalization position labels. Then, based on the pre-trained model, the speech signals, physiological signals, and vocalization position labels of vocalists with different attribute features can be used to fine-tune the pre-trained model to obtain a vocalization position determination model with corresponding attribute features.
[0092] Therefore, in the process of selecting a model for determining the vocal position, the embodiments of the present invention select a matching model from the model library based on the attribute characteristics of the vocal person. These models are trained and optimized, and can more accurately determine the vocal position for different vocal persons.
[0093] Based on any of the above embodiments, physiological signals include body activity signals and / or temperature distribution signals.
[0094] Specifically, body movement signals refer to the motion signals of specific parts of the speaker's body (such as the ears, chin, lips, and mouth) during vocalization, such as the expansion or contraction of the mouth. Body movement signals can be acquired using visual sensors. By continuously collecting changes in specific parts of the speaker's body through visual sensors, body movement signals can be obtained based on these changes. For example, during vocalization, the movement of the speaker's muscles can be detected.
[0095] Temperature distribution signal refers to the temperature information of different parts of the vocalizer during the vocalization process. This signal can be acquired using an infrared sensor. For example, when the vocalization location is the throat cavity, the contraction of the laryngeal muscles during vocalization leads to an increase in temperature, while capillary blood flow accelerates. Specifically, the temperature distribution signal can be a temperature distribution map of the vocalizer during the vocalization process, which characterizes the distribution of body surface temperature.
[0096] Based on any of the above embodiments, the location of the speaker's voice is determined, and then the process further includes:
[0097] Pitch detection is performed on the speech signal to obtain the pitch detection results;
[0098] Based on the pitch detection results and the speaker's vocal position, the speaker's vocal accuracy is determined.
[0099] Specifically, pitch accuracy testing results refer to the degree of match between the speaker's voice signal and the standard pitch, obtained after frequency analysis. In other words, pitch accuracy testing results measure the degree of match between the frequency of the speaker's voice and the fixed frequency of each note in the music. If the notes sung by the speaker are exactly the same as the standard pitch, then the pitch accuracy testing result is perfect. In other words, pitch accuracy testing results are used to measure the accuracy of pitch.
[0100] Vocal accuracy encompasses not only pitch accuracy but also the correctness of vocal placement. It is a crucial indicator for evaluating a vocalist's overall performance, requiring them to achieve both accurate pitch and correct vocal placement when singing or playing an instrument.
[0101] Therefore, the accuracy of a speaker's pitch can be assessed based on the pitch detection results, the correctness of the speaker's vocal position can be assessed based on the speaker's vocal position, and the accuracy of the speaker's vocalization can be accurately determined by combining the pitch detection results and the speaker's position.
[0102] For example, after obtaining the vocal accuracy, a vocal accuracy report can be generated, and based on the analysis results of the vocal accuracy report, specific aspects that need improvement can be determined. For instance, if the pitch detection results show a large pitch deviation, then improving pitch accuracy will be the improvement goal. Similarly, if the vocal placement is incorrect or the sound quality is poor, then correcting the vocal placement will be the improvement goal.
[0103] Based on any of the above embodiments Figure 5 This is a flowchart illustrating another method for determining the sound emission location provided by the present invention, as shown below. Figure 5 As shown, the method includes:
[0104] S1. Data Acquisition:
[0105] The original signal S is obtained by acquiring the sound of the speaker and the surrounding environment through a microphone array.
[0106] Data from specific parts of the speaker's body is collected using visual sensors, and distance measurements are taken based on this data (such as the ears, chin, and lips). A model of the speaker's vocal cavity is then constructed, with the speaker in a non-vocalizing state, denoted as the static state M. _static In subsequent operating modes, the visual sensor continuously collects changes in specific parts of the speaker's body during vocalization, such as muscle expansion or contraction, and records the collected changes as M. _dynamic The difference between the two is denoted as ΔM (i.e., the body activity signal in physiological signals), which represents the muscle changes of the person speaking.
[0107] In addition, an infrared sensor can be used to obtain the temperature distribution map T (i.e., the temperature distribution signal in the physiological signal) of the person making the voice. This temperature distribution map is used to characterize the temperature information of various parts of the person making the voice during the voice-making process.
[0108] S2. Determine the vocalization location:
[0109] First, the original signal S is subjected to noise reduction processing, mainly including denoising and noise reduction. The resulting speech signal is S. _denoise Assuming there are 6 microphones, the corresponding 6 voice signals are denoted as [S]. _1 ;S _2 ;S _3 ;S _4 ;S _5 ;S _6 The above six speech signals are subjected to beamforming to separate external noise from the speech signals, resulting in the denoised speech signal B1.
[0110] After obtaining the denoised speech signal B1, Fourier transform is used to convert B1 to the frequency domain, resulting in the frequency domain signals [F1,…,F6] corresponding to the six microphones. Based on the characteristics of the ring array, beamforming is performed again, employing RTF-GSC and GEVD strategies to divide the frequency domain signal into six zones, corresponding to six vocal positions (i.e., head cavity, nasal cavity, pharynx, oral cavity, laryngeal cavity, and chest cavity). The partitioning result of the frequency domain signal can be considered as the first vocal position, R1, which can be represented by the vocal zone indicator vector V[a,b,c,d,e,f], where a+b+c+d+e+f=1, and each value represents the probability of the corresponding vocal zone appearing.
[0111] Furthermore, the frequency domain signal [F1,…,F6], the body activity signal ΔM, and the temperature distribution signal T are input into the vocalization location determination model. The model converts the frequency domain signal to the Bark domain, performs convolution on the visual signal ΔM and flattens it into one dimension, and also performs convolution on the temperature signal T and flattens it into one dimension. Finally, the feature vectors of the three modalities are concatenated to obtain the final concatenated feature [B1,B2,B3,B4,B5,B6,M1,T1]. Based on this concatenated feature, the second vocalization location R2 is obtained, which can be the vocal register indicator vector V. _final [a,b,c,d,e,f], for example, [0,0,0,0,1,0] represents the oral cavity resonance area.
[0112] After obtaining the first vocal position R1 and the second vocal position R2, the final vocal position R = R1 × W1 + R2 × W2 can be obtained by using weighting coefficients. W1 and W2 can be iterative parameters or parameters trained by the model.
[0113] S3, Vocalization Accuracy:
[0114] The vocal position of the speaker can be projected onto the vocal cavity model of the speaker, and the pitch of the speaker's speech signal B1 can be analyzed. Based on the pitch detection results and the vocal position, the accuracy of the speaker's vocalization can be determined.
[0115] The sound emission position determination device provided by the present invention is described below. The sound emission position determination device described below can be referred to in correspondence with the sound emission position determination method described above.
[0116] Based on any of the above embodiments Figure 6 This is a schematic diagram of the sound emission position determination device provided by the present invention, as shown below. Figure 6 As shown, the device includes:
[0117] Acquisition unit 610 is used to acquire the speech signal and physiological signal of the speaker;
[0118] The positioning unit 620 is used to locate the sound source of the speech signal and determine the first sound source position.
[0119] The prediction unit 630 is used to determine the second vocal position based on the vocal position determination model, by applying speech signals and physiological signals.
[0120] The determining unit 640 is used to determine the vocal position of the vocalist based on the first vocal position and the second vocal position.
[0121] Based on any of the above embodiments, determining the second vocalization location using speech signals and physiological signals includes:
[0122] Speech features are extracted from the speech signal;
[0123] Physiological features are extracted from physiological signals;
[0124] The second vocalization position is determined based on speech and physiological features.
[0125] Based on any of the above embodiments, determining the second vocalization position based on speech features and physiological features includes:
[0126] Speech features and physiological characteristics of spliced speech;
[0127] The second sound source location is determined based on the spliced features.
[0128] Based on any of the above embodiments, the voice signal is acquired by a microphone array;
[0129] Localize the sound source of the speech signal and determine the first sound source location, including:
[0130] Based on the signal differences between the microphones in the microphone array, the voice signals collected by each microphone are enhanced to obtain the target signal of each microphone;
[0131] Based on the target signals from each microphone, the sound source is located to determine the first sound source position.
[0132] Based on any of the above embodiments, the vocal position determination model is selected from the model library based on the attribute characteristics of the vocal person. The model library stores vocal position determination models corresponding to different attribute characteristics.
[0133] Based on any of the above embodiments, physiological signals include body activity signals and / or temperature distribution signals.
[0134] Based on any of the above embodiments, the location of the speaker's voice is determined, and then the process further includes:
[0135] Pitch detection is performed on the speech signal to obtain the pitch detection results;
[0136] Based on the pitch detection results and the speaker's vocal position, the speaker's vocal accuracy is determined.
[0137] Based on any of the above embodiments, the present invention also provides a vocal practice system, comprising:
[0138] Microphone array, camera, infrared sensor, and sound source location determination device as described in any of the above embodiments;
[0139] The microphone array is used to collect voice signals, the camera is used to collect body activity signals from physiological signals, and the infrared sensor is used to collect temperature distribution signals from physiological signals.
[0140] Optionally, the vocal practice system can be head-mounted, meaning it can be worn on the head of the vocalist. This system can be equipped with a microphone array to collect speech signals, and small cameras positioned on both sides and the back of the headband to collect data on the vocalist's head shape and neck. Based on this data, a vocal cavity model can be constructed. Furthermore, the small cameras continuously collect body movement signals during vocalization. Additionally, the vocal practice system can also include an infrared sensor to collect temperature distribution signals from physiological data.
[0141] Based on the voice signals, body activity signals, and temperature distribution signals collected by the system, the vocal position of the speaker can be accurately determined, which greatly improves the practice experience of vocal practitioners and lowers the threshold for scientific vocalization.
[0142] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a voice position determination method, which includes: acquiring the voice signal and physiological signal of the speaker; performing sound source localization on the voice signal to determine a first voice position; determining a second voice position based on a voice position determination model, applying the voice signal and the physiological signal; and determining the voice position of the speaker based on the first voice position and the second voice position.
[0143] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0144] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the vocal position determination method provided by the above methods. The method includes: acquiring the voice signal and physiological signal of the speaker; performing sound source localization on the voice signal to determine a first vocal position; determining a second vocal position based on the vocal position determination model and applying the voice signal and the physiological signal; and determining the vocal position of the speaker based on the first vocal position and the second vocal position.
[0145] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for determining the vocal position provided by the above methods. The method includes: acquiring a speech signal and a physiological signal of a vocal person; performing sound source localization on the speech signal to determine a first vocal position; determining a second vocal position based on a vocal position determination model, applying the speech signal and the physiological signal; and determining the vocal position of the vocal person based on the first vocal position and the second vocal position.
[0146] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for determining the sound emission location, characterized in that, include: Acquire the speech and physiological signals of the speaker; The voice signal was acquired by a microphone array; Based on the signal differences between the microphones in the microphone array, the speech signals collected by each microphone are enhanced to obtain the target signal of each microphone; based on the target signal of each microphone, the sound source is located, the type of resonance cavity is determined, and the first sound emission position is located. Based on the vocalization location determination model, the second vocalization location is determined by applying the speech signal and the physiological signal. Based on the first and second vocal positions, the vocal position of the speaker is determined. The vocal position refers to the location where airflow is obstructed and sound is produced during phonation. It is further subdivided into resonance areas including the head cavity, nasal cavity, pharyngeal cavity, oral cavity, laryngeal cavity, and chest cavity.
2. The method for determining the sound emission location according to claim 1, characterized in that, The method of determining the second vocalization position by applying the speech signal and the physiological signal includes: Speech features are extracted from the speech signal; Physiological features are extracted from the physiological signals; The second vocalization location is determined based on the spoken features and the physiological features.
3. The method for determining the sound emission location according to claim 2, characterized in that, Determining the second vocalization location based on the speech features and the physiological features includes: The speech features and the physiological features are spliced together; The second sound source location is determined based on the spliced features.
4. The method for determining the sound emission location according to any one of claims 1 to 3, characterized in that, The vocal position determination model is selected from a model library based on the attribute characteristics of the vocalist. The model library stores vocal position determination models corresponding to different attribute characteristics.
5. The method for determining the sound emission location according to any one of claims 1 to 3, characterized in that, The physiological signals include body activity signals and / or temperature distribution signals.
6. The method for determining the sound emission location according to any one of claims 1 to 3, characterized in that, After determining the vocal position of the person making the sound, the method further includes: The speech signal is subjected to pitch detection to obtain pitch detection results; Based on the pitch detection results and the vocal position of the speaker, the vocal accuracy of the speaker is determined.
7. A sound emission location determination device, characterized in that, include: The acquisition unit is used to acquire the speech signal and physiological signal of the speaker; The voice signal was acquired by a microphone array; The positioning unit is used to enhance the voice signals collected by each microphone based on the signal differences between each microphone in the microphone array to obtain the target signal of each microphone; and to perform sound source localization based on the target signal of each microphone, determine the type of resonance cavity, and locate the first sound emission position. The prediction unit is used to determine the second vocalization position based on the vocalization position determination model, by applying the speech signal and the physiological signal. A determining unit is configured to determine the vocal position of the vocal person based on the first vocal position and the second vocal position; The vocal position refers to the location where airflow is obstructed and sound is produced during phonation. It is further subdivided into resonance areas including the head cavity, nasal cavity, pharyngeal cavity, oral cavity, laryngeal cavity, and chest cavity.
8. A vocal practice system, characterized in that, include: Microphone array, camera, infrared sensor, and sound emission location determination device as described in claim 7; The microphone array is used to collect the voice signal, the camera is used to collect the body activity signal in the physiological signal, and the infrared sensor is used to collect the temperature distribution signal in the physiological signal.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the sound location determination method as described in any one of claims 1 to 6.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the sound location determination method as described in any one of claims 1 to 6.
11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the sound location determination method as described in any one of claims 1 to 6.