Terminal data processing method and device for audio
By analyzing audio and accompaniment data, predicting the singing status and the state of the part to be sung, and formulating processing strategies, the problem that audio terminal devices in the prior art cannot improve the singing quality, achieving a better singing experience and audio processing effect.
Patent Information
- Application Number
- CN202411188796.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing audio terminal devices cannot effectively help users improve the singing quality and have poor user experience, especially in scenes such as family gatherings, where the singer's audio cannot be intelligently adjusted, resulting in serious out-of-tune.
By obtaining the user's real-time audio data and music accompaniment data, analyzing the singing progress and status, predicting the status of the part to be sung, formulating audio data processing strategies, and performing audio data processing to improve singing effect.
It improves the singing quality, reduces out-of-tune, provides a better user experience, and can intelligently customize audio data processing strategies to ensure that the singing is more perfect.
Smart Images

Figure CN119091926A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and in particular to a terminal data processing method and device for audio. Background Art
[0002] With the development of terminal technology, terminals can provide users with more and more functions. Audio terminals can realize real-time singing functions through recording functions and audio data processing functions. The current technical means select the target song to be played, the audio terminal starts playing the song audio, and the user sings along with the song audio.
[0003] Current technical means only provide singing functions, but cannot help users complete singing more perfectly, resulting in a poor user experience. Summary of the invention
[0004] The present application provides a terminal data processing method and device for audio to solve the above-mentioned problems.
[0005] In a first aspect, the present application provides a terminal data processing method for audio, the method comprising: Acquire the user's real-time audio data and music accompaniment data, analyze the audio data and the accompaniment data, and determine the user's singing progress; Determining the user's singing status according to the user's singing progress, the audio data and the accompaniment data; Analyze the singing status and the singing progress, and predict the status of the part to be sung by the user; Analyze the state of the part to be sung, the audio data and the accompaniment data, and determine an audio data processing strategy; The audio data processing strategy is executed to process the audio data and output the processed audio data.
[0006] Through this solution, the user's real-time audio data is compared with the music accompaniment data to determine the user's singing progress, providing basic information for the subsequent determination of the user's singing status and the status of the part to be sung. According to the user's singing progress, audio data and accompaniment data, a multi-dimensional analysis is performed to determine the user's singing status to ensure the accuracy of the analysis. The singing progress is analyzed to determine the part to be sung, and then the singing status is used to predict the status of the user's part to be sung, providing direct information for the audio modification strategy. The status of the part to be sung, audio data and accompaniment data are analyzed to determine the audio data processing strategy, and the audio data processing strategy is executed, so that the formulation of the modification strategy is faster and the efficiency and accuracy are improved.
[0007] Optionally, the analyzing the audio data and the accompaniment data to determine the user's singing progress includes: accurately identifying the lyrics and backing lyrics sung by the user based on the audio data; determining the melody data and accompaniment progress information based on the accompaniment data; dividing the song intervals based on the melody data; determining the current singing interval based on the accompaniment progress information and the song interval; and determining the user's singing progress based on the lyrics, the backing lyrics and the current singing interval.
[0008] Through this solution, the lyrics and backing lyrics sung by the user can be accurately identified based on audio data, providing a basis for subsequent singing progress analysis and song interval division. The lyrics data and accompaniment progress information are determined based on the accompaniment data, and the song interval is divided based on the lyrics data. The current singing interval is determined by combining the accompaniment progress information and the song interval. The correspondence between the lyrics data and the accompaniment progress information is determined through the accompaniment data, thereby achieving an accurate comparison between the lyrics and the accompaniment sung by the user, and efficiently judging the overall coordination and sense of rhythm of the user's singing. Finally, the user's singing progress is determined based on the lyrics, backing lyrics, and current singing interval. Multi-dimensional data analysis can comprehensively and accurately track the user's singing progress.
[0009] Optionally, determining the singing status of the user according to the user's singing progress, the audio data and the accompaniment data includes: comparing the user's singing progress with the accompaniment progress information to determine the user's sense of rhythm; analyzing the audio data to determine the sampling value and frequency; analyzing the accompaniment data to determine the standard amplitude and standard frequency; determining the amplitude according to the sampling value; analyzing the amplitude to determine the amplitude change information; analyzing the standard amplitude to determine the standard amplitude change information; determining the user's voice volume change information according to the amplitude change information; comparing the amplitude change information with the standard amplitude change information to determine the user's volume balance; analyzing the frequency to determine the frequency change information; analyzing the standard frequency to determine the standard frequency change information; comparing the frequency change information with the standard frequency change information to determine the out-of-tune frequency information; determining the out-of-tune frequency according to the out-of-tune frequency information; determining the user's out-of-tune rate according to the out-of-tune frequency, refer to the following formula: ; in, represents the user's out-of-tune rate, Indicates the frequency of the out-of-tune, represents the standard frequency; The user's singing status is determined based on the user's sense of rhythm, the user's sense of volume balance and the user's out-of-tune rate.
[0010] Through this solution, the user's singing progress and accompaniment progress information are compared to determine the user's sense of rhythm, the sampling value and frequency are obtained from the audio data, the amplitude is obtained from the sampling value and then the amplitude change information is obtained, the standard amplitude and standard frequency are obtained from the accompaniment data, the standard amplitude and amplitude are compared to reflect the user's sense of volume balance, the frequency change information and the standard frequency change information are compared, and the formula is used for calculation to quantify the user's out-of-tune rate. By comparing the user's singing progress with the accompaniment progress, analyzing the audio data and accompaniment data, the user's singing status is comprehensively evaluated from three dimensions: sense of rhythm, volume balance and out-of-tune rate, the amplitude and frequency change information is analyzed in detail and compared with the standard amplitude and frequency, and the volume balance problems and out-of-tune problems of the user during the singing process are accurately located.
[0011] Optionally, the comparing the frequency change information with the standard frequency change information to determine the out-of-tune frequency information includes: comparing the frequency change information with the standard frequency change information to determine frequency difference information; analyzing the frequency difference information to determine the frequency difference value and constructing a frequency difference curve; analyzing the frequency difference curve to divide the curve into a stable part and a fluctuating part and determine the frequency difference time; determining the fluctuation frequency and the fluctuation amplitude according to the fluctuating part of the curve; analyzing the frequency difference information and the frequency difference time to determine the average frequency difference and the frequency difference standard deviation; determining a comprehensive score according to the fluctuation frequency, the frequency difference time, the fluctuation amplitude, the average frequency difference and the frequency difference standard deviation; determining the out-of-tune frequency information according to the comprehensive score.
[0012] Through this scheme, the frequency change information and the standard frequency change information are compared, the frequency difference curve is constructed and the curve is divided into a stable part and a fluctuating part, which improves the recognition accuracy of the out-of-tune phenomenon. The frequency difference time is determined from the frequency difference curve, which can provide time data for the calculation. The fluctuation frequency and fluctuation amplitude are determined from the fluctuating part of the curve, and the frequency average difference and the frequency difference standard deviation are determined from the frequency difference information and the frequency difference time. The analysis of the out-of-tune characteristics is further refined. According to the fluctuation frequency, frequency difference time, fluctuation amplitude, frequency average difference and frequency difference standard deviation, a comprehensive score is determined to determine the out-of-tune frequency information. A plurality of out-of-tune related indicators are comprehensively considered to avoid the one-sidedness of a single indicator evaluation, making the evaluation of the out-of-tune frequency information more objective and comparable.
[0013] Optionally, determining the out-of-tune frequency according to the comprehensive score includes: According to the fluctuation frequency, the frequency difference time and the fluctuation amplitude, the vibrato existence index is determined, referring to the following formula: ; in, There is an index for the vibrato, is the frequency difference time, is the fluctuation frequency, is the preset minimum fluctuation frequency, is the preset maximum value of the fluctuation frequency, is the fluctuation range, The minimum amplitude of the preset fluctuation; According to the average frequency difference, the frequency difference time and the frequency difference information, the pitch shift existence index is determined, referring to the following formula: ; in, There exists an index for the pitch shift, is the frequency difference time, is the frequency difference, is the frequency mean difference, The preset allowable pitch difference fluctuation range; Based on the standard deviation of the frequency difference, the pitch stability score is determined, referring to the following formula: ; in, scoring the pitch stability, is the frequency difference time, is the standard deviation of the frequency difference, The maximum permissible value of the preset pitch difference; According to the vibrato existence index, the pitch shift existence index, and the pitch stability score, a comprehensive score is determined, referring to the following formula: ; in, For the comprehensive score, is the frequency difference value, There is an index for the vibrato, scoring the pitch stability, There is an index for said pitch shift; Based on the comprehensive score, the frequency of the out-of-tune is determined.
[0014] Through this scheme, according to the fluctuation frequency, frequency difference time and fluctuation amplitude, the vibrato existence index is determined by the formula to judge whether the frequency difference is caused by vibrato. According to the average frequency difference, frequency difference time and frequency difference information, the pitch shift existence index is determined by the formula to judge whether the frequency difference is caused by glissando. According to the standard deviation of the frequency difference, the pitch stability score is obtained. The three formulas determine three values, which provide a comprehensive, accurate and intuitive judgment for the evaluation of frequency differences, and also have an accurate judgment on singing skills. Finally, the three values and formulas are used to obtain a comprehensive score. The comprehensive score turns the abstract out-of-tune into intuitive data, and can accurately evaluate the degree of out-of-tune when it exists. Through a series of mathematical models and formulas, the sound signal is comprehensively analyzed, and a detailed evaluation of vibrato, pitch shift and pitch stability is provided.
[0015] Optionally, the analysis of the singing status and the singing progress to predict the status of the user's to-be-sung part includes: determining the user's sung part and the to-be-sung part according to the singing progress; determining the expressiveness of the to-be-sung part according to the user's sense of rhythm and the volume balance; determining the overlapping segment of the to-be-sung part and the sung part according to the sung part, the to-be-sung part and the accompaniment data; using the overlapping segment and the out-of-tune frequency as parameters, inputting a preset machine learning model, and outputting the out-of-tune frequency of the overlapping part in the to-be-sung part; analyzing the accompaniment data and the out-of-tune frequency to determine the out-of-tune probability of each range; determining the out-of-tune frequency of the non-overlapping part in the to-be-sung part according to the out-of-tune probability of each range; determining the out-of-tune frequency of the part to be sung according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part; determining the out-of-tune frequency of the part to be sung according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part; determining the overlapping part out-of-tune frequency, the non-overlapping part out-of-tune frequency and the expressiveness as the user's to-be-sung part status.
[0016] Through this solution, the sung part and the part to be sung are divided according to the singing progress, which provides real-time information for determining the out-of-tune and expressiveness. The expressiveness of the part to be sung is determined in combination with the user's sense of rhythm and volume balance, and the prediction of the part to be sung is realized. Then, based on the sung part, the part to be sung and the accompaniment data, the overlapping segments of the sung part are determined, and the machine learning model is used to use the overlapping segments and the out-of-tune frequency as parameters to obtain the out-of-tune frequency of the overlapping part in the part to be sung. For the non-overlapping part, it is determined according to the out-of-tune probability of each range. The whole solution has a good effect in controlling the singing progress, analyzing the expressiveness of the part to be sung, identifying the overlapping segments and the out-of-tune frequency, evaluating the out-of-tune frequency of the part to be sung, and comprehensively evaluating the status of the user's part to be sung.
[0017] Optionally, determining the expressiveness of the part to be sung according to the user's sense of rhythm and the sense of volume balance includes: analyzing the accompaniment data to determine the rhythm of the song and the main and chorus parts, and planning the number of rhythm words and rhythm words according to the song rhythm; calculating the average number of beats and drags of each rhythm word in the part sung by the user according to the user's sense of rhythm and the number of rhythm words; using the average number of beats and drags as the number of beats and drags of each line of the lyrics of the part to be sung; determining the average decibel of the verse and the average decibel of the chorus according to the sense of volume balance and the main and chorus parts; determining the average decibel of the verse and the average decibel of the chorus as the decibel information of the part to be sung; determining the volume balance of the part to be sung according to the decibel information; determining the expressiveness of the part to be sung according to the number of beats and drags of each line of the part to be sung and the sense of volume balance of the part to be sung.
[0018] Through this solution, the accompaniment data is analyzed to determine the rhythm of the song and the main and chorus parts, the number of rhythm words is divided from the rhythm of the song, and the average number of beats and delays of the rhythm words in each sentence in the sung part is calculated based on the user's sense of rhythm and the number of rhythm words. This provides a direct basis for evaluating the beats and delays of the part to be sung, and then the average decibels of the verse and the chorus are determined based on the sense of volume balance and the main and chorus parts, and the average decibels of the verse and the chorus are used as the average decibels of the verse and the chorus of the part to be sung, which fully considers the difference between the verse and the chorus in the song, making the decibel information prediction of the part to be sung more scientific and accurate.
[0019] Optionally, determining the singing status of the user according to the user's singing progress, the audio data and the accompaniment data also includes: monitoring the user's heart rate and breathing to determine the heart rate and breathing conditions; analyzing the heart rate, breathing and out-of-tune frequency to determine the heart rate and breathing number when out-of-tune; acquiring and analyzing gravity sensor information to determine the movement amplitude information of the user when singing; analyzing the movement amplitude information and the out-of-tune frequency to determine the movement amplitude when out-of-tune; calculating the average heart rate, average breathing number and average movement amplitude when out-of-tune according to the heart rate when out-of-tune, the number of breathings when out-of-tune and the movement amplitude when out-of-tune; modifying the comprehensive score according to the average heart rate, the average number of breathings and the average movement amplitude to determine a corrected comprehensive score; determining the corrected out-of-tune frequency according to the corrected comprehensive score, and determining the corrected user out-of-tune rate according to the corrected out-of-tune frequency; determining the user's singing status according to the user's sense of rhythm, the user's volume balance and the corrected user out-of-tune rate.
[0020] Through this solution, the out-of-tune frequency is obtained to determine the heart rate, breathing rate and movement amplitude information when out-of-tune, and the average heart rate, average breathing rate and average movement amplitude when out-of-tune are calculated to provide a reference for the subsequent correction of the comprehensive score. After obtaining the corrected comprehensive score, the corrected out-of-tune frequency is determined, and finally the corrected user out-of-tune rate is combined with the user's sense of rhythm and the user's sense of volume balance to determine the user's singing status. By monitoring the user's heart rate, breathing, and movement amplitude, a more comprehensive understanding of the user's physiological and behavioral status during the singing process can be obtained. This multi-dimensional data analysis helps to more accurately identify out-of-tune situations.
[0021] Optionally, modifying the comprehensive score according to the average heart rate, the average number of respirations and the average movement amplitude includes: determining the heart rate, respiration rate and movement amplitude when out of tune according to the heart rate condition, the respiration condition, the movement amplitude information and the out-of-tune frequency; determining the heart rate, the respiration rate and the movement amplitude as physiological indicators when out of tune; determining the average heart rate, the average number of respirations and the average movement amplitude as the average physiological indicators when out of tune; comparing the physiological indicators when out of tune with the average physiological indicators when out of tune to determine a comparison result; according to the comparison result, determining the physiological indicator when out of tune that is greater than the average physiological indicator when out of tune as the physiological indicator of frequency that needs to be corrected; modifying the out-of-tune frequency according to the physiological indicator of frequency that needs to be corrected and the out-of-tune frequency; modifying the comprehensive score according to the modified out-of-tune frequency.
[0022] Through this solution, physiological indicators when out of tune and average physiological indicators when out of tune are obtained, and the physiological indicators when out of tune and the average physiological indicators when out of tune are compared. The physiological indicators of the frequency that needs to be corrected are accurately located from the comparison results, providing a targeted basis for subsequent corrections. In addition, using physiological indicators as evaluation criteria is more objective and scientific than the traditional evaluation method that relies only on hearing or subjective feelings. This helps to reduce the interference of human factors, improve the accuracy and fairness of the evaluation, and modify the out-of-tune frequency according to the physiological indicators of the frequency that needs to be corrected and the out-of-tune frequency, and then modify the comprehensive score, providing a more scientific method for the evaluation of the comprehensive score.
[0023] In a second aspect, the present application provides a terminal data processing device for audio, the system comprising: The progress determination module is used to obtain the user's real-time audio data and music accompaniment data, analyze the audio data and the accompaniment data, and determine the user's singing progress; A status determination module, used to determine the user's singing status according to the user's singing progress, the audio data and the accompaniment data; A state prediction module, used to analyze the singing state and the singing progress, and predict the state of the part to be sung by the user; A strategy formulation module, for analyzing the state of the part to be sung, the audio data and the accompaniment data, and determining an audio data processing strategy; The strategy execution module is used to execute the audio data processing strategy, perform audio data processing, and output the processed audio data. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0025] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application; Figure 2 A flowchart of a terminal data processing method for audio provided in one embodiment of the present application; Figure 3 A schematic diagram of the structure of a terminal data processing device for audio provided in one embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.
[0028] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.
[0029] During many family gatherings, existing karaoke equipment cannot intelligently adjust the singer's singing audio according to the singer's singing level, resulting in serious out-of-tune singing, which makes the experience of both the singer and the audience poor. Some equipment also has audio correction function, but it cannot recognize many technical singing skills and also corrects the singer's technical singing, resulting in the singer's failure to show off his skills.
[0030] Based on this, the present application provides a terminal data processing method and device for audio, which first obtains the user's real-time audio data and music accompaniment data, from which the user's singing progress is determined, providing a basic basis for the subsequent analysis of the user's singing status, and using the singing progress, audio data, and accompaniment data to determine the user's singing status. The singing status provides important information for predicting the status of the part to be sung, and the status of the user's to-be-sung part is predicted based on the singing status and singing progress, which provides direct evidence for the audio data processing strategy, and then the status of the to-be-sung part is combined with the audio data and accompaniment data to formulate an audio data processing strategy, and perform audio data processing to enable the singer to sing better. The entire solution starts with the singer's audio data and accompaniment data, and obtains the user's singing status through multiple analyses, and the predicted status of the to-be-sung part can intelligently customize the audio data processing strategy to make the singing more perfect.
[0031] Figure 1 A schematic diagram of an application scenario provided for the present application, in which the method provided by the present application is applied in a home singing scenario. Specifically, the method provided by the present application is applied to any server, and the server interacts with an audio device. The user's real-time audio data and music accompaniment data are obtained from the audio device to determine the user's singing progress, and the user's singing status is determined based on the singing progress, audio data and accompaniment data, thereby predicting the status of the user's to-be-sung part, and then determining the audio data processing strategy based on the audio data and accompaniment data, performing audio data processing, and predicting the status of the to-be-sung part through the user's singing status, so as to formulate an audio data processing strategy based on the status of the to-be-sung part, and replace the user's original sound with the processed audio to provide the user with a better singing effect. The specific implementation method can refer to the following embodiments.
[0032] Figure 2 This is a flow chart of a method for processing terminal data for audio provided in an embodiment of the present application. The method of this embodiment can be applied to the server in the above scenario. Figure 2 As shown, the method includes: S201. Obtain the user's real-time audio data and music accompaniment data, analyze the audio data and the accompaniment data, and determine the user's singing progress.
[0033] The audio data may be data of various parameters of the audio of the user's singing voice, and the audio data may include frequency, amplitude, etc. The accompaniment data may be the accompaniment audio of the original song sung by the user, and may include frequency, amplitude, etc. The user's singing progress may be the position of the part sung by the user in the entire song.
[0034] Specifically, in a family gathering scenario, many people's singing performance is not very good. In order to make the user and listener experience better, it is necessary to adjust the audio data of the user's singing, and to know the accompaniment data of the song as a reference standard, and use the user's audio data to compare the accompaniment data, find the difference, and confirm the time point of the difference. In order to get the time point of the difference, it is necessary to determine the user's singing progress in real time, compare the user's audio data and accompaniment data at the moment of the user's singing progress, get the difference, and then make adjustments based on the difference. Therefore, first obtain the user's real-time audio data and music accompaniment data, use computer speech recognition technology to analyze the audio data and accompaniment data, and determine at which point in the entire song the user's singing part is, that is, determine the user's singing progress.
[0035] S202. Determine the user's singing status according to the user's singing progress, audio data and accompaniment data.
[0036] The singing status may be status information related to the content that the user has sung, and the singing status may include the user's sense of rhythm, volume balance, and out-of-tune rate.
[0037] Specifically, in order to formulate a modification strategy for user audio, we must first know whether there are flaws in the content of the user's singing, and these flaws are reflected in the differences between the user's audio data and the accompaniment data. In order to determine these flaws, it is necessary to determine the initial singing state, and determine the user's singing state based on the user's singing progress, audio data and accompaniment data, combined with mathematical analysis methods.
[0038] S203, analyzing the singing status and singing progress, and predicting the status of the part to be sung by the user.
[0039] The status of the part to be sung may be status information related to the content that the user has not sung yet. The status of the part to be sung may include the frequency of out-of-tune of the overlapping part, the frequency of out-of-tune of the non-overlapping part, and the expressiveness, etc.
[0040] Specifically, the audio processing strategy directly acts on the part to be sung by the user, and determines where adjustments are needed based on the predicted status of the part to be sung. In order to predict the status of the part to be sung, it is necessary to analyze the singing status and singing progress, and use machine learning and mathematical analysis methods to determine the status of the part to be sung.
[0041] S204, analyzing the state of the part to be sung, the audio data and the accompaniment data, and determining the audio data processing strategy.
[0042] Specifically, determining the audio data processing strategy requires knowing the status of the user's to-be-sung part. By comparing the status of the to-be-sung part with the accompaniment data, the difference between the status of the to-be-sung part and the accompaniment data is used as the data to be processed, and the audio processing strategy is determined based on the data to be processed.
[0043] S205: Execute the audio data processing strategy, process the audio data, and output the processed audio data.
[0044] After the audio data processing strategy is determined, the data to be processed that needs to be modified is modified according to the audio data processing strategy, and the modified audio is played.
[0045] Through this solution, the user's real-time audio data is compared with the music accompaniment data to determine the user's singing progress, providing basic information for the subsequent determination of the user's singing status and the status of the part to be sung. According to the user's singing progress, audio data and accompaniment data, a multi-dimensional analysis is performed to determine the user's singing status to ensure the accuracy of the analysis. The singing progress is analyzed to determine the part to be sung, and then the singing status is used to predict the status of the user's part to be sung, providing direct information for the audio modification strategy, analyzing the status of the part to be sung, audio data and accompaniment data, determining the audio data processing strategy, and executing the audio data processing strategy, making the formulation of the modification strategy faster and improving efficiency and accuracy.
[0046] In some embodiments, based on the audio data, the lyrics and backing lyrics sung by the user are accurately identified; based on the accompaniment data, the melody data and accompaniment progress information are determined; based on the melody data, the song intervals are divided; based on the accompaniment progress information and the song intervals, the current singing interval is determined; based on the lyrics, backing lyrics, and the current singing interval, the user's singing progress is determined.
[0047] The backing words can be some modal particles and phonophonetic words inserted in the content of the song in addition to the lyrics. The backing words can include "ah", "hey", "bar", etc. The lyrics data can be the lyrics and backing words in the song sung by the user. The accompaniment progress information can be the position of the current accompaniment audio in the entire accompaniment. The song interval can be the interval divided according to each repeated content of the lyrics.
[0048] Specifically, first, natural speech recognition technology is used to identify the user's audio data to determine the content of the human voice, that is, the lyrics and backing words sung by the user, and then the corresponding song is obtained from the accompaniment data, and the lyrics data and the time data of the accompaniment are extracted. The current time data of the accompaniment is used as the accompaniment progress information, and natural language processing technology is used to divide the intervals where the lyrics are repeated. Each repeated lyrics can be divided into a song interval, and then according to the accompaniment progress information and the song interval, it is determined which interval is currently being sung, and the current interval is determined as the current singing interval. Then, the user's singing progress is determined based on the lyrics, backing words and the current singing interval.
[0049] Through this solution, the lyrics and backing lyrics sung by the user can be accurately identified based on audio data, providing a basis for subsequent singing progress analysis and song interval division. The lyrics data and accompaniment progress information are determined based on the accompaniment data, and the song interval is divided based on the lyrics data. The current singing interval is determined by combining the accompaniment progress information and the song interval. The correspondence between the lyrics data and the accompaniment progress information is determined through the accompaniment data, thereby achieving an accurate comparison between the lyrics and the accompaniment sung by the user, and efficiently judging the overall coordination and sense of rhythm of the user's singing. Finally, the user's singing progress is determined based on the lyrics, backing lyrics, and current singing interval. Multi-dimensional data analysis can comprehensively and accurately track the user's singing progress.
[0050] In some embodiments, the user's singing progress is compared with the accompaniment progress information to determine the user's sense of rhythm; the audio data is analyzed to determine the sampling value and frequency; the accompaniment data is analyzed to determine the standard amplitude and standard frequency; the amplitude is determined according to the sampling value; the amplitude is analyzed to determine the amplitude change information; the standard amplitude is analyzed to determine the standard amplitude change information; according to the amplitude change information, the user's voice volume change information is determined; the amplitude change information is compared with the standard amplitude change information to determine the user's volume balance; the frequency is analyzed to determine the frequency change information; the standard frequency is analyzed to determine the standard frequency change information; the frequency change information is compared with the standard frequency change information to determine the out-of-tune frequency information; according to the out-of-tune frequency information, the out-of-tune frequency is determined; according to the out-of-tune frequency information, the user's out-of-tune rate is determined; according to the user's sense of rhythm, the user's volume balance and the user's out-of-tune rate, the user's singing status is determined.
[0051] According to the out-of-tune frequency information, determine the user's out-of-tune rate, refer to formula (1): (1); in, Indicates the user's out-of-tune rate. Indicates the frequency when out of tune, Indicates standard frequency.
[0052] The user's sense of rhythm may be the user's grasp of the beat of the music being sung. For example, if a certain sentence of a song starts to be sung at the 30th second, and the user starts to sing one second earlier, it means that the user's sense of rhythm is somewhat lacking. The sampling value may be the number of times the analog digital signal is sampled per unit time when collecting the user's voice audio. The frequency may be the vibration frequency of the user's audio. The standard amplitude may be the amplitude of the original sound of the sung song. The standard frequency may be the frequency of the original sound of the sung song. The amplitude may be the intensity of the contraction and expansion experienced by the medium during the propagation of the user's voice, and the amplitude is used to quantify the size of the sound. The amplitude change information may be information about the change in the amplitude of the user's voice as the song is sung. The user's voice size change information may be information about the change in the size of the user's voice as the song is sung. The standard amplitude change information may be information about the change in the amplitude of the original song corresponding to the sung song as the song progresses. The sense of volume balance may be the user's grasp of the volume of the original song. The frequency change information may be information about the change in the frequency of the user's voice as the song is sung. The standard frequency change information may be information about the change in the frequency of the original song corresponding to the sung song as the song progresses. The out-of-tune frequency information may be frequency information when the user is out of tune. The user out-of-tune rate may be the ratio of the difference between the frequency of the user's out-of-tune singing and the frequency of the original singing of the song to the frequency of the original singing of the song.
[0053] Specifically, in order to evaluate the user's singing status, it is necessary to start with the user's ability to catch the beat, the ability to balance the volume during singing, and the out-of-tune situation. First, compare the user's singing progress and accompaniment progress information, record the difference value, and thus determine the user's sense of rhythm. Then, use natural language processing technology and image processing technology to obtain the sampling value and frequency from the audio data, obtain the standard amplitude and standard frequency from the accompaniment data, obtain the amplitude from the sampling value and determine the change of the amplitude as the accompaniment progresses, that is, the amplitude change information, determine the change of the standard amplitude as the accompaniment progresses, that is, the standard amplitude change information, compare the amplitude change information with the standard amplitude change information, determine the difference between the amplitude change information and the standard amplitude change information, and determine the user's volume balance from the difference. Then determine the frequency change information from the frequency, determine the standard frequency change information from the standard frequency, use mathematical formula (1) to compare the frequency change information and the standard frequency change information, take the difference between the two values, and then divide by the standard value to obtain the percentage difference between the user's frequency and the standard frequency, that is, the user's off-tune rate. The user's sense of rhythm, the user's sense of volume balance and the user's off-tune rate are used as the user's singing status.
[0054] Through this solution, the user's singing progress and accompaniment progress information are compared to determine the user's sense of rhythm, the sampling value and frequency are obtained from the audio data, the amplitude is obtained from the sampling value and then the amplitude change information is obtained, the standard amplitude and standard frequency are obtained from the accompaniment data, the standard amplitude and amplitude are compared to reflect the user's sense of volume balance, the frequency change information and the standard frequency change information are compared, and the formula is used for calculation to quantify the user's out-of-tune rate. By comparing the user's singing progress with the accompaniment progress, analyzing the audio data and accompaniment data, the user's singing status is comprehensively evaluated from three dimensions: sense of rhythm, volume balance and out-of-tune rate, the amplitude and frequency change information is analyzed in detail and compared with the standard amplitude and frequency, and the volume balance problems and out-of-tune problems of the user during the singing process are accurately located.
[0055] In some embodiments, the frequency change information is compared with the standard frequency change information to determine the frequency difference information; the frequency difference information is analyzed to determine the frequency difference value and construct a frequency difference curve; the frequency difference curve is analyzed to divide the curve into a stable part and a fluctuating part and determine the frequency difference time; based on the fluctuating part of the curve, the fluctuation frequency and the fluctuation amplitude are determined; the frequency difference information and the frequency difference time are analyzed to determine the average frequency difference and the frequency difference standard deviation; a comprehensive score is determined based on the fluctuation frequency, frequency difference time, fluctuation amplitude, average frequency difference and the frequency difference standard deviation; based on the comprehensive score, the out-of-tune frequency information is determined.
[0056] The frequency difference information may be the difference between the value of the user frequency as the singing progresses and the value of the standard frequency followed by the accompaniment. The frequency difference value may be the difference between the value of the user frequency and the standard frequency. The frequency difference curve may be a curve constructed according to the change of the frequency difference value as the singing progresses. The stable part of the curve may be the part of the frequency difference curve where the frequency difference value remains stable. The stable part of the frequency does not mean absolute stability. Depending on the song category, this stability may deviate. For rock songs, the upper and lower differences of no more than 5% can be considered as a stable part. The fluctuating part of the curve may be the part of the frequency difference curve where the frequency difference value is unstable. The frequency difference time may be the total time when the frequency difference value fluctuates. The fluctuation frequency may be the number of times the fluctuation occurs per unit time. The fluctuation amplitude may be the maximum difference value of the frequency when the fluctuation occurs. The average frequency difference may be the sum of all the frequency difference values divided by the total time. The standard deviation of the frequency difference may be the arithmetic square root of the variance. The comprehensive score may be a score for evaluating whether the difference between the user's frequency and the original song is out of tune.
[0057] Specifically, in order to determine the frequency information of out-of-tune, it is necessary to first determine whether the song is out-of-tune, and the evaluation of whether the song is out-of-tune mainly depends on the difference between the user's frequency and the accompaniment data. First, the frequency difference information is obtained from the frequency change information and the standard frequency change information, and then the frequency difference value is determined and the frequency difference curve is constructed using Origin technology. Image processing technology and song style are used to identify the stable part and the fluctuating part of the curve. Then the time information is identified from the frequency difference curve, the frequency difference time is calculated, and then the fluctuating part of the curve is analyzed to calculate the fluctuation frequency and fluctuation amplitude. After obtaining the frequency difference information and frequency difference time, the average frequency difference and the standard deviation of the frequency difference are calculated. Mathematical analysis methods are used to obtain a comprehensive score based on the fluctuation frequency, frequency difference time, fluctuation amplitude, average frequency difference and standard deviation of the frequency difference. The comprehensive score is used to assess whether the song is out-of-tune and determine the frequency information of out-of-tune.
[0058] Through this scheme, the frequency change information and the standard frequency change information are compared, the frequency difference curve is constructed and the curve is divided into a stable part and a fluctuating part, which improves the recognition accuracy of the out-of-tune phenomenon. The frequency difference time is determined from the frequency difference curve, which can provide time data for the calculation. The fluctuation frequency and fluctuation amplitude are determined from the fluctuating part of the curve, and the frequency average difference and the frequency difference standard deviation are determined from the frequency difference information and the frequency difference time. The analysis of the out-of-tune characteristics is further refined. According to the fluctuation frequency, frequency difference time, fluctuation amplitude, frequency average difference and frequency difference standard deviation, a comprehensive score is determined to determine the out-of-tune frequency information. A plurality of out-of-tune related indicators are comprehensively considered to avoid the one-sidedness of a single indicator evaluation, making the evaluation of the out-of-tune frequency information more objective and comparable.
[0059] In some embodiments, the vibrato existence index is determined according to the fluctuation frequency, the frequency difference time and the fluctuation amplitude, referring to formula (2): (2); in, is the vibrato existence index, is the frequency difference time, is the fluctuation frequency, is the preset minimum fluctuation frequency, is the preset maximum value of the fluctuation frequency, is the fluctuation range, The minimum amplitude of the preset fluctuation; According to the frequency average difference, frequency difference time and frequency difference information, the pitch shift existence index is determined, referring to formula (3): (3); in, is the pitch shift existence index, is the frequency difference time, is the frequency difference, is the average frequency difference, The preset allowable pitch difference fluctuation range; According to the standard deviation of the frequency difference, the pitch stability score is determined, referring to formula (4): (4); in, Score the pitch stability, is the frequency difference time, is the standard deviation of the frequency difference, The maximum permissible value of the preset pitch difference; The comprehensive score is determined based on the vibrato presence index, pitch shift presence index, and pitch stability score, referring to formula (5): (5); in, For the comprehensive rating, is the frequency difference value, is the vibrato existence index, Score the pitch stability, It is the index of pitch shift existence; according to the comprehensive score, the frequency of out-of-tune is determined.
[0060] The vibrato existence index may be an index for determining whether the difference between the user frequency and the accompaniment frequency is caused by the vibrato technique. The pitch shift existence index may be an index for determining whether the difference between the user frequency and the accompaniment frequency is caused by the glissando technique. The pitch stability score may be a score for determining how close the user frequency is to the accompaniment frequency.
[0061] Specifically, in order to determine whether the song is out of tune, when there is a difference between the user frequency and the accompaniment frequency, some technical judgment is required. When the user's fluctuation frequency is between the preset minimum fluctuation frequency value and the preset maximum fluctuation frequency value and the user's amplitude exceeds the preset minimum amplitude, it is determined that vibrato exists. Formula (2) is used to obtain the vibrato existence index. When the vibrato existence index is 1, vibrato exists. When the vibrato existence index is 0, vibrato does not exist. The preset minimum fluctuation frequency value and the preset maximum fluctuation frequency value can be preset according to the song style. The preset minimum fluctuation amplitude can also be preset according to the song style. When the user's fluctuation frequency value is between the preset minimum fluctuation frequency value and the preset maximum fluctuation frequency value, and the fluctuation amplitude is greater than the preset minimum fluctuation amplitude, the vibrato existence index is determined to be 1. If one of the conditions is not met, it is 0.
[0062] According to the frequency average difference, frequency difference time and frequency difference information, the pitch shift existence index is obtained using formula (3). When the values of the frequency difference and the frequency average difference do not exceed the preset allowable pitch difference fluctuation range, the pitch shift existence index can be determined as 1, and it is determined as 0 in other cases. The allowable pitch difference fluctuation range is determined according to the music style. This range gives the singer the freedom to adapt. As long as the pitch shift is within the range, it will be considered as glissando.
[0063] A maximum allowable value of pitch difference is preset according to the song style and singing difficulty. According to the standard deviation of frequency difference, the pitch stability score is calculated using formula (4). When the standard deviation of frequency difference is larger, the pitch stability score is closer to 1, indicating that the pitch stability is worse. When the standard deviation is smaller, the difference from the maximum allowable value of pitch difference is larger, and the pitch stability score is closer to 0, indicating that the pitch stability is very good.
[0064] Finally, the vibrato index, pitch shift index and pitch stability score are combined and the comprehensive score is obtained using formula (5). When one or both of the vibrato index and the pitch shift index are 1, the comprehensive score is directly 0, indicating that there is singing skill. When the vibrato index and the pitch shift index are both 0, the comprehensive score is the frequency difference value multiplied by the pitch stability score. The higher the pitch stability, the lower the score, which means the degree of out-of-tune is small. When out-of-tune exists, the frequency of out-of-tune is determined according to the comprehensive score.
[0065] Through this scheme, according to the fluctuation frequency, frequency difference time and fluctuation amplitude, the vibrato existence index is determined by the formula to judge whether the frequency difference is caused by vibrato. According to the average frequency difference, frequency difference time and frequency difference information, the pitch shift existence index is determined by the formula to judge whether the frequency difference is caused by glissando. According to the standard deviation of the frequency difference, the pitch stability score is obtained. The three formulas determine three values, which provide a comprehensive, accurate and intuitive judgment for the evaluation of frequency differences, and also have an accurate judgment on singing skills. Finally, the three values and formulas are used to obtain a comprehensive score. The comprehensive score turns the abstract out-of-tune into intuitive data, and can accurately evaluate the degree of out-of-tune when it exists. Through a series of mathematical models and formulas, the sound signal is comprehensively analyzed, and a detailed evaluation of vibrato, pitch shift and pitch stability is provided.
[0066] In some embodiments, the user's sung part and the part to be sung are determined according to the singing progress; the expressiveness of the part to be sung is determined according to the user's sense of rhythm and volume balance; the overlapping segments of the part to be sung and the sung part are determined according to the sung part, the part to be sung and the accompaniment data; the overlapping segments and the out-of-tune frequency are used as parameters, input into a preset machine learning model, and the out-of-tune frequency of the overlapping part in the part to be sung is output; the accompaniment data and the out-of-tune frequency are analyzed to determine the out-of-tune probability of each range; according to the out-of-tune probability of each range, the out-of-tune frequency of the non-overlapping part in the part to be sung is determined; according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part, the out-of-tune frequency of the part to be sung is determined; the out-of-tune frequency of the overlapping part, the out-of-tune frequency of the non-overlapping part and the expressiveness are determined as the state of the user's part to be sung.
[0067] The expressiveness of the part to be sung may be the performance of the part to be sung by the user except for the pitch. The overlapping segment may be the part of a song with overlapping tunes. The probability of being out of tune in each range may be the ratio of the number of times each range is out of tune in the part sung by the user to the total number of each range in the part sung. The frequency of being out of tune in the non-overlapping part may be the frequency of being out of tune in the part to be sung by the user that does not overlap with the part already sung. The frequency of being out of tune in the part to be sung may be the frequency of being out of tune in the part to be sung by the user.
[0068] Specifically, first, according to the progress of singing, determine the part that has been sung and the part to be sung, and use mathematical analysis methods to determine the expressiveness of the part to be sung according to the user's sense of rhythm and volume balance. Determine the repeated segments from the accompaniment data, and obtain the time points of these segments in the accompaniment data, so as to determine whether the repeated segments exist in the part that has been sung and the part to be sung, so as to determine the overlapping segments of the part to be sung and the part that has been sung. After obtaining the overlapping segments, the out-of-tune frequency of the overlapping segments is obtained from the out-of-tune frequency of the sung part, and the out-of-tune frequency of the overlapping segments of the sung part is predicted by using a recurrent neural network in combination with the out-of-tune frequency of the overlapping segments of the sung part. For the non-overlapping part, statistics can be used to obtain the number of occurrences of each range from the accompaniment data, and then the number of out-of-tune of each range is obtained from the out-of-tune frequency, and the out-of-tune probability of each range is calculated. According to the out-of-tune probability of each range, the out-of-tune frequency of the non-overlapping part of the part to be sung is determined. The out-of-tune frequency of the overlapping part and the out-of-tune frequency and expressiveness of the non-overlapping part of the part to be sung are used as the state of the user's part to be sung.
[0069] Through this solution, the sung part and the part to be sung are divided according to the singing progress, which provides real-time information for determining the out-of-tune and expressiveness. The expressiveness of the part to be sung is determined in combination with the user's sense of rhythm and volume balance, and the prediction of the part to be sung is realized. Then, based on the sung part, the part to be sung and the accompaniment data, the overlapping segments of the sung part are determined, and the machine learning model is used to use the overlapping segments and the out-of-tune frequency as parameters to obtain the out-of-tune frequency of the overlapping part in the part to be sung. For the non-overlapping part, it is determined according to the out-of-tune probability of each range. The whole solution has a good effect in controlling the singing progress, analyzing the expressiveness of the part to be sung, identifying the overlapping segments and the out-of-tune frequency, evaluating the out-of-tune frequency of the part to be sung, and comprehensively evaluating the status of the user's part to be sung.
[0070] In some embodiments, accompaniment data is analyzed to determine the rhythm of the song and the main and chorus parts, and the number of rhythm words and rhythm words are planned according to the rhythm of the song; the average number of beats and drags of each rhythm word in the user's sung part is calculated according to the user's sense of rhythm and the number of rhythm words; the average number of beats and drags is used as the number of beats and drags of each line of lyrics in the part to be sung; the average decibel of the verse and the average decibel of the chorus are determined according to the sense of volume balance and the main and chorus parts; the average decibel of the verse and the average decibel of the chorus are determined as the decibel information of the part to be sung; based on the decibel information, the sense of volume balance of the part to be sung is determined; based on the decibel information, the expressiveness of the part to be sung is determined according to the number of beats and drags of each line of the part to be sung and the sense of volume balance of the part to be sung.
[0071] The rhythm of a song can be the speed of singing each line of lyrics in the song. The main chorus part can be a section of accompaniment in the song that belongs to the main song or chorus. The number of rhythm words can be the number of lyrics used to determine the user's grasp of the rhythm of the song. For example, when singing, the wrong rhythm only appears in the first few words of a line of lyrics, and the person will automatically align it in the later words. For example, if a song has a moderate rhythm, the number of rhythm words can be three. When the user does not catch the beat when singing, the first three words will not match, and the user will adjust to the matching rhythm later. The average number of beats can be the number of beats that the user beats or drags in each sentence when the rhythm is not matched. The average decibel of the verse can be the average decibel size when the user sings the content of the verse. The average decibel of the chorus can be the average decibel size when the user sings the content of the chorus. The decibel information can be the decibel size of the part to be sung by the user.
[0072] Specifically, according to the accompaniment data, the number of syllables per second is determined, the speed of the rhythm of the song is determined, and the verse part and the chorus part are determined from the accompaniment data. Then, according to the rhythm of the song, the number of rhythm words is planned, such as the number of rhythm words for a slow-tempo song can be 2. According to the number of rhythm words and the user's sense of rhythm, the number of words for each sentence in the sung part is calculated, and the average number of words for each sentence is calculated. Then, according to the sense of volume balance and the main and chorus parts, the average decibel in the verse part and the average decibel in the chorus part, as well as the average decibel of the verse and the average decibel of the chorus, are determined, and the average decibel of the verse and the average decibel of the chorus are determined as the decibels of the verse and the chorus corresponding to the part to be sung, and the sense of volume balance is determined according to the decibels of the corresponding verse and chorus compared with the standard decibel of the part to be sung, and finally the number of words for each sentence and the sense of volume balance are used as the expressiveness of the part to be sung. Expressiveness does not depend on the off-tune situation, but it will affect the handling of off-tune in the audio processing strategy. A person with good expressiveness may have a loud voice, that is, the volume balance is somewhat lacking, but one can feel the singer's high emotions. In this case, the comprehensive score of the user's off-tune will be higher than the comprehensive score under normal off-tune conditions.
[0073] Through this solution, the accompaniment data is analyzed to determine the rhythm of the song and the main and chorus parts, the number of rhythm words is divided from the rhythm of the song, and the average number of beats and delays of the rhythm words in each sentence in the sung part is calculated based on the user's sense of rhythm and the number of rhythm words. This provides a direct basis for evaluating the beats and delays of the part to be sung, and then the average decibels of the verse and the chorus are determined based on the sense of volume balance and the main and chorus parts, and the average decibels of the verse and the chorus are used as the average decibels of the verse and the chorus of the part to be sung, which fully considers the difference between the verse and the chorus in the song, making the decibel information prediction of the part to be sung more scientific and accurate.
[0074] In some embodiments, the user's heart rate and breathing are monitored to determine the heart rate and breathing conditions; the heart rate, breathing conditions and out-of-tune frequency are analyzed to determine the heart rate and the number of breaths when out-of-tune; gravity sensor information is acquired and analyzed to determine the user's motion amplitude information when singing; the motion amplitude information and the out-of-tune frequency are analyzed to determine the motion amplitude when out-of-tune; based on the heart rate, the number of breaths and the motion amplitude when out-of-tune, the average heart rate, the average number of breaths and the average motion amplitude when out-of-tune are calculated; based on the average heart rate, the average number of breaths and the average motion amplitude, the comprehensive score is modified to determine a corrected comprehensive score; based on the corrected comprehensive score, the corrected out-of-tune frequency is determined, and based on the corrected out-of-tune frequency, the corrected user out-of-tune rate is determined; based on the user's sense of rhythm, the user's volume balance and the corrected user out-of-tune rate, the user's singing status is determined.
[0075] The heart rate condition may be the heart rate information of the user during singing. The breathing condition may be the breathing information of the user during singing. The heart rate when out of tune may be the number of heart beats per minute when the user is out of tune. The number of respirations when out of tune may be the number of breaths per minute when the user is out of tune. The gravity sensor information may be the information monitored by the gravity sensor. The motion amplitude information may be the information on the intensity of the user's movements when singing. The average heart rate may be the average number of heart beats per minute when the user is out of tune. The average number of respirations may be the average number of breaths per minute when the user is out of tune. The average motion amplitude may be the average intensity of the user's movements when out of tune. The revised comprehensive score may be the comprehensive score adjusted after evaluating the physiological indicators of the user when singing. The revised out-of-tune frequency may be the out-of-tune frequency adjusted according to the revised comprehensive score. The revised user out-of-tune rate may be the user's out-of-tune rate adjusted according to the revised out-of-tune frequency.
[0076] Specifically, in a family singing scenario, there are many games, fights, etc. These external factors will cause the user's out-of-tune rate and frequency to increase. In this case, if the user's out-of-tune audio is blindly modified, it will be quite different from the actual situation, which will affect the user experience. Therefore, it is necessary to monitor the user's physiological indicators when singing to determine whether the user has a good singing environment. The heart rate sensor, breathing sensor, gravity sensor, etc. can be integrated into the microphone. The user's heart rate and breathing are monitored by the heart rate sensor and breathing sensor to determine the user's heart rate and breathing number when the user is out of tune. Then, the gravity sensor information is obtained and analyzed to determine the gravity change. The degree of gravity change is used as the movement amplitude information, and combined with the out-of-tune frequency, the user's movement amplitude information when out of tune is determined. According to the heart rate, breathing number and movement amplitude when out of tune, the average heart rate, average breathing number and average movement amplitude are determined, so as to determine whether these physiological indicators are abnormal. When there is an abnormality, the comprehensive score needs to be corrected to obtain a corrected comprehensive score, and then the corrected out-of-tune frequency is obtained. The corrected user out-of-tune rate is combined with the user's sense of rhythm and the user's volume balance as the user's singing status.
[0077] Through this solution, the out-of-tune frequency is obtained to determine the heart rate, breathing rate and movement amplitude information when out-of-tune, and the average heart rate, average breathing rate and average movement amplitude when out-of-tune are calculated to provide a reference for the subsequent correction of the comprehensive score. After obtaining the corrected comprehensive score, the corrected out-of-tune frequency is determined, and finally the corrected user out-of-tune rate is combined with the user's sense of rhythm and the user's sense of volume balance to determine the user's singing status. By monitoring the user's heart rate, breathing, and movement amplitude, a more comprehensive understanding of the user's physiological and behavioral status during the singing process can be obtained. This multi-dimensional data analysis helps to more accurately identify out-of-tune situations.
[0078] In some embodiments, the heart rate, breathing rate, and movement amplitude when out of tune are determined based on the heart rate, breathing rate, movement amplitude information, and out-of-tune frequency; the heart rate, breathing rate, and movement amplitude are determined as physiological indicators when out of tune; the average heart rate, average breathing rate, and average movement amplitude are determined as average physiological indicators when out of tune; the physiological indicators when out of tune are compared with the average physiological indicators when out of tune to determine the comparison results; based on the comparison results, the physiological indicators when out of tune that are greater than the average physiological indicators when out of tune are determined as physiological indicators of frequency that need to be corrected; based on the physiological indicators of frequency that need to be corrected and the out-of-tune frequency, the out-of-tune frequency is modified; based on the modified out-of-tune frequency, the comprehensive score is modified.
[0079] The physiological index when out of tune can be the heart rate, breathing rate and movement range when out of tune. The average physiological index when out of tune can be the average heart rate, average breathing rate and average movement range when out of tune. The physiological index of frequency to be corrected can be the physiological index situation that needs to be corrected for the frequency of out of tune Specifically, after monitoring the physiological condition of the singer, the comprehensive score needs to be corrected. The correction requires a reasonable correction strategy. First, the physiological index when out of tune is determined based on the heart rate, breathing rate and movement amplitude when out of tune. Then, the average physiological index is determined based on the average heart rate, average breathing rate and average movement amplitude when out of tune. The physiological index when out of tune is compared with the average physiological index. If the physiological index when out of tune is lower than the average physiological index, it means that the influence of the self-state caused by the external environment is very small. The existing out-of-tune situation is normal out-of-tune and no correction is required. When the physiological index when out of tune is higher than the average physiological index, it means that the external environment causes problems with the self-state. At this time, the frequency of out-of-tune needs to be corrected. The correction strategy is based on the physiological index when out of tune. For every 5% of the physiological index exceeding the average physiological index, the frequency of out-of-tune will be reduced by 5%. This exceeding means that one or more of them exceed the average physiological index. Modify the comprehensive score according to the modified frequency of out-of-tune, referring to the original formula.
[0080] Through this solution, physiological indicators when out of tune and average physiological indicators when out of tune are obtained, and the physiological indicators when out of tune and the average physiological indicators when out of tune are compared. The physiological indicators of the frequency that needs to be corrected are accurately located from the comparison results, providing a targeted basis for subsequent corrections. In addition, using physiological indicators as evaluation criteria is more objective and scientific than the traditional evaluation method that relies only on hearing or subjective feelings. This helps to reduce the interference of human factors, improve the accuracy and fairness of the evaluation, and modify the out-of-tune frequency according to the physiological indicators of the frequency that needs to be corrected and the out-of-tune frequency, and then modify the comprehensive score, providing a more scientific method for the evaluation of the comprehensive score.
[0081] Figure 3 A schematic diagram of a terminal data processing device for audio provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the terminal data processing device 300 for audio of this embodiment includes: a progress determination module 301, a state determination module 302, a state prediction module 303, a strategy formulation module 304 and a strategy execution module 305. Among them, the progress determination module 301 is used to obtain the user's real-time audio data and music accompaniment data, analyze the audio data and accompaniment data, and determine the user's singing progress; the state determination module 302 is used to determine the user's singing state according to the user's singing progress, audio data and accompaniment data; the state prediction module 303 is used to analyze the singing state and singing progress, and predict the state of the user's to-be-sung part; the strategy formulation module 304 is used to analyze the state of the to-be-sung part, audio data and accompaniment data, and determine the audio data processing strategy; the strategy execution module 305 is used to execute the audio data processing strategy, perform audio data processing, and output the processed audio data.
[0082] Optionally, the progress determination module 301 is specifically used to: accurately identify the lyrics and backing lyrics sung by the user based on the audio data; determine the melody data and accompaniment progress information based on the accompaniment data; divide the song intervals based on the melody data; determine the current singing interval based on the accompaniment progress information and the song interval; determine the user's singing progress based on the lyrics, backing lyrics, and the current singing interval.
[0083] Optionally, the state determination module 302 is specifically used to: compare the user's singing progress with the accompaniment progress information to determine the user's sense of rhythm; analyze the audio data to determine the sampling value and frequency; analyze the accompaniment data to determine the standard amplitude and standard frequency; determine the amplitude according to the sampling value; analyze the amplitude to determine the amplitude change information; analyze the standard amplitude to determine the standard amplitude change information; determine the user's voice size change information according to the amplitude change information; compare the amplitude change information with the standard amplitude change information to determine the user's volume balance; analyze the frequency to determine the frequency change information; analyze the standard frequency to determine the standard frequency change information; compare the frequency change information with the standard frequency change information to determine the out-of-tune frequency information; determine the out-of-tune frequency according to the out-of-tune frequency information; determine the user's out-of-tune rate according to the out-of-tune frequency, refer to the following formula: ; in, Indicates the user's out-of-tune rate. Indicates the frequency when out of tune, Indicates the standard frequency; determines the user's singing status based on the user's sense of rhythm, user's sense of volume balance and user's out-of-tune rate.
[0084] Optionally, the state determination module 302 is specifically used to: compare the frequency change information with the standard frequency change information to determine the frequency difference information; analyze the frequency difference information to determine the frequency difference value and construct a frequency difference curve; analyze the frequency difference curve to divide the curve into a stable part and a fluctuating part and determine the frequency difference time; determine the fluctuation frequency and the fluctuation amplitude according to the fluctuating part of the curve; analyze the frequency difference information and the frequency difference time to determine the average frequency difference and the frequency difference standard deviation; determine a comprehensive score according to the fluctuation frequency, frequency difference time, fluctuation amplitude, average frequency difference and the frequency difference standard deviation; determine the out-of-tune frequency information according to the comprehensive score.
[0085] Optionally, the state determination module 302 is specifically used to determine the vibrato existence index according to the fluctuation frequency, the frequency difference time and the fluctuation amplitude, referring to the following formula: ; in, is the vibrato existence index, is the frequency difference time, is the fluctuation frequency, is the preset minimum fluctuation frequency, is the preset maximum value of the fluctuation frequency, is the fluctuation range, is the preset minimum amplitude of fluctuation; according to the average frequency difference, frequency difference time and frequency difference information, the pitch shift existence index is determined, referring to the following formula: ; in, is the pitch shift existence index, is the frequency difference time, is the frequency difference, is the average frequency difference, is the preset allowable pitch difference fluctuation range; according to the frequency difference standard deviation, the pitch stability score is determined, referring to the following formula: ; in, Score the pitch stability, is the frequency difference time, is the standard deviation of the frequency difference, is the maximum allowable value of the preset pitch difference; the comprehensive score is determined according to the vibrato existence index, pitch shift existence index, and pitch stability score, referring to the following formula: ; in, For the comprehensive rating, is the frequency difference value, is the vibrato existence index, Score the pitch stability, It is the index of pitch shift existence; according to the comprehensive score, the frequency of out-of-tune is determined.
[0086] Optionally, the state prediction module 303 is specifically used to: determine the user's sung part and the part to be sung according to the singing progress; determine the expressiveness of the part to be sung according to the user's sense of rhythm and volume balance; determine the overlapping segment of the part to be sung and the sung part according to the sung part, the part to be sung and the accompaniment data; use the overlapping segment and the out-of-tune frequency as parameters, input the preset machine learning model, and output the out-of-tune frequency of the overlapping part in the part to be sung; analyze the accompaniment data and the out-of-tune frequency to determine the out-of-tune probability of each range; determine the out-of-tune frequency of the non-overlapping part in the part to be sung according to the out-of-tune probability of each range; determine the out-of-tune frequency of the part to be sung according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part; determine the out-of-tune frequency of the part to be sung according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part; determine the out-of-tune frequency of the overlapping part, the out-of-tune frequency of the non-overlapping part and the expressiveness as the state of the user's part to be sung.
[0087] Optionally, the state prediction module 303 is specifically used to: analyze the accompaniment data, determine the rhythm of the song and the main and chorus parts, and plan the number of rhythm words and rhythm words according to the rhythm of the song; calculate the average number of beats and drags of each rhythm word in the part sung by the user according to the user's sense of rhythm and the number of rhythm words; use the average number of beats and drags as the number of beats and drags of each sentence of the lyrics in the part to be sung; determine the average decibel of the main song and the average decibel of the chorus according to the sense of volume balance and the main and chorus parts; determine the average decibel of the main song and the average decibel of the chorus as the decibel information of the part to be sung; determine the volume balance of the part to be sung according to the decibel information; determine the expressiveness of the part to be sung according to the number of beats and drags of each sentence in the part to be sung and the sense of volume balance of the part to be sung.
[0088] Optionally, the device 300 also includes a score correction module 306, which is used to: monitor the user's heart rate and breathing to determine the heart rate and breathing conditions; analyze the heart rate, breathing and out-of-tune frequency to determine the heart rate and breathing number when out-of-tune; obtain and analyze gravity sensor information to determine the user's movement amplitude information when singing; analyze the movement amplitude information and the out-of-tune frequency to determine the movement amplitude when out-of-tune; calculate the average heart rate, average breathing number and average movement amplitude when out-of-tune based on the heart rate, breathing number and movement amplitude when out-of-tune; modify the comprehensive score based on the average heart rate, average breathing number and average movement amplitude to determine a corrected comprehensive score; determine the corrected out-of-tune frequency based on the corrected comprehensive score, and determine the corrected user out-of-tune rate based on the corrected out-of-tune frequency; determine the user's singing status based on the user's sense of rhythm, the user's volume balance and the corrected user out-of-tune rate.
[0089] Optionally, the score correction module 306 is specifically used to: determine the heart rate, breathing rate, and movement amplitude when out of tune according to the heart rate, breathing rate, movement amplitude information and out-of-tune frequency; determine the heart rate, breathing rate and movement amplitude as physiological indicators when out of tune; determine the average heart rate, average breathing rate and average movement amplitude as the average physiological indicators when out of tune; compare the physiological indicators when out of tune with the average physiological indicators when out of tune to determine the comparison results; according to the comparison results, determine the physiological indicators when out of tune that are greater than the average physiological indicators when out of tune as the physiological indicators of the frequency that needs to be corrected; modify the out-of-tune frequency according to the physiological indicators of the frequency that needs to be corrected and the out-of-tune frequency; and modify the comprehensive score according to the modified out-of-tune frequency.
Claims
1. A terminal data processing method for audio, characterized in that: include: Acquire the user's real-time audio data and music accompaniment data, analyze the audio data and the accompaniment data, and determine the user's singing progress; Determining the user's singing status according to the user's singing progress, the audio data and the accompaniment data; Analyze the singing status and the singing progress, and predict the status of the part to be sung by the user; Analyze the state of the part to be sung, the audio data and the accompaniment data, and determine an audio data processing strategy; The audio data processing strategy is executed to process the audio data and output the processed audio data.
2. The method according to claim 1, characterized in that The step of analyzing the audio data and the accompaniment data to determine the user's singing progress includes: According to the audio data, accurately identify the lyrics sung by the user and the backing words sung by the user; Determine the lyrics data and accompaniment progress information according to the accompaniment data; Dividing the song intervals according to the lyrics data; Determine the current singing interval according to the accompaniment progress information and the song interval; The user's singing progress is determined according to the lyrics, the backing words, and the current singing interval.
3. The method according to claim 2, characterized in that The step of determining the singing status of the user according to the singing progress of the user, the audio data and the accompaniment data comprises: Comparing the user's singing progress with the accompaniment progress information to determine the user's sense of rhythm; Analyze the audio data to determine sampling value and frequency; Analyzing the accompaniment data to determine a standard amplitude and a standard frequency; Determine the amplitude according to the sampled value; analyzing the amplitude to determine amplitude change information; Analyzing the standard amplitude to determine standard amplitude change information; Determining the user's voice volume change information according to the amplitude change information; Comparing the amplitude change information with the standard amplitude change information to determine the user's volume balance sense; analyzing the frequency to determine frequency change information; Analyzing the standard frequency to determine standard frequency change information; Comparing the frequency change information with the standard frequency change information to determine the out-of-tune frequency information; Determining the frequency of the out-of-tune according to the out-of-tune frequency information; According to the out-of-tune frequency, determine the user out-of-tune rate, referring to the following formula: ; in, represents the user's out-of-tune rate, Indicates the frequency of the out-of-tune, represents the standard frequency; The user's singing status is determined based on the user's sense of rhythm, the user's sense of volume balance and the user's out-of-tune rate.
4. The method according to claim 3, characterized in that The comparing the frequency change information with the standard frequency change information to determine the out-of-tune frequency information includes: Comparing the frequency change information with the standard frequency change information to determine frequency difference information; Analyzing the frequency difference information, determining the frequency difference value and constructing a frequency difference curve; Analyze the frequency difference curve, divide the curve into a stable part and a fluctuating part, and determine the frequency difference time; Determine the fluctuation frequency and fluctuation amplitude according to the fluctuation part of the curve; Analyze the frequency difference information and the frequency difference time to determine the average frequency difference and the frequency difference standard deviation; Determine a comprehensive score according to the fluctuation frequency, the frequency difference time, the fluctuation amplitude, the frequency average difference and the frequency difference standard deviation; According to the comprehensive score, out-of-tune frequency information is determined.
5. The method according to claim 4, characterized in that Determining the frequency of out-of-tune according to the comprehensive score includes: According to the fluctuation frequency, the frequency difference time and the fluctuation amplitude, the vibrato existence index is determined, referring to the following formula: ; in, There is an index for the vibrato, is the frequency difference time, is the fluctuation frequency, is the preset minimum fluctuation frequency, is the preset maximum value of the fluctuation frequency, is the fluctuation range, The minimum amplitude of the preset fluctuation; According to the average frequency difference, the frequency difference time and the frequency difference information, the pitch shift existence index is determined, referring to the following formula: ; in, There exists an index for the pitch shift, is the frequency difference time, is the frequency difference, is the frequency mean difference, The preset allowable pitch difference fluctuation range; Based on the standard deviation of the frequency difference, the pitch stability score is determined, referring to the following formula: ; in, Score the pitch stability, is the frequency difference time, is the standard deviation of the frequency difference, The maximum permissible value of the preset pitch difference; According to the vibrato existence index, the pitch shift existence index, and the pitch stability score, a comprehensive score is determined, referring to the following formula: ; in, For the comprehensive score, is the frequency difference value, There is an index for the vibrato, Score the pitch stability, There is an index for said pitch shift; Based on the comprehensive score, the frequency of the out-of-tune is determined.
6. The method according to claim 5, characterized in that The analyzing the singing status and the singing progress and predicting the status of the part to be sung by the user includes: Determining the part that has been sung and the part to be sung by the user according to the singing progress; Determining the expressiveness of the part to be sung according to the user's sense of rhythm and the sense of volume balance; Determine the overlapping segment of the part to be sung and the part to be sung according to the part already sung, the part to be sung and the accompaniment data; The overlapping segments and the out-of-tune frequency are used as parameters, input into a preset machine learning model, and the out-of-tune frequency of the overlapping portion in the part to be sung is output; Analyzing the accompaniment data and the out-of-tune frequency to determine the out-of-tune probability of each range; Determining the out-of-tune frequency of the non-overlapping part in the part to be sung according to the out-of-tune probability of each range; Determining the out-of-tune frequency of the part to be sung according to the out-of-tune frequency of the overlapping part and the out-of-tune frequency of the non-overlapping part; The out-of-tune frequency of the overlapping part, the out-of-tune frequency of the non-overlapping part and the expressiveness are determined as the state of the part to be sung by the user.
7. The method according to claim 6, characterized in that The step of determining the expressiveness of the part to be sung according to the user's sense of rhythm and the sense of volume balance includes: Analyze the accompaniment data to determine the rhythm of the song and the main and chorus parts, and plan the number of rhythm words and rhythm words according to the rhythm of the song; Calculate the average number of beats snatched and dragged for each rhythm word in the part sung by the user according to the user's sense of rhythm and the number of rhythm words; The average number of beats-forwarded and beat-backed is used as the number of beats-forwarded and beat-backed of each line of the lyrics in the to-be-sung part; Determine the average decibel level of the verse and the average decibel level of the chorus according to the volume balance and the verse and chorus parts; Determine the average decibel of the verse and the average decibel of the chorus as the decibel information of the part to be sung; Determine the volume balance of the part to be sung according to the decibel information; The expressiveness of the part to be sung is determined based on the number of beats in each sentence of the part to be sung and the volume balance of the part to be sung.
8. The method according to claim 6, characterized in that The step of determining the singing status of the user according to the singing progress of the user, the audio data and the accompaniment data further includes: Monitoring the user's heart rate and breathing to determine the heart rate and breathing conditions; Analyze the heart rate, the breathing and the out-of-tune frequency to determine the heart rate and the number of breathing when out-of-tune occurs; Acquire and analyze gravity sensor information to determine the movement amplitude information of the user when singing; Analyze the movement amplitude information and the out-of-tune frequency to determine the movement amplitude when out-of-tune occurs; Calculate the average heart rate, average breathing rate and average movement amplitude when out of tune according to the heart rate when out of tune, the number of breathing times when out of tune and the movement amplitude when out of tune; Modifying the comprehensive score according to the average heart rate, the average number of respirations and the average movement amplitude to determine a revised comprehensive score; Determine a corrected out-of-tune frequency according to the corrected comprehensive score, and determine a corrected user out-of-tune rate according to the corrected out-of-tune frequency; The user's singing status is determined based on the user's sense of rhythm, the user's sense of volume balance and the corrected user's out-of-tune rate.
9. The method according to claim 8, characterized in that The step of modifying the comprehensive score according to the average heart rate, the average number of respirations and the average movement amplitude includes: Determine the heart rate, breathing rate, and movement amplitude when out of tune according to the heart rate condition, the breathing condition, the movement amplitude information, and the out-of-tune frequency; Determine the heart rate, the breathing rate and the movement amplitude as physiological indicators of being out of tune; Determine the average heart rate, the average number of respirations and the average movement amplitude as average physiological indicators when out of tune; Comparing the physiological index during the out-of-tune period with the average physiological index during the out-of-tune period to determine a comparison result; According to the comparison result, the physiological index when the out-of-tune is greater than the average physiological index when the out-of-tune is determined as the physiological index of the frequency to be corrected; Modifying the out-of-tune frequency according to the physiological index of the frequency to be corrected and the out-of-tune frequency; The comprehensive score is modified according to the modified off-tune frequency.
10. A terminal data processing device for audio, characterized in that: include: The progress determination module is used to obtain the user's real-time audio data and music accompaniment data, analyze the audio data and the accompaniment data, and determine the user's singing progress; A status determination module, used to determine the user's singing status according to the user's singing progress, the audio data and the accompaniment data; A state prediction module, used to analyze the singing state and the singing progress, and predict the state of the part to be sung by the user; A strategy formulation module, for analyzing the state of the part to be sung, the audio data and the accompaniment data, and determining an audio data processing strategy; The strategy execution module is used to execute the audio data processing strategy, perform audio data processing, and output the processed audio data.
Citation Information
Patent Citations
Dynamic vocal accompaniment method and device
CN107093419A
Singing accompaniment automatic adjustment method, device and KTV jukebox
CN109272975A
Singing scoring method based on lyric and voice alignment
CN110660383A
Method and device for determining volume adjustment proportion information, equipment and storage medium
CN110688082A
Vocal music singing skill detection system
CN111382931A
