Audio tuning method, device and electronic equipment

Abstract

The present application relates to the technical field of audio processing, and in particular to an audio tuning method, device and electronic device, the method comprising: obtaining singing audio to be processed; performing speech recognition and pitch recognition on the singing audio to be processed to obtain audio description information and initial pitch information; performing boundary information correction on the initial pitch information according to the audio description information to obtain pitch information; correcting the original song pitch template corresponding to the singing audio to be processed according to the pitch information; determining the adjusted pitch information and performing tuning according to the corrected reference original song pitch template and pitch information. The present application combines the pitch recognition result and the speech recognition result to jointly determine the pitch information, and dynamically determines the reference original song pitch template based on the pitch information, so that the obtained adjusted pitch information is more accurate, and the pitch of the tuned audio is more consistent with the original song and the actual singing performance of the user himself, thereby improving the overall quality of the audio.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to an audio tuning method, device and electronic device. Background Art

[0002] With the development of mobile Internet, karaoke software has gradually become an important way for people to relax and entertain. The application of vocal tuning technology allows users to enjoy a better singing experience when singing karaoke, and even if their singing skills are average, they can still get a satisfactory singing effect.

[0003] Intelligent vocal tuning technology is a basic algorithm processing capability in the Internet's pan-entertainment karaoke scenarios. It analyzes the singer's vocal data and processes the vocal signal based on reference song information to obtain vocal audio that is closer to the original singer's singing level, greatly improving the singer's singing quality and producing high-quality music works.

[0004] In the related art, after obtaining the user's singing audio data, the singing audio data is analyzed to obtain pitch information; the pitch adjustment data of each note is determined by comparing the spectrum information with the obtained pitch information, and the singing audio data is adjusted according to the pitch adjustment data, so that the obtained audio is closer to the original audio in terms of pitch trend. Using only pitch information as a reference for tuning makes the tuning result rigid and the adjusted audio quality is average.

[0005] Therefore, how to provide a solution to improve the quality of sound correction has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0006] The purpose of this application is to provide an audio tuning method, device and electronic device, which can improve the tuning result.

[0007] In a first aspect, an audio tuning method is provided, comprising:

[0008] Get the singing audio to be processed;

[0009] Performing speech recognition on the singing audio to be processed to obtain audio description information; and performing pitch recognition on the singing audio to be processed to obtain initial pitch information;

[0010] Perform boundary information correction on the initial pitch information according to the audio description information to obtain pitch information;

[0011] Determine the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and correct the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain a reference original song pitch template;

[0012] According to the reference original song pitch template and the pitch information, the adjusted pitch information is determined, and the singing audio to be processed is tuned based on the adjusted pitch information to obtain the tuned audio.

[0013] In a preferred example, the present application may be further configured as follows: performing speech recognition on the singing audio to be processed to obtain audio description information includes: extracting acoustic features of the singing audio to be processed;

[0014] According to the acoustic features and the decoding graph, the acoustic features are decoded to obtain a phoneme sequence, wherein the decoding graph is generated based on a phoneme-level acoustic model, a lyrics-based language model, and a phoneme-lyrics conversion table;

[0015] The phoneme sequence is mapped to the lyrics text to obtain audio description information.

[0016] In a preferred example, the present application may be further configured as follows: the audio description information includes: lyrics information and boundary time information of phonemes;

[0017] The step of performing boundary information correction on the initial pitch information according to the audio description information to obtain the pitch information includes:

[0018] Filter invalid pitch information in the initial pitch information according to boundary time information corresponding to the phonemes in the audio description information;

[0019] Obtain the front pitch information corresponding to the front singing audio and the rear pitch information corresponding to the rear singing audio of the singing audio to be processed;

[0020] The filtered pitch information is subjected to overall pitch adjustment according to the front pitch information and / or the rear pitch information.

[0021] In a preferred example, the present application may be further configured as follows: the pitch adjustment information is used to adjust the pitch of the singing audio to be processed to obtain the adjusted audio, including:

[0022] Perform noise reduction filtering on the processed singing audio based on the first method and / or the second method to obtain filtered audio;

[0023] Performing pitch adjustment on the filtered audio based on the pitch adjustment information to obtain pitch-adjusted audio;

[0024] Among them, the first method is to first remove the background music and environmental noise in the audio based on the Speak-X module, and then use the neural network model to extract the human voice; the second method is to eliminate the background noise in the audio based on the multi-subband neural network model.

[0025] In a preferred example, the present application can be further configured as follows: the original song pitch template corresponding to the to-be-processed singing audio is corrected according to the pitch information to obtain a reference original song pitch template, including:

[0026] Determine the pitch mean and the pitch of the individual sound corresponding to the singing audio to be processed based on the pitch information;

[0027] If the difference between the pitch mean and the target pitch mean corresponding to the original song pitch template exceeds a first preset pitch threshold, the pitch of the original song pitch template is adjusted as a whole to obtain a first original song pitch template, and the difference between the pitch mean and the target pitch mean corresponding to the first original song pitch template does not exceed the first preset pitch threshold;

[0028] If the difference between the pitch of the individual sound and the pitch of the target sound in the first original song pitch template exceeds a second preset pitch threshold, a first pitch set is determined, wherein the target sound corresponds to the individual sound, and the first pitch set is a pitch set of the target sound with a preset pitch difference;

[0029] If there is a pitch in the first pitch set whose difference from the pitch of the individual sound is within a second preset pitch threshold, the pitch whose difference from the pitch of the individual sound is within the second preset pitch threshold is used as the dynamic adjustment target of the target sound;

[0030] If there is no pitch in the first pitch set whose difference with the pitch of the single tone is within the second preset pitch threshold, then determine a second pitch set, where the second pitch set is a set of in-key tones of the original song; select a dynamic adjustment target that meets the target condition from the second pitch set, where the target condition includes: an in-key tone that is closest to the target tone and whose difference with the pitch of the single tone is within the second preset pitch threshold;

[0031] After determining the dynamic adjustment targets of all individual sounds, the first original song pitch template is adjusted according to the dynamic adjustment targets of all individual sounds to obtain a reference original song pitch template.

[0032] In a preferred example, the present application may be further configured as follows: after determining the dynamic adjustment targets of all individual sounds, adjusting the first original song pitch template according to the dynamic adjustment targets of all individual sounds, and obtaining the reference original song pitch template, further comprising at least one of the following:

[0033] The reference target signal whose time domain variation coefficient in the original song pitch template exceeds a preset multiple is processed by white noise insertion, and the start position is ensured to be aligned, so as to obtain the aligned reference original song pitch template;

[0034] When the target tone in the first original song pitch template corresponding to the individual tone cannot be determined, the pitch of the first original song pitch template is adjusted as a whole to an in-key tone, and the pronunciation start point is adjusted to a preset time position.

[0035] In a preferred example, the present application may be further configured as follows: after determining the adjustment pitch information according to the reference original song pitch template and the pitch information, the present application may further include:

[0036] Acquire a secondary neural network vocoder, wherein the secondary neural network vocoder includes a first GAN network structure and a second GAN network structure;

[0037] According to the singing audio to be processed, a speed-changing and pitch-changing process is performed using a first GAN network structure to obtain an initial audio;

[0038] Based on the initial audio, the second GAN network structure is used to increase the sampling rate to obtain high-quality singing audio to be processed.

[0039] In a preferred example, the present application may be further configured as follows: after determining the adjustment pitch information according to the reference original song pitch template and the pitch information, the present application may further include:

[0040] According to the single word of the pitch information adjustment, determining the first music score information corresponding to each single word from the midi file of the original song;

[0041] According to each phoneme of the single word of the pitch information adjustment, determining the second music score information corresponding to each phoneme from the first music score information;

[0042] According to each waveform of the phoneme, determining the third music score information corresponding to each waveform from the second music score information;

[0043] The final adjusted pitch information is determined according to the third music score information corresponding to all waveforms to achieve smooth processing of the adjusted pitch information.

[0044] In a second aspect, an audio tuning device is provided, comprising:

[0045] An audio acquisition module is used to acquire the singing audio to be processed;

[0046] A recognition module is used to perform speech recognition on the singing audio to be processed to obtain audio description information; and perform pitch recognition on the singing audio to be processed to obtain initial pitch information;

[0047] A boundary correction module, used for performing boundary information correction on the initial pitch information according to the audio description information to obtain pitch information;

[0048] A dynamic template generation module is used to determine the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and to correct the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain a reference original song pitch template;

[0049] The tuning module is used to determine the adjusted pitch information according to the reference original song pitch template and the pitch information, and to tune the singing audio to be processed based on the adjusted pitch information to obtain the tuned audio.

[0050] In a third aspect, an electronic device is provided, including:

[0051] one or more processors;

[0052] Memory;

[0053] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and one or more programs are configured to: perform operations corresponding to the method shown in any possible implementation of the first aspect.

[0054] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded by a processor and executes the steps of the method shown in any possible implementation of the first aspect.

[0055] In a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, operations corresponding to the method shown in any possible implementation manner in the first aspect are implemented.

[0056] In summary, the audio tuning method provided by the present application includes beneficial technical effects: performing speech recognition and pitch recognition on the processed singing audio to obtain audio description information and initial pitch information respectively, so as to understand the content and pitch characteristics of the audio at the same time; using the audio description information to correct the boundary information of the initial pitch information, effectively removing the noise or invalid parts in the pitch information, and improving the accuracy of the pitch information; adaptively reconstructing the pitch template according to the pitch information, and obtaining a reference original song pitch template that allows the speed change and pitch change range to be generated; based on the corrected reference original song pitch template and pitch information, determining to adjust the pitch information and performing tuning, thereby achieving accurate tuning of the singing audio, so that the pitch of the tuned audio is more consistent with the original song and more consistent with the actual singing performance of the user himself, thereby improving the overall quality of the audio.

[0057] The present application also provides an audio tuning device and an electronic device, both of which have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 A schematic diagram of a process for voice tuning in the related art;

[0060] Figure 2 A schematic diagram of a scenario in which an audio tuning method provided in an embodiment of the present application is applied;

[0061] Figure 3 It is a flowchart of an audio tuning method provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of a smoothing process provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of a specific audio tuning solution is provided for the embodiment of the present application;

[0064] Figure 6 A comparison chart of the sound correction effect mos provided in the embodiment of the present application;

[0065] Figure 7 is a schematic diagram of the structure of a device provided in an embodiment of the present application;

[0066] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed. However, as long as they are within the scope of the present application, they are protected by the patent law.

[0068] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0071] Intelligent vocal tuning technology is now widely used in karaoke APP applications. It is a basic algorithm processing capability in the Internet pan-entertainment karaoke scene. Through machine learning algorithms, the singer's vocal data is analyzed, and the vocal signal is processed according to the reference song information to obtain a singing audio that is closer to the original singer's singing level, which greatly improves the singer's singing quality and produces high-quality music works. At the same time, intelligent tuning technology will greatly reduce the participation threshold of ordinary karaoke users and bring a stable increase in active users to the corresponding APP.

[0072] The main focus of related vocal tuning solutions when processing vocal data is only the pitch information, that is, the frequency information of the audio. The pitch information of the audio is analyzed through frequency analysis tools, and the pitch information is compared with the music score, and the pitch is adjusted in units of a complete note. In the adjustment process, an acoustic pitch change solution is often used to change the frequency domain audio, process it in segments, and then splice it to produce vocal data that is close to the original music score in pitch.

[0073] like Figure 1 As shown, Figure 1The figure is a flow chart of the vocal tuning process in the related art. In the related art, the vocal tuning solution has two core steps:

[0074] Singing analysis stage: When recording and obtaining the user's singing data, a third-party audio interface is usually used. The audio sampling rate is limited, and the highest sampling rate can often only reach 16K. Basic linear noise reduction can be used to obtain the cleanest audio possible. After obtaining the audio data, the pitch information of the audio is obtained through a pitch calculation tool, such as Figure 1 The diagram of A in the time-frequency (pitch) coordinate system is shown in the figure.

[0075] Vocal processing stage: According to the music score information (usually Figure 1 B in the figure, the information shown in the diagram in the time-pitch coordinate system), is compared with the obtained audio pitch information to determine the pitch adjustment data of each sound, that is, Figure 1 After obtaining the adjustment information, the frequency domain change tool is used to adjust the audio in the frequency domain; the final output audio is closer to the original audio in the frequency dimension, such as Figure 1 The C in the figure is represented in the time-frequency (pitch) coordinate system, which is the adjusted audio key information data, and the pitch trend is more consistent with the original singer, so as to obtain the processed audio data.

[0076] The related art only uses pitch information as a reference for tuning, and cannot make decisions based on comprehensive information. The intelligence level is low, and the tuning results are rigid. The related art only uses pitch information as a reference for tuning, and other information is ignored, such as text information, mode information, etc., and is not extracted to improve the intelligence level and make more reasonable tuning operations.

[0077] Therefore, improving the quality of music works produced by users in karaoke scenarios, lowering the threshold for participation, allowing users to more easily feel the fun and satisfaction of singing songs, and facilitating users to produce high-quality and appealing singing audio works; in music creation scenarios, it is convenient for authors to verify the effects of their own works more conveniently, and high-quality original music works can be produced without extremely high singing skills, which requires the focus of technical personnel in this field.

[0078] Based on this, see Figure 2 , Figure 2 This is a schematic diagram of a scenario in which an audio tuning method provided in an embodiment of the present application is applied. The audio tuning method can be applied to an audio tuning system.

[0079] In some embodiments, the audio tuning system may only include a first electronic device for collecting the singing audio to be processed and tuning the singing audio to be processed. The first electronic device may be a terminal device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a karaoke microphone, etc., but is not limited thereto.

[0080] In some embodiments, the audio tuning system may include a data collection device and a second electronic device, etc. The data collection device is used to collect the singing audio to be processed, and the second electronic device is used to implement the audio tuning method. The method provided in the embodiment of the present application can be performed by a second electronic device, and the second electronic device is a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The data collection device can be a terminal device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a karaoke microphone, etc., but is not limited to this. The data collection device and the second electronic device can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application is not limited here.

[0081] It is understandable that the above is only an example and is not limited to this embodiment.

[0082] A method for implementing audio tuning by an electronic device (a first electronic device / a second electronic device) is further described.

[0083] The electronic device may include a front-end speech recognition module, a singing pitch recognition module, and a sound correction processing module that combines information from multiple parties to make sound correction decisions and produce sound-corrected audio. The initial pitch information obtained by the pitch recognition obtained by the singing pitch recognition module is corrected using the audio description information obtained by the speech recognition module to obtain pitch information. The sound correction processing module is used to adaptively adjust the original song pitch template based on the pitch information to generate a new song template that allows a speed and pitch change range, so as to combine the pitch information and the new song template to jointly correct the singing data.

[0084] In some embodiments, the electronic device may further include a noise reduction filter module, which may be a noise reduction filter module with custom frequency band adjustment. The singing data is subjected to noise reduction processing by the streaming parallel noise reduction filter module. Further, the pitch correction processing module includes: an adaptive pitch correction template reconstruction unit, a smooth unit, and a speed change and pitch change unit.

[0085] Among them, after the user records the human voice singing audio, the singing audio to be processed passes through three data processing units at the same time: a speech recognition module built for the singing data, a pitch recognition module, and a noise reduction filter module with custom frequency band adjustment.

[0086] The audio description information generated by the speech recognition module and the initial pitch information generated by the pitch recognition module; the pitch information is obtained based on the audio description information and the initial pitch information; based on the pitch information, a reference original song pitch template that is allowed by the speed change and pitch change range is generated in the adaptive pitch change template reconstruction unit of the pitch change processing module; based on the pitch information, by referring to the original song pitch template, the accurate pitch information produced helps to improve the speed change and pitch change information to obtain the adjusted pitch information; the improved speed change and pitch change information (adjusted pitch information) passes through the smoothing unit to generate a more reliable small-grained audio operation information sequence for the speed change and pitch change unit; the noise reduction filter module outputs audio data as the operating material of the speed change and pitch change algorithm in the subsequent process. The speed change and pitch change are processed by the front-end link, and the audio is processed in parallel, and the audio processing is completed with an effectiveness of no more than rtf 0.03.

[0087] Specifically, the present application embodiment provides an audio tuning method, such as Figure 3 As shown, the method includes:

[0088] S101, obtaining singing audio to be processed;

[0089] The singing audio to be processed is the real-time singing audio of the user when singing karaoke.

[0090] S102, performing speech recognition on the processed singing audio to obtain audio description information; and performing pitch recognition on the processed singing audio to obtain initial pitch information;

[0091] Among them, when performing speech recognition on the singing audio, a machine learning algorithm can be used. Through the speech recognition algorithm aimed at improving the accuracy of boundary recognition, both semantic and boundary information can be obtained, that is, the audio description information includes but is not limited to the lyrics information and the boundary time information corresponding to each phoneme, the time boundary information of each phoneme, that is, the start and end time points of each phoneme.

[0092] The initial pitch information is the original pitch contour of the song. The specific initial pitch information can be obtained using a fundamental frequency detection algorithm (such as an autocorrelation method, a cepstrum method, etc.).

[0093] S103, performing boundary information correction on the initial pitch information according to the audio description information to obtain pitch information;

[0094] In the related art, in the process of audio correction, it is difficult to perform precise processing in the time dimension. Only the pitch can be controlled, but the rhythm accuracy cannot be controlled. In addition, the basic unit for processing audio is an audio segment of a complete note length analyzed in the singing analysis stage. The overall frequency multiplier is adjusted at this time length. Due to the inaccuracy in time, this audio segment may not necessarily be the audio that needs to be adjusted in frequency under ideal circumstances. Uniformly adjusting a fixed multiplier will cause some high-frequency sounds in the audio and small audio segments without stable fundamental frequencies to be incorrectly adjusted, which is intuitively manifested as what we often call "popping sound", "electrical sound in the audio", "audio pause", and "audio freeze". However, in the embodiment of the present application, based on the result of pitch recognition, that is, the initial pitch information, the boundary time information corresponding to the phonemes of the accurate speech recognition result will be combined to filter out invalid pitch information in the initial pitch information, such as the pitch information corresponding to non-speech segments (such as silence, noise segments). These segments usually show unstable or extremely low / high pitch values, so as to facilitate the calibration of the boundaries for the inherent pitch information and ensure that the pitch information of each valid speech segment (mainly vowel segments) is strictly aligned with the phoneme boundaries in the speech recognition result in the time dimension, so that the pitch data can be accurately intercepted and processed in the time dimension.

[0095] In the embodiment of the present application, the traditional pitch extraction algorithm and the speech recognition result are cross-checked to supplement the accurate boundary information for the pitch information, making the obtained pitch information more accurate.

[0096] Furthermore, in order to accurately correct the pitch, in the embodiment of the present application, the initial pitch information can be corrected again in combination with the pitch information corresponding to the front and rear audios of the singing audio to be processed. Specifically, the invalid pitch information in the initial pitch information is filtered out according to the boundary time information corresponding to the phonemes in the audio description information; the front pitch information corresponding to the front singing audio and the rear pitch information corresponding to the rear singing audio of the singing audio to be processed are obtained; and the overall pitch adjustment is performed on the filtered pitch information according to the front pitch information and / or the rear pitch information.

[0097] According to the front pitch information and / or the rear pitch information, an overall correction value is jointly determined, and the filtered pitch information is re-corrected using the overall correction value. In one feasible manner, the overall correction value can be obtained based on the pitch mean of the front audio or the pitch mean of the rear audio, and the pitch mean of the singing audio to be processed is subtracted; in another feasible manner, the average of the pitch mean of the front audio and the pitch mean of the rear audio is calculated to obtain a comprehensive pitch mean, and the overall correction value is obtained by subtracting the pitch mean of the singing audio to be processed from the comprehensive pitch mean. The specific method to be selected is not limited in the embodiments of the present application, and the user can choose according to actual needs.

[0098] Through the above technical scheme, the phoneme boundary time information in the audio description information is utilized to effectively filter out the invalid parts in the initial pitch information, thereby reducing the noise interference in pitch recognition; combining the pitch information of the previous singing audio and the subsequent singing audio, the filtered pitch is adjusted as a whole, so that the corrected pitch information is smoother and conforms to the laws of music.

[0099] S104, determining the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and correcting the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain a reference original song pitch template;

[0100] According to the audio description information and the reference description information, the target description information corresponding to the audio description information in the reference description information is determined, the lyrics information of the target description information corresponds to the lyrics information of the audio description information, and the time information of the phoneme boundary of the target description information corresponds to the time information of the phoneme boundary of the audio description information; and based on the target description information, the original song pitch template corresponding to the singing audio to be processed is determined. The original song pitch template refers to the ideal pitch sequence of each note or syllable in the original song, which represents the standard pitch contour of the song.

[0101] The reference original song pitch template is used to indicate a pitch template that is adjusted according to the actual pitch of the singer while retaining the pitch characteristics of the original song. The reference original song pitch template is closer to the actual singing situation of the singer.

[0102] S105. Determine the pitch adjustment information based on the pitch template and pitch information of the original song, and perform tuning on the singing audio to be processed based on the pitch adjustment information to obtain the tuned audio.

[0103] By referring to the pitch difference between the original song pitch template and the pitch information, the adjusted pitch information is obtained, and then the adjusted pitch information is used to tune the singing audio, so that the final tuned audio is more reasonable, thereby improving the tuning result.

[0104] It can be seen that in the embodiment of the present application, speech recognition and pitch recognition are performed on the singing audio to be processed, and audio description information and initial pitch information are obtained respectively, so that the content and pitch characteristics of the audio can be understood at the same time; the initial pitch information is corrected by using the audio description information to perform boundary information correction, effectively removing noise or invalid parts in the pitch information, and improving the accuracy of the pitch information; the pitch template is adaptively reconstructed according to the pitch information, and a reference original song pitch template that allows the speed change and pitch change range is obtained; based on the corrected reference original song pitch template and pitch information, the pitch adjustment information is determined and the tuning is performed, thereby achieving accurate tuning of the singing audio, so that the pitch of the tuned audio is more consistent with the original song and more consistent with the actual singing performance of the user himself, thereby improving the overall quality of the audio.

[0105] In a possible implementation of the embodiment of the present application, in S102, speech recognition is performed on the singing audio to be processed to obtain audio description information, including: S1021-S1023, wherein:

[0106] S1021, extracting acoustic features of the singing audio to be processed;

[0107] Extract standard features (F BANK), i.e. acoustic features, from the singing audio to be processed.

[0108] Furthermore, the singing audio to be processed can be subjected to additional feature processing, such as processing pitch and tone changes, and then feature extraction can be performed. Moreover, the extracted acoustic features can also be subjected to feature normalization processing to ensure that the acoustic features meet certain standards.

[0109] S1022, decoding the acoustic features according to the acoustic features and the decoding graph to obtain a phoneme sequence, wherein the decoding graph is generated based on a phoneme-level acoustic model, a lyrics-based language model, and a phoneme-lyrics conversion table;

[0110] In the embodiment of the present application, in order to improve the boundary recognition capability, an acoustic model in which the phoneme information is a single tone and a speech model based on lyrics are adopted.

[0111] The redesigned speech recognition module has a boundary recognition accuracy of 93.2% at the 30ms level.

[0112] In some embodiments, in order to obtain more accurate speech recognition results, the obtained decoding graph can more accurately recognize the speech. In the embodiment of the present application, when generating the decoding graph, an acoustic model, a language model and a pronunciation dictionary (phoneme-lyrics conversion table) are used to finally obtain the decoding graph, wherein the generation process of the decoding graph may include:

[0113] A1. Obtain audio data. The audio data includes: ordinary voice data and singing audio data. Ordinary voice data: includes various daily conversations, speeches, etc.; singing audio data: collect a large amount of singing audio data to ensure data diversity, including songs of different styles, languages ​​and tones.

[0114] For ordinary speech data, the transcribed text can be annotated at the phoneme level; for singing audio data, the lyrics text can be annotated at the phoneme level considering that the pronunciation of singing may be different.

[0115] Extract standard features from the audio data. Additional feature processing can also be performed on the singing data, such as processing pitch and tone changes.

[0116] The extracted standard features are normalized to ensure that all data are trained under the same standard.

[0117] A2. Acoustic model training. Use the phoneme alignment tool to align the standard features with the phoneme-level annotations; use phonemes as modeling units to train the acoustic model, combining phoneme labels with deep learning techniques (such as DNN, CNN, or LSTM) for training. Since singing audio data is involved in the training, the resulting acoustic model can process audio with a singing style.

[0118] A3. Language model training. The language model can include an HMM structure (Hidden Markov Model) based on lyrics, taking lyrics as hidden states, and learning the relationship between phoneme sequences and lyrics through training; creating an n-gram language model based on lyrics or a language model based on RNN, focusing on the language structure and common vocabulary of lyrics;

[0119] A4. Phoneme-to-vocabulary conversion table, which maps phoneme sequences to lyrics text in order to better combine the phoneme-level model and the lyrics-level language model.

[0120] A5. Generate a decoding graph based on the phoneme-level acoustic model, the lyrics-based language model, and the phoneme-lyrics conversion table.

[0121] In an embodiment of the present application, after obtaining the decoding graph, the singing audio to be processed is decoded according to the acoustic features and the decoding graph to determine the recognition results of the words in the singing audio to be processed.

[0122] S1023: Map the phoneme sequence to the lyrics text to obtain audio description information.

[0123] The audio description information includes lyrics information and boundary time information.

[0124] It can be seen that in the embodiment of the present application, by extracting the acoustic features of the singing audio to be processed, and decoding it using a decoding graph generated by an acoustic model based on the phoneme level, a language model mainly based on lyrics, and a phoneme-lyrics conversion table, an accurate phoneme sequence is obtained, which not only improves the accuracy of speech recognition, but also ensures that the recognition results can accurately reflect the lyrics content in the audio; the phoneme sequence is then mapped to the lyrics text, and accurate audio description information containing lyrics information and phoneme boundary time information is obtained.

[0125] Furthermore, after obtaining the audio description information, for some songs whose endings may be relatively long, a post-processing process of boundary operations can be performed, such as boundary detection to obtain the time position information of the separation between lines of lyrics and the natural breaks of words in the lyrics, and then smooth the boundaries of the lyrics based on the time position information to make them more in line with the segmentation of natural language; and, in the singing data, the impact of changes in pitch and tone on the boundaries can be adjusted to ensure the accuracy of the recognition results.

[0126] Furthermore, in the karaoke scene, due to the uncertainty of the gap between the recorded singing audio to be processed and the original song template, it is difficult to determine a safe audio adjustment range. In this application, the template is adaptively rebuilt according to the recorded singing audio to be processed, so that the adjustment range of the new template matches the speed change algorithm capability.

[0127] Specifically, the pitch-correction template reconstruction module of the embodiment of the present application refers to the song mode and chord progression, and calculates the pitch of replacement notes of different recommended levels at each point (note) to achieve reconstruction of the score pitch template.

[0128] A possible implementation of the embodiment of the present application is to correct the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain a reference original song pitch template, including:

[0129] Determine the pitch mean and the pitch of the individual sound corresponding to the singing audio to be processed based on the pitch information;

[0130] If the difference between the pitch mean and the target pitch mean corresponding to the original song pitch template exceeds the first preset pitch threshold, the pitch of the original song pitch template is adjusted as a whole to obtain a first original song pitch template, and the difference between the pitch mean and the target pitch mean corresponding to the first original song pitch template does not exceed the first preset pitch threshold;

[0131] If the difference between the pitch of the individual sound and the pitch of the target sound in the first original song pitch template exceeds a second preset pitch threshold, a first pitch set is determined, wherein the target sound corresponds to the individual sound, and the first pitch set is a pitch set of the target sound that differs by a preset pitch;

[0132] If there is a pitch in the first pitch set whose difference from the pitch of the single tone is within the second preset pitch threshold, the pitch whose difference from the pitch of the single tone is within the second preset pitch threshold is used as the dynamic adjustment target of the target tone;

[0133] If there is no pitch in the first pitch set whose pitch difference with the single tone is within the second preset pitch threshold, then determine a second pitch set, the second pitch set being a set of in-key tones of the original song; select a dynamic adjustment target that meets the target conditions from the second pitch set, the target conditions including: an in-key tone that is closest to the target tone and whose pitch difference with the single tone is within the second preset pitch threshold;

[0134] After determining the dynamic adjustment targets of all individual sounds, the first original song pitch template is adjusted according to the dynamic adjustment targets of all individual sounds to obtain a reference original song pitch template.

[0135] Specifically, when the pitch deviation of the tuning target is too large or there is a large-scale change in the time domain, which may far exceed the design capability of the speed-changing and pitch-changing module, the tuning reference target is dynamically adjusted to match the capability of the speed-changing and pitch-changing module, where:

[0136] Determine whether the difference (in absolute value form) between the pitch mean of the entire segment and the target pitch mean of the song segment itself exceeds the first preset pitch threshold. The first preset pitch threshold can be set according to actual needs. For example, the first preset pitch threshold can be 10 semitones. If it exceeds the first preset pitch threshold, the input audio cannot be forced to approach the target audio, which will exceed the algorithm capability limit and cause serious sound quality degradation. In the embodiment of the present application, the original song pitch template is adjusted as a whole by an integer octave. Adjusting 8 degrees will not affect the use of accompaniment and other music resources until the difference in the mean of the entire segment just does not exceed the first preset pitch threshold, and the first original song pitch template is obtained. If the pitch mean is greater than the target pitch mean, the original song pitch template is adjusted up, and if the pitch mean is not greater than the target pitch mean, the original song pitch template is adjusted down; if it does not exceed the first preset pitch threshold, the original song pitch template is used as the first original song pitch template;

[0137] For each individual sound of the entire segment, determine whether the difference (in absolute value form) between the pitch of the individual sound and the pitch of the target sound in the first original song pitch template exceeds a second preset pitch threshold, wherein the second preset pitch threshold can be set according to actual needs, and illustratively, the second preset pitch threshold is 8 semitones, and the second preset pitch threshold is less than the first preset pitch threshold;

[0138] If the pitch does not exceed the second preset pitch threshold, the pitch of the target sound in the first original song pitch template is kept unchanged;

[0139] If it exceeds the second preset pitch threshold, the pitch of the target sound whose pitch is closer to the preset pitch is selected as the first pitch set. The preset pitch can be set according to actual needs. For example, the preset pitch can be 2 degrees. If there is a pitch in the first pitch set whose difference with the pitch of the individual sound is within the second preset pitch threshold, the pitch whose difference with the pitch of the individual sound is within the second preset pitch threshold is used as the dynamic adjustment target of the target sound. For example, the pitch of the individual sound is a, and the pitch of the target sound is b=a+9 semitones. At this time, the determined first pitch set is a+11, a+7. At this time, b is modified to a+7.

[0140] If there is no pitch in the first pitch set whose pitch difference with the single tone is within the second preset pitch threshold, then the in-key tones are taken as the second set, and a number of first in-key tones whose pitch difference with the single tone is within the second preset pitch threshold are selected from the second pitch set, and the second in-key tones closest to the target tone are selected from the first in-key tones as the dynamic adjustment target. Specifically, the in-key tones in the second set whose pitch difference with the single tone is from near to far can be traversed, and the tone closest to the target tone and whose pitch difference with the single tone is within 8 semitones can be found as the dynamic adjustment target.

[0141] After determining the dynamic adjustment targets of all individual sounds, adjust the first original song pitch template according to the dynamic adjustment targets of all individual sounds to obtain the reference original song pitch template. Through the DTW (Dynamic Time Warping) algorithm, the distance calculation expression is constructed with the mode to obtain the most economical new template, that is, to provide different tuning strategies for different audios.

[0142] It can be seen that in the embodiment of the present application, when the pitch deviation of the tuning target to be adjusted is too large to the point that it is likely to far exceed the design capacity of the speed-changing and pitch-changing module, the tuning reference target is dynamically adjusted to match the capacity of the speed-changing and pitch-changing module. Specifically, the overall template is first adjusted based on the pitch mean, and after the adjustment, a secondary adjustment is performed based on the difference between the individual sound and the target sound to achieve score reconstruction.

[0143] A possible implementation of the embodiment of the present application, after determining the dynamic adjustment targets of all individual sounds, adjusting the first original song pitch template according to the dynamic adjustment targets of all individual sounds, and obtaining the reference original song pitch template, further includes:

[0144] The reference target signal whose time domain variation coefficient in the reference original song pitch template exceeds a preset multiple is processed by white noise insertion, and the starting position is ensured to remain aligned to obtain the aligned reference original song pitch template.

[0145] The time domain variation coefficient can be customized by the user. For example, the time domain variation coefficient is 2.

[0146] In an embodiment of the present application, when the tuning target undergoes a large-scale change in the time domain, which may far exceed the design capability of the speed change and pitch change module, the tuning reference target is dynamically adjusted to achieve score adjustment.

[0147] It can be seen that the reference target signal whose time domain variation coefficient in the original song pitch template exceeds the preset multiple is processed by white noise insertion, which effectively smooths the abnormal or abrupt pitch changes in the template and avoids the adverse effects of extreme pitch changes on the subsequent tuning process. At the same time, by ensuring that the starting position of the white noise insertion is aligned with the original signal, the continuity and consistency of the processed template on the time axis are ensured, making the aligned reference original song pitch template more in line with the natural laws of music.

[0148] A possible implementation of the embodiment of the present application, after determining the dynamic adjustment targets of all individual sounds, adjusting the first original song pitch template according to the dynamic adjustment targets of all individual sounds, and obtaining the reference original song pitch template, further includes:

[0149] When the target tone in the first original song pitch template corresponding to the individual tone cannot be determined, the pitch of the first original song pitch template is adjusted as a whole to an in-key tone, and the pronunciation start point is adjusted to a preset time position.

[0150] Among them, in the case of singing the wrong lyrics, when the target reference tuning target cannot be determined, the pitch is adjusted to the corresponding in-key tone of the paragraph, and the pronunciation starting point is adjusted to a time position that is an integer multiple of the 1 / 16 beat position.

[0151] It can be seen that when the target tone in the first original song pitch template corresponding to a single tone cannot be determined, the harmony of the overall pitch of the template can be achieved to a certain extent through global adjustments, and the position of the pronunciation point is adjusted to standardize the timing characteristics of the target tone.

[0152] In the embodiment of the present application, the audio adjustment amplitude can be controlled within a certain range through the template reconstruction algorithm, and the audio can all pass through the audio correction module, so that the output sound quality damage rate is lower than the acceptable range of streaming music products (0.005%). After all audios are corrected, all audios are placed in the accompaniment, and their rhythm and harmony are fully guaranteed.

[0153] In a possible implementation of the embodiment of the present application, after obtaining the pitch adjustment information, the singing audio to be processed needs to be tuned. At this time, the quality of the singing audio to be processed is crucial to the effect of the tuning. In order to improve the tuning effect, in the embodiment of the present application, sound enhancement processing can be performed to obtain cleaner vocal data.

[0154] For sound enhancement technology, it can be implemented by using a noise reduction module that eliminates sudden noise and continuous noise (method two), or by using an echo cancellation module that eliminates leaked background music (method one); of course, it can also be implemented by first using an echo cancellation module that eliminates leaked background music, and then using a noise reduction module that eliminates sudden noise and continuous noise. The present application embodiment is no longer limited, and users can set it according to actual needs. After obtaining the denoised singing audio to be processed, the tuned audio is obtained based on the denoised singing audio to be processed and the pitch information is adjusted, which further improves the effect of the tuning.

[0155] In one possible implementation, the process of achieving sound enhancement by using an echo cancellation module that eliminates leaked background music may include: based on the Speak-X module, processing the background music and environmental noise of the singing audio to be processed to obtain a first singing audio; using a neural network model to extract the human voice in the first singing audio to obtain a second singing audio.

[0156] Furthermore, the pitch information is adjusted based on the second singing audio to obtain the tuned audio.

[0157] The algorithm structure of front-end Speak-X and rear-end neural network is used to eliminate the background music produced by the playback device, and obtain a clearer human voice, that is, the second singing audio. This is particularly important in the scenario of external playback of the device, and can significantly improve the voice clarity.

[0158] The Speak-X module is a complex filter or acoustic model that focuses on the processing of background music. In the embodiment of the present application, the front Speak-X module is used to process background music and environmental noise to generate a clear audio signal, namely the first singing audio.

[0159] The training process of the subsequent neural network is further explained, including:

[0160] 1. Data preparation.

[0161] Collect audio data in a variety of environments, especially background music and clear vocals produced by playback devices, and collect audio data recorded from different types of music and vocals. Ensure that the data sampling rate can cover application requirements, and use a higher sampling rate (such as 48kHz or 32kHz) to retain sufficient audio details. In related technologies, third-party recording and noise reduction services are usually used, and the audio sampling rate is forcibly locked at 16K. This sampling rate meets the usage requirements in speaking scenarios, but in K-song scenarios, the texture of the singing voice at high frequencies will be significantly amplified by the sound effector, and the 16K sampling rate audio has a problem of missing information in the sound quality, especially in the high-frequency part. Therefore, in an embodiment of the present application, a high sampling rate can be used for model training during the training process, and a high sampling rate can also be used when recording the singing audio to be processed, so as to ensure sufficient details and reduce the occurrence of information loss in the high-frequency part.

[0162] Detailed annotation of audio data, including background music, vocals, and the separation between them.

[0163] Perform feature extraction on audio data, including short-time Fourier transform (STFT) or other time-frequency domain representations.

[0164] Furthermore, data enhancement can be performed, such as adding different types of noise, adjusting the volume, etc., to simulate changes in various actual usage scenarios.

[0165] Divide the collected audio data into training, validation, and test sets. Make sure the training set contains a variety of background music and vocal combinations.

[0166] 2. Model design and training.

[0167] The structure of the post-neural network model can adopt a hybrid architecture of convolutional neural networks (CNN) plus transformers to further process the output of the front module, enhance the clarity of the human voice, remove residual background noise or echo, and extract clear human voice. The number of layers and nodes of the neural network model is not limited in this embodiment of the application, as long as it can process complex audio signals.

[0168] Use supervised learning methods to train the neural network model and optimize the loss function (such as mean square error MSE) to minimize the gap between the predicted noise and the actual noise; based on the gap, adjust the model's hyperparameters, such as learning rate, batch size, number of network layers, etc., to achieve the best model performance.

[0169] Use the validation set to evaluate the performance of the trained neural network model, focusing on indicators such as echo cancellation effect, speech clarity, and sound quality. Based on the validation results, adjust the model architecture or training parameters to improve performance.

[0170] 3. Model testing and evaluation.

[0171] Use the test set to evaluate the actual performance of the verified neural network model to obtain the echo cancellation effect, speech clarity, and background music suppression effect. The test data should include audio from various device playback scenarios to verify the generalization ability of the model; and evaluate the echo cancellation effect, speech clarity, and background music suppression effect according to the set standards.

[0172] Furthermore, feedback can be collected from actual users to understand the performance of the neural network model in actual use, and the model can be further optimized based on user feedback.

[0173] In the embodiment of the present application, after obtaining the trained model, the trained model is integrated into the actual application, which can be the device corresponding to the karaoke scene, or the conference system or real-time communication system. Ensure that the deployment environment can process audio data with high sampling rate. In actual application, performance optimization can also be performed, such as accelerating the reasoning process and reducing computing resource consumption.

[0174] It can be seen that in the embodiment of the present application, real-time audio is processed in the device external speaker scenario, the front Speak-X module is used to process the background music, and then the clarity of the human voice is enhanced through the rear neural network module, and the clear human voice is output after echo cancellation and background music suppression processing, thereby ensuring better audio quality, and making the audio adjustment effect better after the audio adjustment is performed based on the pitch information.

[0175] In another possible implementation, the process of implementing sound enhancement by using a noise reduction module that eliminates sudden noise and continuous noise may include:

[0176] Using a multi-subband neural network model, background noise in the singing audio to be processed is eliminated to obtain a third singing audio;

[0177] According to the third singing voice audio, the pitch information is adjusted to obtain the tuned audio.

[0178] Specifically, obtain audio data in the karaoke scene, including singing audio and corresponding clean audio, ensure that the data contains a variety of styles, pitches and background noise conditions, and the data sampling rate must support more than 32KHz to meet the requirements of the high frequency range. Audio data can also be enhanced, including noise addition, volume adjustment, etc., to simulate the actual karaoke environment and sound quality changes. Label the lyrics and background noise in the audio data; convert the audio data into a time-frequency domain representation, use short-time Fourier transform or other suitable transforms, and for high sampling rate data, ensure that feature extraction can retain rich frequency information.

[0179] A neural network model based on multiple subbands is constructed. The number of subbands is no longer limited in the embodiment of the present application and can be 24, 20, etc., wherein each subband is responsible for processing speech signals in a specific frequency band. The hidden layer configuration of the multi-subband neural network model can be fixed at 512 and 384 nodes to increase the network's expressive power and ability to process complex audio signals. In addition, the multi-subband neural network model includes multiple convolutional layers (for time-frequency feature extraction), fully connected layers (for feature fusion), and activation functions (such as ReLU).

[0180] Divide the audio data into training set, validation set and test set to ensure that the training set contains rich karaoke scene data.

[0181] Use supervised learning methods to train the model and optimize the loss function (such as mean squared error, MSE) to minimize the gap between the predicted noise and the true noise; adjust the learning rate, batch size, and regularization parameters to obtain the best model performance.

[0182] Use the validation set to evaluate the model's noise reduction effect and speech quality, focusing on indicators such as signal-to-noise ratio (SNR), speech clarity, and sound quality; based on the validation results, adjust the network architecture or training parameters to improve model performance.

[0183] Use unseen test data to evaluate the actual performance of the model. The test data should include audio from various karaoke scenarios to verify the generalization ability of the model.

[0184] The test results evaluate the overall performance of the model. The neural network model that passes the test can handle various noise and sound quality issues in practical applications.

[0185] Integrate the trained model into karaoke applications or other speech processing systems. Ensure that the integration process can process high-sampling-rate audio data and perform real-time processing. Also, optimize performance in actual applications, such as accelerating the inference process and reducing computing resource consumption.

[0186] In an embodiment of the present application, the singing audio to be processed in the input karaoke scene is subjected to noise reduction by a sub-band neural network model, and an enhanced audio signal, i.e., the third singing audio, is output to ensure clear sound quality and good noise suppression effect.

[0187] In order to further improve the sound quality, the output audio signal can be further adjusted as needed, such as pitch correction, dynamic range adjustment, etc., and / or possible audio boundary problems can be processed to ensure the continuity and naturalness of the audio.

[0188] In another possible implementation, the process of achieving sound enhancement by first using an echo cancellation module to eliminate leaked background music and then using a noise reduction module to eliminate sudden noise and continuous noise may include: based on the Speak-X module, processing the background music and environmental noise of the singing audio to be processed to obtain the first singing audio; using a neural network model to extract the human voice in the first singing audio to obtain the second singing audio; using a multi-subband neural network model to eliminate the background noise in the second singing audio to obtain the fourth singing audio; adjusting the pitch information according to the fourth singing audio to obtain the trimmed audio.

[0189] Furthermore, in the process of audio correction, pitch change and speed change are two crucial technical means, which play an indispensable role in improving audio quality, meeting specific music needs and creating unique auditory effects.

[0190] In some embodiments, pitch change and speed change can be achieved in a variety of ways: such as neural network vocoder (two-level neural network vocoder, single-layer GAN (Generative Adversarial Networks, based on generative adversarial network) vocoder, GPT (Generative Pre-trained Transformer, pre-trained language model based on Transformer structure) structural sound generator, Diffusion structural sound generator and other neural network structures) and traditional acoustic technology (Tdpsola algorithm, sola, wpsola). At the same time, due to different service deployments, the computing power that can be provided is also different. Traditional acoustic technologies such as the Tdpsola algorithm use traditional acoustic technology for speed and pitch change processing, with fast processing speed and low operating environment requirements (it can run on a server with only a CPU), which is suitable for the application and deployment of audio correction algorithms in most cases. The two-level neural network vocoder uses a two-layer GAN network structure. The first layer performs speed and pitch change, and the second layer performs high sampling rate audio output. It requires additional GPU (Graphics Processing Unit) resources on the server to run under conditions that meet the real-time design requirements of the algorithm. It has better vocal preservation effects and higher sound quality, and the situations it can cope with can be steadily improved with the expansion of training data.

[0191] In one possible scenario, when the computing resources of the electronic device are large enough to meet the computing requirements of the neural network vocoder, the neural network vocoder is used for speed and pitch changing, otherwise traditional acoustic technology is used for speed and pitch changing; in another scenario, the user specifies the speed and pitch changing method.

[0192] A possible implementation method of the embodiment of the present application is further explained by taking the Tdpsola algorithm as an example.

[0193] Analyze the pitch period of the singing audio to be processed. The pitch period refers to the periodic part of the speech signal, which usually corresponds to a sound cycle. Commonly used methods include autocorrelation function (ACF, AutocorrelationFunction) or periodogram-based techniques;

[0194] By analyzing the periodicity of the signal, the location and length of the pitch period are determined;

[0195] The singing audio to be processed is divided into multiple frames according to the pitch cycle, and these frames overlap on the time axis to facilitate subsequent overlap-addition processing;

[0196] By adjusting the length and overlap area of ​​the frames, the pitch can be changed. For example, increasing the length of the frame will result in a lower pitch, while decreasing the length of the frame will result in an increase in pitch. By adjusting the sampling frequency of the frame, the playback speed of the audio signal can be changed. Time scaling can affect the pitch and the speed of speech. During time scaling or pitch adjustment, the length and overlap ratio of the frame can be adjusted to achieve the target effect.

[0197] The overlap-addition technique is applied on the processed frames to synthesize the final audio signal, which ensures the smooth transition of the audio signal and reduces the artifacts caused by frame processing.

[0198] A possible implementation of the embodiment of the present application is further described by taking the two-level neural network vocoder method as an example:

[0199] Obtaining a secondary neural network vocoder, the secondary neural network vocoder comprising a first GAN network structure and a second GAN network structure;

[0200] According to the singing audio to be processed, the first GAN network structure is used to perform speed and pitch change processing to obtain the initial audio;

[0201] Based on the initial audio, the second GAN network structure is used to increase the sampling rate to obtain high-quality singing audio to be processed.

[0202] The two-level neural network vocoder is an improved hifigan, where:

[0203] Get a large-scale audio dataset to ensure the quality and diversity of the audio. The audio duration should be long enough to cover different voices, music or sound effects. The audio data needs to be processed into the model input format, that is, the original audio signal is converted into a Mel-spectrogram. This can be done through librosa or other audio processing libraries. Furthermore, data preprocessing can also be performed, including normalization, cutting, and denoising operations to ensure data quality and consistency.

[0204] The audio dataset is divided into training set, validation set and test set to ensure that the model has good generalization ability on different data.

[0205] Select HiFi-GAN model configuration: Configure the model’s hyperparameters, such as learning rate, batch size, optimizer, loss function, etc., based on the characteristics of the dataset and computing resources.

[0206] Use GPU / TPU for large-scale training. During the training process, the model will learn how to generate high-quality audio from the Mel-spectrogram. During training, the validation set is regularly used to evaluate the model performance to prevent overfitting. The quality of the model can be monitored through loss curves and audio examples.

[0207] Model optimization and fine-tuning: Adjust the model's hyperparameters or network structure based on the training results to improve sound quality or training speed. Model fine-tuning uses specific audio datasets to further optimize the model's performance in specific scenarios.

[0208] Save the trained HiFi-GAN model in the format required for deployment (such as a PyTorch model file). Export the model to ONNX format for inference deployment on different platforms.

[0209] It can be seen that in the embodiment of the present application, a two-level neural network vocoder including a first GAN network structure and a second GAN network structure is introduced. The first GAN network structure is responsible for performing speed and pitch changing processing on the processed singing audio, and can flexibly adjust the playback speed and pitch of the audio; the second GAN network structure increases the sampling rate of the initial audio, which effectively improves the clarity and quality of the audio, reduces the sound quality loss caused by insufficient sampling rate, and makes the audio processing result more excellent.

[0210] Furthermore, the granularity of the information generated by speed change and pitch change is in note units. Direct use will result in abrupt changes and unnatural transitions. For this reason, a smooth transition module is designed in the embodiment of the present application. After obtaining the basic tuning processing information, it is gradually layered and downgraded to obtain an operation unit with a smaller granularity that can correspond to the reference information (midi). During the splitting process, a geometric curve is used to maintain the audio frequency so that it does not lose its original continuity as it is split step by step. Finally, the processing unit is split to the waveform level, and the time granularity is reduced to below 5ms. At a fine granularity, the connection between different phonemes is natural, and the original human voice tone and sound characteristics are retained. Specifically, a possible implementation method of the embodiment of the present application, based on the reference original song pitch template and pitch information, determines that after adjusting the pitch information, it also includes:

[0211] According to the single word of the adjusted pitch information, the first music score information corresponding to each single word is determined from the midi file of the original song;

[0212] According to each phoneme of the single word whose pitch information is adjusted, second music score information corresponding to each phoneme is determined from the first music score information;

[0213] According to each waveform of the phoneme, determining the third music score information corresponding to each waveform from the second music score information;

[0214] The final adjusted pitch information is determined according to the third music score information corresponding to all waveforms to achieve smooth processing of the adjusted pitch information.

[0215] Reference Figure 4 , Figure 4A flow chart of smoothing processing provided for the embodiment of the present application shows the process of gradually decomposing a vocal signal: from words to phonemes and then to waveform signals, among which, full text ASR (automatic speech recognition); Midi information is general music notation information used to represent music score information; Note is the pitch information extracted from a piece of audio; Pitch constitutes a more detailed unit of note, pitch is continuous, and note is discrete. In this step-by-step decomposition process, smaller units are used step by step to align with a general music score information, so that audio adjustments can be made at a smaller granularity.

[0216] It can be seen that in the embodiment of the present application, according to the single word of adjusting the pitch information, the corresponding first score information is extracted from the MIDI file of the original song, which provides an accurate score reference for the subsequent pitch adjustment; then it is refined to each phoneme level, and the second score information corresponding to each phoneme is determined from the first score information. This detailed processing to the phoneme level ensures the accuracy and continuity of the pitch adjustment; then according to each waveform of the phoneme, the third score information corresponding to each waveform is determined from the second score information, which fully considers the changes in the waveform inside the phoneme, making the pitch adjustment smoother and more natural.

[0217] Based on any of the above embodiments, refer to Figure 5 , Figure 5 A schematic diagram of a specific audio tuning solution is provided for the embodiment of the present application, which includes a pre-speech recognition and singing pitch recognition module, a streaming parallel noise reduction filtering module, and a tuning processing module that combines multi-party information to make tuning decisions and output tuned audio. It is a comprehensive solution with a full-process 32K high sampling rate, and of course it can also be 28K or 22.5K.

[0218] According to the karaoke scenario, the tuning algorithm used in the embodiment of the present application adds a speech recognition module with enhanced boundary capabilities, a speed and pitch change smooth transition module, an adaptive tuning template reconstruction module, and a speed and pitch change module with waveform segments as processing units, compared to related tuning technologies, and readjusts the data flow.

[0219] The audio tuning method includes:

[0220] Speech recognition step: Perform speech recognition on the singing audio to be processed to obtain audio description information.

[0221] Pitch recognition step: perform pitch recognition on the singing audio to be processed to obtain initial pitch information;

[0222] Noise reduction filtering step: performing noise reduction filtering on the singing audio to be processed to obtain the singing audio after noise reduction;

[0223] Related information acquisition step: acquiring the front pitch information and / or the back pitch information corresponding to the singing audio to be processed, that is, the historical and future information;

[0224] The step of determining the correspondence between the audio and the reference: determining the target lyrics in the reference lyrics corresponding to the audio description information according to the audio description information;

[0225] The pitch adjustment step includes: determining the original song pitch template corresponding to the singing audio to be processed according to the target lyrics; filtering the invalid pitch information in the initial pitch information according to the boundary time information corresponding to the phoneme in the audio description information; obtaining the front pitch information corresponding to the front singing audio and the rear pitch information corresponding to the rear singing audio of the singing audio to be processed; adjusting the overall pitch of the filtered pitch information according to the front pitch information and / or the rear pitch information to obtain the pitch information; correcting the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain the reference original song pitch template; determining the adjusted pitch information according to the reference original song pitch template and the pitch information;

[0226] Smooth step: smooth the pitch adjustment information;

[0227] Speed ​​and pitch changing steps: The singing audio after noise reduction will be processed with speed and pitch changing to obtain high-quality singing audio to be processed, and the high-quality singing audio to be processed will be pitch processed based on the pitch adjustment information to obtain the tuned audio.

[0228] It can be seen that in the embodiment of the present application, the accurate pronunciation information and reference information generated by the speech recognition module are used together to generate a new song template that allows the speed change and pitch change range in the adaptive tuning template reconstruction module to form basic speed change and pitch change information; the accurate pitch information produced by the pitch recognition module helps to improve the speed change and pitch change information, and the improved speed change and pitch change information passes through the smoothing module to generate a more reliable small-grained audio operation information sequence for use by the speed change and pitch change module; the noise reduction filter module outputs audio data as the operating material of the speed change and pitch change algorithm in the subsequent process. The speed change and pitch change are processed by the front-end link, and the audio operation information sequence is used to process the audio in parallel, and the audio processing is completed at an effectiveness of no more than rtf 0.03.

[0229] In addition, the tuning technology used in the embodiment of the present application is compared with the tuning effect of the Kugou software by MOS points. Figure 6 As shown, through the intelligent singing correction technology designed in the embodiment of the present application, different audios obtain different processing solutions, and all recorded audios are processed uniformly. The processed audio performs well in rhythm and pitch control, which lowers the usage threshold for users in karaoke scenarios, allowing most novice users to produce singing works that conform to musical aesthetics.

[0230] An audio tuning device provided in an embodiment of the present application is introduced below. The device described below and the method described above can be referred to each other. The device of this embodiment is set in an electronic device. Figure 7 , Figure 7 700 is a structural block diagram of an audio tuning device according to one embodiment of the present application. The audio tuning device 700 includes:

[0231] The audio acquisition module 710 is used to acquire the singing audio to be processed;

[0232] The recognition module 720 is used to perform speech recognition on the processed singing audio to obtain audio description information; and perform pitch recognition on the processed singing audio to obtain initial pitch information;

[0233] A boundary correction module 730 is used to correct the initial pitch information according to the audio description information to obtain the pitch information;

[0234] The dynamic template generation module 740 is used to determine the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and to correct the original song pitch template corresponding to the singing audio to be processed according to the pitch information to obtain a reference original song pitch template;

[0235] The tuning module 750 is used to determine the adjusted pitch information based on the pitch template and pitch information of the reference original song, and to tune the singing audio to be processed based on the adjusted pitch information to obtain the tuned audio.

[0236] In one achievable manner, the identification module 720 is specifically configured to:

[0237] Extracting acoustic features of the singing audio to be processed;

[0238] Decoding the acoustic features to obtain a phoneme sequence according to the acoustic features and the decoding graph, wherein the decoding graph is generated based on a phoneme-level acoustic model, a lyrics-based language model, and a phoneme-lyrics conversion table;

[0239] Map the phoneme sequence to the lyrics text to obtain the audio description information.

[0240] In an achievable manner, the audio description information includes: lyrics information and phoneme boundary time information;

[0241] The boundary correction module 730 is specifically used for:

[0242] Filter invalid pitch information in the initial pitch information according to boundary time information corresponding to the phonemes in the audio description information;

[0243] Obtain the front pitch information corresponding to the front singing audio and the rear pitch information corresponding to the rear singing audio of the singing audio to be processed;

[0244] The filtered pitch information is subjected to overall pitch adjustment according to the front pitch information and / or the rear pitch information.

[0245] In one practicable manner, the audio modification module 750 is specifically configured to:

[0246] Perform noise reduction filtering on the processed singing audio based on the first method and / or the second method to obtain filtered audio;

[0247] Performing pitch adjustment on the filtered audio based on the pitch adjustment information to obtain a pitch-adjusted audio;

[0248] Among them, the first method is to first remove the background music and environmental noise in the audio based on the Speak-X module, and then use the neural network model to extract the human voice; the second method is to eliminate the background noise in the audio based on the multi-subband neural network model.

[0249] In one achievable manner, the dynamic template generation module 740 is specifically used to:

[0250] Determine the pitch mean and the pitch of the individual sound corresponding to the singing audio to be processed based on the pitch information;

[0251] If the difference between the pitch mean and the target pitch mean corresponding to the original song pitch template exceeds the first preset pitch threshold, the pitch of the original song pitch template is adjusted as a whole to obtain a first original song pitch template, and the difference between the pitch mean and the target pitch mean corresponding to the first original song pitch template does not exceed the first preset pitch threshold;

[0252] If the difference between the pitch of the individual sound and the pitch of the target sound in the first original song pitch template exceeds a second preset pitch threshold, a first pitch set is determined, wherein the target sound corresponds to the individual sound, and the first pitch set is a pitch set of the target sound that differs by a preset pitch;

[0253] If there is a pitch in the first pitch set whose difference from the pitch of the single tone is within the second preset pitch threshold, the pitch whose difference from the pitch of the single tone is within the second preset pitch threshold is used as the dynamic adjustment target of the target tone;

[0254] If there is no pitch in the first pitch set whose pitch difference with the single tone is within the second preset pitch threshold, then determine a second pitch set, the second pitch set being a set of in-key tones of the original song; select a dynamic adjustment target that meets the target conditions from the second pitch set, the target conditions including: an in-key tone that is closest to the target tone and whose pitch difference with the single tone is within the second preset pitch threshold;

[0255] After determining the dynamic adjustment targets of all individual sounds, the first original song pitch template is adjusted according to the dynamic adjustment targets of all individual sounds to obtain a reference original song pitch template.

[0256] In an achievable manner, the dynamic template generation module 740 is further used to: process the reference target signal whose time domain variation coefficient in the reference original song pitch template exceeds a preset multiple by inserting white noise, and ensure that the starting positions are aligned, so as to obtain an aligned reference original song pitch template;

[0257] When the target tone in the first original song pitch template corresponding to the individual tone cannot be determined, the pitch of the first original song pitch template is adjusted as a whole to an in-key tone, and the pronunciation start point is adjusted to a preset time position.

[0258] In one achievable manner, the device further includes: a speed change and modulation change module, which is used to:

[0259] Obtaining a secondary neural network vocoder, the secondary neural network vocoder comprising a first GAN network structure and a second GAN network structure;

[0260] According to the singing audio to be processed, the first GAN network structure is used to perform speed and pitch change processing to obtain the initial audio;

[0261] Based on the initial audio, the second GAN network structure is used to increase the sampling rate to obtain high-quality singing audio to be processed.

[0262] In one achievable manner, the method further includes: a smoothing module, configured to:

[0263] According to the single word of the adjusted pitch information, the first music score information corresponding to each single word is determined from the midi file of the original song;

[0264] According to each phoneme of the single word whose pitch information is adjusted, second music score information corresponding to each phoneme is determined from the first music score information;

[0265] According to each waveform of the phoneme, determining the third music score information corresponding to each waveform from the second music score information;

[0266] The final adjusted pitch information is determined according to the third music score information corresponding to all waveforms to achieve smooth processing of the adjusted pitch information.

[0267] An electronic device is provided in an embodiment of the present application, such as Figure 8 As shown, Figure 8 The electronic device 800 shown includes: a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, such as through a bus 802. Optionally, the electronic device 800 may also include a transceiver 804. It should be noted that in actual applications, the transceiver 804 is not limited to one, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the present application.

[0268] The processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 801 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0269] The bus 802 may include a path to transmit information between the above components. The bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 802 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0270] The memory 803 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0271] The memory 803 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 801. The processor 801 is used to execute the application code stored in the memory 803 to implement the content shown in the above method embodiment.

[0272] Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0273] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0274] An embodiment of the present application provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0275] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0276] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio tuning method, characterized in that: include: Get the singing audio to be processed; Performing speech recognition on the singing audio to be processed to obtain audio description information; And performing pitch recognition on the singing audio to be processed to obtain initial pitch information; The audio description information includes: lyrics information and boundary time information corresponding to each phoneme; Performing boundary information correction on the initial pitch information according to the audio description information so as to remove noise or invalid parts in the initial pitch information to obtain the pitch information; Determine the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and modify the original song pitch template corresponding to the singing audio to be processed according to the pitch information, so as to adaptively reconstruct the pitch template and obtain a reference original song pitch template; wherein the reference original song pitch template is used to represent the pitch template adjusted according to the actual pitch of the singer on the basis of retaining the original song pitch characteristics; According to the reference original song pitch template and the pitch information, the adjusted pitch information is determined, and the singing audio to be processed is tuned based on the adjusted pitch information to obtain the tuned audio.

2. The method according to claim 1, characterized in that The step of performing speech recognition on the singing audio to be processed to obtain audio description information includes: extracting acoustic features of the singing audio to be processed; According to the acoustic features and the decoding graph, the acoustic features are decoded to obtain a phoneme sequence, wherein the decoding graph is generated based on a phoneme-level acoustic model, a lyrics-based language model, and a phoneme-lyrics conversion table; The phoneme sequence is mapped to the lyrics text to obtain audio description information.

3. The method according to claim 2, characterized in that The audio description information includes: lyrics information and phoneme boundary time information; The step of performing boundary information correction on the initial pitch information according to the audio description information to obtain the pitch information includes: Filter invalid pitch information in the initial pitch information according to boundary time information corresponding to the phonemes in the audio description information; Obtain the front pitch information corresponding to the front singing audio and the rear pitch information corresponding to the rear singing audio of the singing audio to be processed; The filtered pitch information is subjected to overall pitch adjustment according to the front pitch information and / or the rear pitch information.

4. The method according to claim 1, characterized in that The method of performing pitch adjustment on the singing audio to be processed based on the pitch adjustment information to obtain the pitch-adjusted audio includes: Perform noise reduction filtering on the processed singing audio based on the first method and / or the second method to obtain filtered audio; Performing pitch adjustment on the filtered audio based on the pitch adjustment information to obtain pitch-adjusted audio; Among them, the first method is to first remove the background music and environmental noise in the audio based on the Speak-X module, and then use the neural network model to extract the human voice; the second method is to eliminate the background noise in the audio based on the multi-subband neural network model.

5. The method according to any one of claims 1 to 4, characterized in that: The step of correcting the original song pitch template corresponding to the to-be-processed singing audio according to the pitch information to obtain a reference original song pitch template comprises: Determine the pitch mean and the pitch of the individual sound corresponding to the singing audio to be processed based on the pitch information; If the difference between the pitch mean and the target pitch mean corresponding to the original song pitch template exceeds a first preset pitch threshold, the pitch of the original song pitch template is adjusted as a whole to obtain a first original song pitch template, and the difference between the pitch mean and the target pitch mean corresponding to the first original song pitch template does not exceed the first preset pitch threshold; If the difference between the pitch of the individual sound and the pitch of the target sound in the first original song pitch template exceeds a second preset pitch threshold, a first pitch set is determined, wherein the target sound corresponds to the individual sound, and the first pitch set is a pitch set of the target sound with a preset pitch difference; If there is a pitch in the first pitch set whose difference from the pitch of the individual sound is within a second preset pitch threshold, the pitch whose difference from the pitch of the individual sound is within the second preset pitch threshold is used as the dynamic adjustment target of the target sound; If there is no pitch in the first pitch set whose difference with the pitch of the single tone is within the second preset pitch threshold, then determine a second pitch set, where the second pitch set is a set of in-key tones of the original song; select a dynamic adjustment target that meets the target condition from the second pitch set, where the target condition includes: an in-key tone that is closest to the target tone and whose difference with the pitch of the single tone is within the second preset pitch threshold; After determining the dynamic adjustment targets of all individual sounds, the first original song pitch template is adjusted according to the dynamic adjustment targets of all individual sounds to obtain a reference original song pitch template.

6. The method according to claim 5, characterized in that After determining the dynamic adjustment targets of all individual sounds, adjusting the first original song pitch template according to the dynamic adjustment targets of all individual sounds to obtain the reference original song pitch template, at least one of the following is further included: The reference target signal whose time domain variation coefficient in the original song pitch template exceeds a preset multiple is processed by white noise insertion, and the start position is ensured to be aligned, so as to obtain the aligned reference original song pitch template; When the target tone in the first original song pitch template corresponding to the individual tone cannot be determined, the pitch of the first original song pitch template is adjusted as a whole to an in-key tone, and the pronunciation start point is adjusted to a preset time position.

7. The method according to claim 1, characterized in that After determining the adjustment pitch information according to the reference original song pitch template and the pitch information, the method further includes: A secondary neural network vocoder is obtained, wherein the secondary neural network vocoder includes a first GAN network structure and a second GAN network structure; according to the singing audio to be processed, the first GAN network structure is used to perform speed and pitch change processing to obtain initial audio; according to the initial audio, the second GAN network structure is used to increase the sampling rate to obtain high-quality singing audio to be processed.

8. The method according to claim 7, characterized in that After determining the adjustment pitch information according to the reference original song pitch template and the pitch information, the method further includes: According to the single words of the adjusted pitch information, the first music score information corresponding to each single word is determined from the MIDI file of the original song; according to each phoneme of the single words of the adjusted pitch information, the second music score information corresponding to each phoneme is determined from the first music score information; according to each waveform of the phoneme, the third music score information corresponding to each waveform is determined from the second music score information; according to the third music score information corresponding to all waveforms, the final adjusted pitch information is determined to achieve smooth processing of the adjusted pitch information.

9. An audio tuning device, characterized in that: include: An audio acquisition module is used to acquire the singing audio to be processed; A recognition module, used for performing speech recognition on the singing audio to be processed to obtain audio description information; and performing pitch recognition on the singing audio to be processed to obtain initial pitch information; the audio description information includes: lyrics information and boundary time information corresponding to each phoneme; A boundary correction module, used for performing boundary information correction on the initial pitch information according to the audio description information, so as to remove noise or invalid parts in the initial pitch information and obtain the pitch information; A dynamic template generation module is used to determine the original song pitch template corresponding to the singing audio to be processed according to the audio description information, and to correct the original song pitch template corresponding to the singing audio to be processed according to the pitch information, so as to adaptively reconstruct the pitch template and obtain a reference original song pitch template; wherein the reference original song pitch template is used to represent the pitch template adjusted according to the actual pitch of the singer on the basis of retaining the original song pitch characteristics; The tuning module is used to determine the adjusted pitch information according to the reference original song pitch template and the pitch information, and to tune the singing audio to be processed based on the adjusted pitch information to obtain the tuned audio.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: perform the steps of the method according to any one of claims 1 to 8.

Images