Audio processing method, electronic equipment and readable storage medium

Abstract

The invention relates to the field of voice recognition, in particular to an audio processing method, electronic equipment and a readable storage medium. The audio processing method comprises the steps of processing audio data input by a user in a reading mode to obtain user characteristic parameters such as speech speed, pitch and timbre, and obtaining music characteristic parameters, such as music emotion, music style, music rhythm and music tonality, corresponding to the audio data by utilizing a pre-training algorithm based on a plurality of user characteristic parameters. And then, inputting the music characteristic parameters and an audio text corresponding to the audio data into an AI composition model for composing to obtain music data corresponding to the audio data. And finally, synthesizing user characteristic parameters such as timbre, pitch and the like with music data obtained based on the AI composition model by using a speech synthesis technology to obtain a music work singing by simulating the timbre of the user. Therefore, the music sung by the simulated user can be obtained based on the user audio, and a personalized music creation way is provided for the user.

Description

Technical Field

[0001] The present application relates to the field of speech recognition, and in particular to an audio processing method, an electronic device and a readable storage medium. Background Art

[0002] Artificial intelligence (AI) has functions such as speech recognition, image recognition, natural language processing, and music creation. For example, AI's music creation function can include music recognition, algorithmic composition, and automatic arrangement.

[0003] Specifically, in some technical solutions, users can input a fixed melody into an application with a music model, and then the application can convert the timbre of the fixed melody into a specific instrument timbre, such as selecting a violin timbre, thereby obtaining a musical work played by a violin. In other technical solutions, users can also input a melody and text into an application with a music model to obtain a song synthesized from the melody and text. It can be seen that these music creation functions can only convert the melody input by the user into a specific instrument timbre, or perform a simple preset combination of the melody and text input by the user to obtain a song, and cannot provide users with personalized creation. Summary of the invention

[0004] In order to solve the above problems, the embodiments of the present application provide an audio processing method, an electronic device and a readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides an audio processing method, which is applied to an electronic device, and the method includes: acquiring audio data including user voice; determining user characteristic parameters of the audio data; based on the user characteristic parameters of the audio data, determining music characteristic parameters corresponding to the audio data; and obtaining music data corresponding to the audio data based on the music characteristic parameters.

[0006] It can be understood that the user characteristic parameters can be the timbre, pitch, speaking speed, volume, etc. mentioned in the embodiments of the present application, and the music characteristic parameters can be the music style, music emotion, music rhythm and music tonality, etc. mentioned in the embodiments of the present application.

[0007] Based on the above scheme, the music data recorded or uploaded by the user can be used to obtain the music emotion, music style, music rhythm, music tonality and other arrangement features, that is, music feature parameters, corresponding to the audio data, and then combined with the user's voice characteristics (such as timbre, etc.), to achieve personalized music creation to meet the user's personalized creation. In addition, the process of users manually setting various parameters for generating songs, such as song emotion, rhythm, style, instruments, chords, and lyrics, can be omitted, providing convenience for users to create personalized music.

[0008] In a possible implementation of the first aspect, obtaining audio data including user voice includes: obtaining audio data including user voice obtained by the user through reading aloud or humming.

[0009] In a possible implementation of the first aspect above, user characteristic parameters of audio data are determined, including: framing, windowing and Mel-frequency cepstral coefficient feature extraction of audio data including user speech to obtain user characteristic parameters corresponding to the audio data, the user characteristic parameters including the user's timbre, pitch, volume or speaking speed.

[0010] In a possible implementation of the first aspect above, the music feature parameters include music style; based on the user feature parameters of the audio data, the music feature parameters corresponding to the audio data are determined, including: performing semantic analysis on the audio text corresponding to the audio data through a pre-trained semantic recognition algorithm to determine the music style corresponding to the audio data.

[0011] In a possible implementation of the first aspect above, the music feature parameters include music emotions; based on the user feature parameters of the audio data, the music feature parameters corresponding to the audio data are determined, and it also includes: classifying the audio data through a pre-trained emotion recognition algorithm to determine the music emotion corresponding to the audio data; the pre-trained emotion recognition algorithm is a sentiment analysis method based on a sentiment dictionary, by extracting sentiment words from the audio text corresponding to the audio data, and then calculating the total weight of the audio text based on the weights of the sentiment tendencies corresponding to the extracted sentiment words, and determining the music emotion corresponding to the audio data through the total weight of the audio text.

[0012] In a possible implementation of the first aspect above, the music feature parameters include music rhythm; based on the user feature parameters of the audio data, the music feature parameters corresponding to the audio data are determined, and it also includes: inputting the speaking speed and music style into a first mapping function, the first mapping function is a mapping function of music rhythm, music style and speaking speed, and the music rhythm is determined by the first mapping function.

[0013] In a possible implementation of the first aspect above, music feature parameters corresponding to the audio data are determined based on user feature parameters of the audio data, and the music feature parameters also include: the music feature parameters include music tonality; the pitch and music style are input into a second mapping function, and the second mapping function is a mapping function of music tonality, music style and pitch, and the music tonality is determined by the second mapping function.

[0014] In a possible implementation of the first aspect above, music data corresponding to audio data is obtained based on music feature parameters, including: inputting music style, music emotion, music rhythm, music tonality and audio text corresponding to the audio data into a first composition model, and the first composition model obtains first music data corresponding to the audio data based on the music style, music emotion, music rhythm, music tonality and audio text corresponding to the audio data.

[0015] It can be understood that the first composition model can be the AI ​​composition model mentioned in the embodiment of the present application, and the first music data can be the arranged music obtained by the AI ​​composition model mentioned in the embodiment of the present application.

[0016] In a possible implementation of the first aspect above, obtaining music data corresponding to audio data based on music feature parameters also includes: synthesizing the first music data and user feature parameters through speech synthesis technology to obtain second music data with user voice features.

[0017] It can be understood that the second music data can be the music data that simulates the user's timbre, pitch, etc. mentioned in the embodiments of the present application.

[0018] In a possible implementation of the first aspect above, the method also includes: acquiring music data including user voice; determining user feature parameters corresponding to the music data; based on the music data, determining audio feature parameters corresponding to the music data; and obtaining audio data corresponding to the music data based on the audio feature parameters.

[0019] Based on the above scheme, it is possible to obtain audio data (reading audio) corresponding to the music data based on the music data recorded or uploaded by the user, and the audio data is audio that simulates the user's tone of reading.

[0020] That is, based on the audio processing method provided in the embodiment of the present application, it is possible to obtain music data corresponding to the audio data based on the audio data recorded or uploaded by the user, and the music data is music sung by simulating the user's timbre. It is also possible to achieve two-way conversion between song creation and user recitation to meet the user's personalized creation.

[0021] In a possible implementation of the first aspect above, based on the music data, audio feature parameters corresponding to the music data are determined, including: the audio feature parameters corresponding to the music data include audio emotions, an audio text corresponding to the music data is obtained through a pre-trained speech processing algorithm, and based on the audio text, the audio emotion corresponding to the music data is determined through a pre-trained emotion recognition algorithm; audio data corresponding to the music data is obtained based on the audio feature parameters, including: the audio text, audio emotion and user feature parameters are synthesized through speech synthesis technology to obtain audio data with user voice characteristics, and the audio data is audio simulating the user reading the audio text.

[0022] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing instructions; a processor for executing instructions so that the electronic device implements the audio processing method provided by the above-mentioned first aspect and any possible implementation of the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the audio processing method provided by the above-mentioned first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 According to some embodiments provided by the present application, a schematic diagram of an interface of an online program is shown;

[0025] Figure 2 According to some embodiments provided by the present application, a schematic diagram of a framework of an audio processing system is shown;

[0026] Figure 3 According to some embodiments provided by the present application, a schematic diagram of a sentiment analysis method based on a sentiment dictionary is shown;

[0027] Figure 4 According to some embodiments provided by the present application, a flowchart of an audio processing method is shown;

[0028] Figure 5 According to some embodiments provided by the present application, a brief schematic diagram of determining music feature parameters based on audio data is shown;

[0029] Figure 6 According to some embodiments provided by the present application, a flowchart of another audio processing method is shown;

[0030] Figure 7 According to some embodiments provided by the present application, a schematic diagram of a brief design process of an audio processing method is shown;

[0031] Figure 8 According to some embodiments provided in the present application, a schematic diagram of a process of pre-training an emotion recognition model is shown;

[0032] Fig. 9 According to some embodiments provided by the present application, a schematic diagram of a process of pre-training a semantic analysis model is shown;

[0033] Fig.10 According to some embodiments provided in the present application, a schematic diagram of the pre-training process of an AI speech synthesis model is shown. DETAILED DESCRIPTION

[0034] The illustrative embodiments of the present application include, but are not limited to, an audio processing method, an electronic device, and a readable storage medium.

[0035] It can be understood that the audio processing method provided in the embodiments of the present application can be applied to any electronic device, including but not limited to mobile phones, tablet computers, laptops, desktop computers, smart wearable devices, and any other electronic device that supports recording function, without limitation here.

[0036] The technical solution of the present application is described below in conjunction with the accompanying drawings.

[0037] As mentioned above, in some technical solutions, users can input a fixed melody into an application with a music model, and then the application can convert the timbre of the fixed melody into the timbre of a specific instrument. For example, a ToneTransfer program can be used to convert an audio file into music. Among them, Tone Transfer is an online program developed by Google that can convert the timbre of any audio file into the timbre of four instruments: flute, saxophone, trumpet and violin. For example, Figure 1 A schematic diagram of the interface of Tone Transfer is shown. In the interface 200 of Tone Transfer, the user can upload the audio file to be processed by clicking the control "Record or Upload", or hum with a microphone and upload it, that is, input the melody of the audio to be processed to Tone Transfer; further, in the interface 201 of Tone Transfer, the user can select the timbre of the required instrument by clicking the instrument timbre selection control: "Flute", "Saxophone", "Trumpet", "Violin". Then Tone Transfer can convert the above audio file or recording file into music of any one of the above four timbres and play it.

[0038] However, Tone Transfer transfers the pitch and amplitude information in the audio file at a 1:1 ratio. If the input audio sample is pronounced in a way that the four preset instrument timbres cannot express, the Tone Transfer algorithm will guess its pronunciation and play unnatural music results, such as playing the higher frequency sound in the audio sample with a violin, which is very harsh. In addition, Tone Transfer can only simply replace the timbre of the audio with the timbre of the four preset instruments mentioned above, and the melody of the input song is fixed and does not involve arrangement.

[0039] As mentioned above, in other technical solutions, users can also input melody and text into an application with a music model to obtain a song synthesized from the melody and text. For example, an audio file can be converted into music through an audio generation model (MusicLM). MusicLM has been trained on a 280,000-hour music data set and can generate corresponding music based on the input text description; users can enter text and melody in MusicLM (in the form of humming, singing, whistling or playing an instrument) to obtain music that follows the melody and text description.

[0040] However, the music generated by MusicLM based on the input text description and melody does not have the user's own timbre characteristics, and the generated music is the melody provided by the user, and cannot provide users with personalized music works. If the user wants to create personalized works, he needs to manually adjust the various parameters used to generate songs, such as music emotion, music rhythm, music style, etc., and then generate songs according to the parameter information set by the user. It is difficult for users who do not have professional music knowledge to manually set song parameters.

[0041] In order to solve the above problems, an audio processing method is provided in an embodiment of the present application. In this method, user characteristic parameters (i.e., audio characteristic parameters) such as speech speed, pitch, and timbre are obtained by processing the audio data input by the user in the form of reading aloud, etc., and based on multiple user characteristic parameters, a pre-trained algorithm is used to obtain music characteristic parameters corresponding to the audio data, such as music emotion, music style, music rhythm, music tonality, etc. Then, the audio text corresponding to the above music characteristic parameters and the audio data is input into the AI ​​composition model for arrangement to obtain music data (i.e., music melody) corresponding to the audio data. Finally, the user characteristic parameters such as timbre and pitch are synthesized with the music data obtained based on the AI ​​composition model using speech synthesis technology to obtain a music work simulating the user's timbre singing.

[0042] Based on the above scheme, the music data recorded or uploaded by the user can be used to obtain the music emotion, music style, music rhythm, music tonality and other arrangement features corresponding to the audio data, that is, the music feature parameters, and then combined with the user's voice characteristics (such as timbre, etc.) to achieve personalized music creation; users do not need to manually set various parameters used to generate songs, such as song emotion, rhythm, style, instruments, chords, lyrics, etc.

[0043] Specifically, in some embodiments, the user records audio data by reading aloud, humming, etc., and then processes the audio data by methods such as framing, windowing, and Mel frequency cepstrum coefficient (MFCC) feature extraction to obtain user characteristic parameters such as speech rate, volume, timbre, and pitch. Based on the user characteristic parameters, the music characteristic parameters corresponding to the audio data can be obtained. First, the audio data is classified by a pre-trained semantic recognition algorithm, and the music style parameters corresponding to the audio data are determined based on the semantics of the audio text corresponding to the audio data. For example, the music style parameter is determined to be M1 based on the semantic recognition algorithm, and the music style corresponding to the music style parameter M1 is Chinese style music. Then, the audio data is classified by a pre-trained emotion recognition algorithm to determine the music emotion parameters corresponding to the audio data. For example, the music emotion parameter is determined to be N1 based on the emotion recognition algorithm, and the music emotion corresponding to N1 is pleasure. Further, based on the mapping function of speech rate, music style, and music rhythm, the music rhythm corresponding to the audio data is determined. For example, the music rhythm can be 60 beats per minute. Then, based on the mapping function of pitch, music style and music tonality, the music tonality corresponding to the audio data is determined. For example, the music tonality is F major. Finally, the above-mentioned music feature parameters and the audio text corresponding to the audio data are input into the AI ​​composition model, and the music corresponding to the audio data is arranged. Then, the user feature parameters (such as timbre and pitch) are synthesized with the music obtained based on the AI ​​composition model using speech synthesis technology to obtain a music work simulating the user's singing.

[0044] For example, Figure 2 A schematic diagram of the framework of an audio processing system is shown. Figure 2 , users can record audio data through music applications or recording applications, and then input the audio data recorded by users into the recording analysis module for processing to obtain user feature parameters. The recording analysis module can include audio emotion analysis, audio speech rate analysis, audio semantic analysis and pitch analysis. Among them, music emotion analysis can map music emotion; audio speech rate analysis can map music rhythm; audio semantic analysis can map music genre (style); pitch analysis can map music tonality.

[0045] First, the recording analysis module needs to perform framing, windowing, and MFCC feature extraction on the audio data to obtain user feature parameters in the audio data, including timbre, pitch, volume, speaking speed, etc. Among them, framing is to divide the audio data into multiple frames of audio data according to 10ms to 50ms per frame. Windowing is to use a window function to add a rectangular window to multiple frames of audio data, and truncate multiple frames of audio data in the time domain through the rectangular window, so that each frame of audio data can meet the periodicity requirements of Fourier transform processing in the time domain. MFCC feature extraction refers to the fast Fourier transform of the audio data after framing and windowing to convert it from a time domain signal to a frequency domain signal, and then obtain the Mel spectrogram based on the Mel filter group. The horizontal axis of the Mel spectrogram is time and the vertical axis is frequency. Based on the Mel spectrogram, the envelope and details can be obtained by performing logarithmic operation on the spectrum of the audio data. The envelope represents the timbre and the details represent the pitch. Furthermore, the spectrum after the logarithmic operation is subjected to an inverse Fourier transform (cepstrum) to obtain a cepstrum, through which the timbre corresponding to the audio data can be directly obtained.

[0046] Audio sentiment analysis can use sentiment analysis methods based on sentiment dictionaries to determine the music sentiment corresponding to the user audio. Sentiment analysis methods based on sentiment dictionaries refer to determining the music sentiment corresponding to the audio data by extracting sentiment words from the audio text corresponding to the audio data based on a constructed sentiment dictionary. For example Figure 3 , Figure 3 A schematic diagram of a sentiment analysis method based on a sentiment dictionary is shown. Figure 3 First, the sentences of the audio text corresponding to the audio data are preprocessed, that is, the audio text is segmented and the sentiment words are extracted, and then the sentiment words are matched with the words in the sentiment dictionary, where the sentiment dictionary is a pre-trained sentiment word library, and each word in the sentiment dictionary has a corresponding weight of sentiment tendency. For example, the weight of positive sentiment words is "1", and the weight of negative sentiment words is "-1", etc. Finally, according to the preset judgment rules, that is, according to the positive and negative total weight corresponding to the audio text, the music emotion corresponding to the audio data is judged.

[0047] It can be understood that in the sentiment analysis method based on the sentiment dictionary, the sentiment dictionary includes a positive sentiment dictionary, a negative sentiment dictionary, a negative sentiment dictionary, etc. The sentiment dictionary includes positive sentiment words, negative sentiment words, negative sentiment words, etc. For example, negative sentiment words may include "no", positive sentiment words may include "like", "love", and negative sentiment words may include "hate", "hate", etc., which are not limited here.

[0048] It is understood that in other embodiments, the musical emotion corresponding to the audio data can also be determined by using a sentiment analysis method based on machine learning. The sentiment analysis method based on machine learning refers to selecting sentiment words in the audio data as feature words, and using a logistic regression analysis model, a naive Bayes model, a support vector machine (SVM) and other methods to classify the audio text to determine the musical emotion after the text is matrixed, which is not limited here.

[0049] Audio semantic analysis can use a pre-trained semantic recognition algorithm to analyze the semantics of the audio text corresponding to the audio data, thereby determining the music style corresponding to the audio data. It can be understood that the semantic recognition algorithm can be used to enable the computer to understand the meaning of human language, and classify the semantics of the audio text, and determine the arrangement school corresponding to the semantics of the audio text, that is, the music style, which is the music style.

[0050] It can be understood that the audio text (ie, lyrics) corresponding to the audio data can be directly obtained based on audio semantic analysis.

[0051] It is understandable that audio semantic analysis can also be combined with the top song styles that users often listen to in music applications, which is not limited here.

[0052] It can be understood that the music genre may include traditional Chinese music, pop music, hip-hop music, jazz music, etc., and is not limited here.

[0053] Audio speech rate analysis and pitch analysis can be obtained based on the above-mentioned framing, windowing, and MFCC feature extraction.

[0054] Since the speech rate and music style are known, the music rhythm corresponding to the audio data can be determined by the mapping function of music rhythm, music style and speech rate. In some embodiments, it is assumed that the mapping function of music rhythm is y1=Ax1+Bx2+C, y1 represents the music rhythm, x1 represents the speech rate, x2 represents the music style, and A, B, and C are preset mapping coefficients. Then, the music rhythm can be obtained based on the mapping function of music rhythm when the speech rate and music style are known. For example, the music rhythm obtained based on the mapping function of music rhythm is y1=60, that is, the music rhythm is 60 beats per minute.

[0055] It is understandable that in other embodiments, the music rhythm can also be determined based on the recommendation algorithm in combination with the music rhythm of songs that the user liked in the past and the music rhythm of songs that the user has created, which is not limited here.

[0056] Since the pitch and music style are known, the music tonality corresponding to the audio data can be determined by the mapping function of music tonality, music style and pitch. In some embodiments, it is assumed that the mapping function of the music rhythm is y2=Ex3+Fy1+G, y2 represents the music tonality, x3 represents the pitch, y1 represents the music style, and E, F, and G are preset mapping coefficients. Then, the music tonality can be obtained based on the mapping function of the music rhythm when the pitch and music style are known. For example, the music tonality obtained based on the mapping function of the music tonality is F major.

[0057] The audio detail features are the user feature parameters corresponding to the audio data, namely, pitch, timbre, phoneme, loudness, mel spectrum, etc. Among them, phonemes can be divided into vowels and consonants. For example, the Chinese syllable (ā) has only one phoneme, the syllable (ài) has two phonemes, and the syllable (dài) has three phonemes. Loudness indicates the volume.

[0058] After analyzing the audio data based on the above-mentioned recording analysis module, the music feature parameters corresponding to the audio data can be obtained. Furthermore, the music emotion, music rhythm, music genre (style), music tonality, lyrics (audio text) can be input into the AI ​​composition model to obtain the music (i.e., song) corresponding to the audio data, wherein the AI ​​composition model refers to an algorithm that can use music feature parameters such as lyrics, music genre, music tonality, and music rhythm as input, and output songs corresponding to the music feature parameters. For example, the AI ​​composition model can be an AI Singer model, which is not limited here. Finally, the user feature parameters (such as timbre, pitch) corresponding to the audio data are synthesized with the music obtained based on the AI ​​composition model using speech synthesis technology, so as to synthesize the user's voice features such as the user's timbre with the song, and obtain a musical work that simulates the user's timbre singing.

[0059] In order to further understand the audio processing method provided in the embodiment of the present application, Figure 4 A flowchart of an audio processing method is shown. Figure 4 The specific process includes:

[0060] S101: Acquire audio data.

[0061] In some embodiments, the user can use a music application or a recording application to record a piece of audio data, which can be recorded by reading aloud, humming, etc. Alternatively, the user can upload pre-recorded audio data that needs to be processed to the music application. Then, the music application can process the audio data recorded or uploaded by the user.

[0062] S102: Perform pre-processing such as framing, windowing, and noise reduction on the audio data to obtain user characteristic parameters such as speech rate, volume, timbre, and pitch corresponding to the audio data.

[0063] In some embodiments, the audio data obtained above is preprocessed, and the preprocessing may include framing, windowing, noise reduction, and MFCC feature extraction. First, the audio data is framed, and framing is to divide the audio data into multiple frames of audio data of fixed size, and the frame length of each frame of audio data is generally 10ms to 30ms. That is, the audio data is divided into multiple frames of audio data according to 10ms to 50ms per frame. Then, each frame of audio data is windowed, and windowing is to add a rectangular window to each frame of audio data using a window function, and each frame of audio data is truncated in the time domain by the rectangular window, so that each frame of audio data can meet the periodicity requirements of Fourier transform processing in the time domain. In addition, the audio data is denoised to filter out steady-state noise and non-steady-state noise in the audio data. Finally, MFCC feature extraction is performed on the audio data. MFCC refers to the fast Fourier transform of the audio data after framing, windowing and noise reduction, so as to convert the audio data from the time domain to the frequency domain, and then obtain the Mel spectrum based on the Mel filter bank, where the horizontal axis of the Mel spectrum is time and the vertical axis is frequency. Based on the Mel spectrum, the spectrum of the audio data is logarithmically operated to obtain the envelope and details, where the envelope represents the timbre and the details represent the pitch. Then, the spectrum after the logarithmic operation is subjected to an inverse Fourier transform (i.e., cepstrum) to obtain the cepstrum, through which the pitch, timbre and loudness (i.e., volume) of the audio data can be directly obtained.

[0064] S103: Classify the audio data using a pre-trained semantic recognition algorithm, and determine the music genre corresponding to the audio data based on the semantics of the audio text corresponding to the audio data.

[0065] In some embodiments, the audio text corresponding to the audio data is subjected to semantic analysis, and audio semantic analysis refers to analyzing the semantics of the audio text corresponding to the audio data using a pre-trained semantic recognition algorithm to determine the music style corresponding to the audio data. The semantic recognition algorithm can be used to enable the computer to understand the meaning of human language, and classify the semantics of the text, and determine the arrangement faction (i.e., music style) corresponding to the text semantics. Therefore, the music style corresponding to the audio data can be determined based on the semantic recognition algorithm. For example, based on the semantic recognition algorithm, the music style parameter corresponding to the audio data is determined to be M1. In the semantic recognition algorithm, the music style corresponding to the music style parameter M1 is Chinese style music, and then the music style corresponding to the audio data can be determined to be Chinese style music. It can be understood that music style refers to music style. Based on audio semantic analysis, the audio text (i.e., lyrics) corresponding to the audio data can be obtained in detail.

[0066] It can be understood that the music genre may include traditional Chinese music, pop music, hip-hop music, jazz music, etc., and is not limited here.

[0067] S104: Classify the audio data using a pre-trained emotion recognition algorithm to determine the music emotion corresponding to the audio data.

[0068] In some embodiments, a sentiment analysis method based on a sentiment dictionary can be used to determine the music sentiment corresponding to the audio data. The sentiment analysis method based on a sentiment dictionary refers to determining the sentiment tendency of the audio text by extracting sentiment words in the audio text corresponding to the audio data based on a constructed sentiment dictionary, thereby determining the music sentiment corresponding to the audio data.

[0069] For example, the above Figure 3 A schematic diagram of a sentiment analysis method based on a sentiment dictionary is shown. First, the sentences in the audio text corresponding to the audio data are preprocessed, that is, the audio data is segmented and sentiment words are extracted, and then the sentiment words are matched with the words in the sentiment dictionary, wherein the sentiment dictionary is a pre-trained sentiment word library, and each word in the sentiment dictionary has a corresponding weight of sentiment tendency. For example, the weight of positive sentiment words is "1", and the weight of negative sentiment words is "-1", etc. Finally, according to the preset judgment rule, that is, according to the positive and negative total weight corresponding to the audio text, the music emotion corresponding to the audio data is determined.

[0070] It can be understood that in the sentiment analysis method based on the sentiment dictionary, the sentiment dictionary includes positive sentiment dictionary, negative sentiment dictionary, negative sentiment dictionary, etc. The sentiment dictionary includes positive sentiment words, such as "like", "love", etc.; negative sentiment words, such as "hate", "hate", etc.; negative sentiment words, such as "no", etc., which are not limited here.

[0071] S105: Determine the music rhythm corresponding to the audio data through a mapping function of the music rhythm, speech speed, and music style.

[0072] In some embodiments, based on the determined speech rate and music style, the music rhythm corresponding to the audio data can be determined by the mapping function of music rhythm, music style and speech rate. Assume that the mapping function of music rhythm is y1=Ax1+Bx2+C, where y1 represents the music rhythm, x1 represents the speech rate, x2 represents the music style, and A, B, and C are preset mapping coefficients. Then, when the speech rate and music style are known, the music rhythm can be obtained based on the mapping function of music rhythm. For example, the music rhythm y1 obtained by the mapping function of music rhythm is 60, that is, the music rhythm is 60 beats per minute.

[0073] It is understandable that in other embodiments, the music rhythm can also be determined based on the recommendation algorithm in combination with the music rhythm of songs that the user liked in the past and the music rhythm of songs that the user has created, which is not limited here.

[0074] S106: Determine the music tonality corresponding to the audio data through a mapping function of tone, pitch, and music style.

[0075] In some embodiments, based on the pitch and music style that have been determined, the music tonality corresponding to the user audio can be determined by the mapping function of music tonality, music style, and pitch. Assume that the mapping function of the music rhythm is y2=Ex3+Fy1+G, where y2 represents the music tonality, x3 represents the pitch, y1 represents the music style, and E, F, and G are preset mapping coefficients. Then, when the pitch and music style are known, the music tonality can be obtained based on the mapping function of the music rhythm. For example, the music tonality obtained by the mapping function of the music tonality is F major.

[0076] S107: Create music based on musical style, musical emotion, musical rhythm, musical tonality and audio text.

[0077] In some embodiments, music emotion, music rhythm, music genre (style), music tonality, and lyrics (audio text) can be input into the AI ​​composition model to obtain music (i.e., song) corresponding to the audio data. For example, if the music genre is Chinese style music, the music emotion is joyful, the music tonality is F major, and the music rhythm is 60 beats per minute, the AI ​​composition model can create music (i.e., song) corresponding to the audio data based on the above music feature parameters.

[0078] It can be understood that the AI ​​composition model refers to an algorithm model that can take music feature parameters such as lyrics, music style, music tonality and music rhythm as input, and output songs corresponding to the music feature parameters. For example, the AI ​​composition model can be an AI Songsinger model, which is not limited here.

[0079] S108: synthesizing the user characteristic parameters corresponding to the audio data with the music created based on the music characteristic parameters through speech synthesis technology to obtain a music work simulating the user's singing.

[0080] In some embodiments, music (i.e., song) corresponding to the audio data is obtained through an AI composition model. In order to obtain a musical work that can simulate the user's singing, speech synthesis technology can be further used to synthesize the user's characteristic parameters (such as timbre and pitch) corresponding to the audio data with the music obtained based on the AI ​​composition model, thereby synthesizing the user's voice characteristics such as the user's timbre and pitch with the music to obtain a musical work that simulates the user's timbre singing.

[0081] It can be understood that the processing process of speech synthesis technology can include a language analysis part and an acoustic system part. Among them, the language analysis part mainly analyzes the audio text to generate a corresponding linguistic specification, that is, how to sing the audio text; the acoustic system part mainly generates corresponding music according to the phonetic specification provided by the speech analysis part, that is, to realize the function of voice generation. Specifically, the language analysis part mainly conducts text structure and language type judgment, text standardization, text-to-phoneme conversion, and intonation and rhythm prediction based on the audio text. Text structure and language type judgment means determining the language type of the audio text and splitting the audio text into individual sentences according to the grammar rules of the corresponding language type. Text standardization means converting the Arabic numerals or letters in the audio text into words according to preset rules to standardize the synthesized text. For example, "8787" is converted into the Chinese characters "八七八七". Text-to-phoneme conversion is to label the text with pinyin and convert the text in the audio text into the corresponding pinyin. For polyphonic characters, the pronunciation and tone are determined through word segmentation and part-of-speech and syntactic analysis. For example, "希望在前方" is "xi1wang4zai4qian2fang1". Intonation and rhythm prediction means performing rhythm prediction on the audio text to determine the places of pauses and the pause durations, the stress and unstressed readings of words and phrases, and to achieve the cadence of the music.

[0082] The acoustic system part then uses a vocoder to synthesize and convert the acoustic features obtained based on the language analysis part with the user's voice characteristics such as timbre and pitch into a waveform, and this waveform is the music that simulates the user's singing.

[0083] Through the above solution, music feature parameters corresponding to the audio data can be obtained based on the audio data recorded or uploaded by the user for music arrangement and creation, and the user's feature parameters such as the user's timbre are synthesized with the created music to obtain the music data corresponding to the audio data, and this music data is the music that simulates the user's timbre for singing. It meets the user's personalized music creation needs, and the user can obtain personalized music works with their own voice characteristics.

[0084] For example, Figure 5A simple schematic diagram of determining music feature parameters based on audio data is shown. Specifically, feature analysis and feature mapping are performed on the audio data (audio file) recorded or uploaded by the user. The music style corresponding to the audio data can be determined by semantic analysis of the audio text corresponding to the user audio, and then feature mapping can be performed to determine the music style. The music emotion corresponding to the audio data can be determined by emotion recognition of the audio data, and then feature mapping can be performed to determine the music emotion. The pitch of the audio data can be determined by pitch recognition of the audio data, and then feature mapping can be performed in combination with the pitch and music style to determine the music tonality. The speech rate can be determined by analyzing the audio data, and then feature mapping can be performed in combination with the speech rate and music style to determine the music rhythm. Feature analysis also includes text conversion of the audio data to obtain the audio text corresponding to the audio data, and then the lyrics can be determined. Based on this, a series of music feature parameters are determined based on the audio data.

[0085] It is understandable that in other embodiments, the user can manually change musical instruments. Different musical instruments have different timbres, and the timbres of musical instruments can also be integrated into the music arrangement to enrich the music arrangement.

[0086] In some other embodiments provided by the present application, audio data corresponding to the music data simulating the user's reading can also be obtained based on the music data sung or uploaded by the user. Figure 6 , Figure 6 A flowchart of another audio processing method is shown. Figure 6 The specific process includes:

[0087] S201: Acquire music data.

[0088] In some embodiments, the user can use a music application or a recording application to sing and record a piece of music data, and can record by humming, etc. Alternatively, the user can upload pre-recorded music data that needs to be processed to the music application. Then, the music application can process the music data.

[0089] S202: Pre-process the music data by framing, adding windows, etc., to obtain user characteristic parameters such as the user's timbre and pitch.

[0090] In some embodiments, the music data sung by the user is framed, windowed, and subjected to MFCC feature extraction. First, the music data is framed. Framing is to divide the music data into multiple frames of music data of fixed size, and the frame length of each frame of music data is generally 10ms to 30ms. Then, each frame of music data is windowed. Windowing is to use a window function to add a rectangular window to each frame of music data, and each frame of music data is truncated in the time domain through the rectangular window, so that each frame of music data can meet the periodicity requirements of Fourier transform processing in the time domain. Further, MFCC feature extraction is performed on the music data to obtain user feature parameters with user voice characteristics such as timbre and pitch.

[0091] S203: Convert the music data into audio text (reading text) through a pre-trained speech processing algorithm.

[0092] In some embodiments, the music data sung by the user can be processed by a pre-trained voice processing algorithm to convert the music data into text that can be read aloud. It can be understood that the processing of the pre-trained voice processing algorithm for converting music data to text is mainly divided into two stages: the signal processing stage and the signal recognition stage. In the signal processing stage, the music data is converted from an analog form to a digital form that can be processed by a computer, and pre-processing such as noise reduction is performed to improve the recognition accuracy of the music data. In the signal recognition stage, the processed music data can be input into the voice recognition model, so that the audio text corresponding to the music data can be directly obtained based on the voice recognition model, that is, the text to be read aloud.

[0093] It can be understood that the speech recognition model can be a model based on a hidden Markov model (HMM), a recurrent neural network model (RNN) based on deep learning, a long short-term memory network model (LSTM), and a transcription attention model, etc., without limitation here.

[0094] S204: Determine the audio emotion corresponding to the music data through a pre-trained emotion recognition algorithm.

[0095] In some embodiments, the audio emotion corresponding to the music data can be determined by a pre-trained emotion recognition algorithm. The pre-trained speech processing algorithm can determine the audio emotion corresponding to the music data based on the keywords (such as emotional words) in the music text corresponding to the music data. For example, if positive words such as "happy" and "hope" appear in large numbers in the music text corresponding to the music data, then "happy" and "hope" can be used as keywords, and based on the emotional tendency of "happy" and "hope" as positive words, the emotion of the read text is determined to be pleasure.

[0096] S205: synthesizing the audio text, audio emotion and user characteristic parameters through speech synthesis technology to obtain audio simulating the user's reading.

[0097] In some embodiments, based on the determined audio text and audio emotion, the user's timbre, pitch and other user characteristic parameters can be synthesized with the audio text and audio emotion through speech synthesis technology. That is, based on the audio emotion, the audio text is read aloud with the user's timbre and other user characteristic parameters to obtain audio that simulates the user's timbre reading.

[0098] Based on the above scheme, the music data corresponding to the audio data can be obtained based on the audio data recorded or uploaded by the user, and the music data is music simulating the user's timbre. It is also possible to obtain the audio data (reading audio) corresponding to the music data based on the music data recorded or uploaded by the user, and the audio data is audio simulating the user's timbre reading. That is, based on the audio processing method provided in the embodiment of the present application, the two-way conversion between song creation and user reading can be realized to meet the user's personalized creation.

[0099] The following is based on Figure 7 The schematic diagram of the brief design process of the audio processing method shown in the figure illustrates the brief design process of the audio processing method provided in the embodiment of the present application.

[0100] First of all, the main idea of ​​the audio processing method provided in the embodiment of the present application is to create the user's audio data into a musical work with the user's voice characteristics corresponding to the audio data. Therefore, it is necessary to be based on certain technical prerequisites. When processing audio, this application mainly uses various pre-trained models, such as emotion recognition (algorithm) models, semantic recognition (algorithm) models, etc., as well as AI onomatopoeia / onomatopoeia skills, AI arrangement technology, etc. And various training sets will be used when training the model, such as music that the user likes, music collections that the user has created in the past, and audio files of different instruments. The following will briefly describe the process of pre-training of each model, which will not be repeated here.

[0101] Secondly, after obtaining the user feature parameters (such as timbre, pitch, etc.), the audio text (lyrics) corresponding to the audio data, and the music feature parameters (such as emotion, style, etc.) corresponding to the audio data through the pre-trained model, these feature parameters are input into the AI ​​composition model for music arrangement and creation. During the arrangement and creation process, the instruments in the arrangement can be changed, and the timbres of different types of instruments can be incorporated into the arrangement, making the final music arrangement richer.

[0102] Finally, speech synthesis technology can be used to synthesize the user's timbre with the music created by the AI ​​composition model to obtain music data that simulates the user's singing.

[0103] The following is a brief description of the pre-training process of the emotion recognition (algorithm) model mentioned in the embodiments of the present application.

[0104] The process of emotion recognition of audio data (i.e., user speech) can be divided into two main parts: acoustic feature extraction and emotion analysis. Acoustic feature extraction refers to extracting emotion-related sound features from audio data (speech signals). Emotion-related sound features may include the fundamental frequency, tone, speech rate, pitch, energy, and phonemes of speech, and then the short-time energy, short-time zero-crossing rate, linear prediction coding, and Mel frequency cepstrum coefficients of the above-mentioned emotion-related sound features are extracted through digital signal processing technology. Emotion analysis refers to analyzing acoustic features using machine learning algorithms to obtain the user's emotional state. It can be understood that emotion analysis usually involves classification, such as classifying emotions into yearning, lyricism, lightness, cheerfulness, pleasure, enthusiasm, vitality, sacredness, and sadness. It can be understood that classification can be trained using supervised learning algorithms, such as support vector machines, random forests, neural networks, and deep learning algorithms, which are not limited here.

[0105] Combine the following Figure 8 The process diagram of the pre-training of the emotion recognition model shown is briefly described. The pre-training process of the emotion recognition model may specifically include: S301, pre-processing the training set, the pre-processing may include denoising the audio data in the training set, etc., to filter out the noise of the audio data in the training set; S302, extracting emotion features from the audio data, that is, by extracting emotion words from the audio data and using the extracted emotion words as emotion features. For example, if "scenery", "very" and "spectacular" are extracted, the sentiment of the sentence corresponding to the above sentiment words will be classified as "positive"; S303, feature dimensionality reduction is performed on the extracted sentiment features. Feature dimensionality reduction refers to reducing the number of sentiment features. By performing feature dimensionality reduction on the sentiment features, the calculation speed of the algorithm for determining the sentiment can be accelerated, and the accuracy of sentiment feature classification can be effectively improved by reducing the noise caused by the rich sentiment features; S304, the above sentiment features are input into the classification model for classification; S305, the sentiment corresponding to the audio data is predicted based on the classification results of the sentiment features by the above classification model; S306, it is determined whether the sentiment prediction results corresponding to the above-obtained audio data are accurate. If accurate, the sentiment of the prediction results is directly used as the sentiment corresponding to the audio data. If inaccurate, the parameters of the classification model are optimized to improve the accuracy of the classification model in sentiment classification based on the sentiment features; S308, the processed training set is retrained based on the optimized classification model to improve the accuracy of the classification results obtained by the classification model based on the sentiment features.

[0106] The following is a brief description of the pre-training process of the semantic analysis (algorithm) model mentioned in the embodiments of the present application.

[0107] To perform semantic analysis on audio data (i.e., user voice), the audio data must be converted into text, and the text must be classified by a text classification model. The semantic analysis model (i.e., text classification model) in the embodiment of the present application is to convert the audio data (i.e., user voice) into text and then determine the music faction (i.e., style) that matches the audio data according to the meaning (semantics) of the text. For example, the text classification model can be TextCNN, TextRCNN, BERT, etc., which are not limited here.

[0108] Combine the following Fig. 9 The pre-training process diagram of the semantic analysis model (i.e., text classification model) shown in FIG. 1 is briefly described. Here, the pre-trained language model BERT (bidirectional encoder representations from transformers) is used as an example for description.

[0109] The pre-training process of the language model BERT can specifically include: first pre-processing the training set. The pre-processing stage can include identifying the speech (audio data) in the training set and converting it into text corresponding to the audio data, judging the language of the audio data to determine the grammatical rules corresponding to the language of the audio text, and dividing the audio text into individual sentences according to the grammatical rules of the corresponding language. Further, the divided sentences are input into the language model (BERT), and the language model (BERT) can obtain the text vector according to the input text sequence. The text vector can be understood as a feature vector that has extracted features and key information from the text, which is an intermediate product of the language model. Then the text vector is used as the input of the classifier; finally, the classifier can be used to determine the style of music corresponding to the text vector (i.e., the classification result).

[0110] It can be understood that the classifier can be a single-layer or multi-layer neural network. If it is necessary to add more genre classification labels (such as pop, Chinese style, classical and other genre labels), the number of neurons can be increased by adjusting the classifier network. There is no limitation here.

[0111] The following is a brief description of the process of pre-training music data with user voice characteristics using an AI composition model and speech synthesis technology mentioned in the embodiments of the present application, that is, using an AI speech synthesis model.

[0112] The process of obtaining music data based on the AI ​​speech synthesis model can specifically include: First, based on the relevant information of the music arrangement (song score) that has been obtained through the AI ​​composition model, it is regarded as a phoneme sequence as the input of the AI ​​speech synthesis model. It can be understood that phonemes refer to the smallest speech unit, which can be divided into vowels and consonants. For example, the Chinese syllable (ā) has only one phoneme, and the syllable (ài) has two phonemes. Then, by analyzing the song score, the pronunciation, duration, pauses, pitch, style, and singing skills of each phoneme in the song score are extracted and predicted from multiple dimensions such as text, melody, and rhythm; finally, the deep neural network acoustic model and the vocoder model are used to synthesize music data (i.e., singing audio) that simulates the user's singing with similar timbre and pitch to the user.

[0113] Combine the following Fig.10 The schematic diagram of the pre-training process of the AI ​​speech synthesis model shown is briefly described.

[0114] like Fig.10 In the example, suppose there is a line of music score "The scenery here is really spectacular", and the music score is analyzed to obtain the music score duration, lyrics text and music score pitch; wherein, the music score duration refers to the relative duration between each note in the music score, and the music score pitch refers to the high and low pitch of each note in the music score. Further, by predicting the singing time of the music score duration, the singing time when sung by the user can be obtained; by predicting the pronunciation and pause of the lyrics text, the text features corresponding to the lyrics text can be obtained, for example, the pronunciation and pause method of the words when singing the text; by predicting the pitch of the music score pitch, the singing pitch when sung by the user can be obtained. Then, the singing time, text features and singing pitch obtained above are jointly input into the singing acoustic feature prediction to determine the pronunciation, duration, pause, pitch, style and singing skills of each phoneme in the lyrics. Finally, the deep neural network acoustic model and the vocoder model are used to synthesize the music data (i.e., singing audio) that simulates the user's timbre, pitch, etc.

[0115] It can be understood that the vocoder model can be obtained by performing end-to-end acoustic modeling based on an autoregressive model, which is not limited here.

[0116] The various embodiments disclosed in the present application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0117] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0118] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0119] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0120] In the accompanying drawings, some structural or method features are shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0121] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0122] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. Although the present application has been illustrated and described with reference to certain preferred embodiments of the present application, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the scope of the present application.

Claims

1. An audio processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquiring audio data including user voice; Determining user characteristic parameters of the audio data; Based on the user characteristic parameters of the audio data, determining music characteristic parameters corresponding to the audio data; Music data corresponding to the audio data is obtained based on the music feature parameters.

2. The method according to claim 1, characterized in that: The obtaining of audio data including user voice includes: The audio data including the user's voice is obtained by the user reading aloud or humming.

3. The method according to claim 1, characterized in that The step of determining the user characteristic parameters of the audio data comprises: The audio data including the user's voice is framed, windowed, and subjected to Mel-frequency cepstral coefficient feature extraction to obtain user feature parameters corresponding to the audio data, wherein the user feature parameters include the user's timbre, pitch, volume, or speech speed.

4. The method according to claim 1, characterized in that: The music characteristic parameters include music style; The determining, based on the user characteristic parameter of the audio data, a music characteristic parameter corresponding to the audio data comprises: The audio text corresponding to the audio data is semantically analyzed by a pre-trained semantic recognition algorithm to determine the music style corresponding to the audio data.

5. The method according to claim 4, characterized in that The music characteristic parameters include music emotion; The determining, based on the user characteristic parameters of the audio data, music characteristic parameters corresponding to the audio data further includes: Classifying the audio data using a pre-trained emotion recognition algorithm to determine the music emotion corresponding to the audio data; The pre-trained emotion recognition algorithm is a emotion analysis method based on an emotion dictionary. It extracts emotion words from the audio text corresponding to the audio data, calculates the total weight of the audio text based on the weights of the emotion tendencies corresponding to the extracted emotion words, and determines the music emotion corresponding to the audio data through the total weight of the audio text.

6. The method according to claim 4, characterized in that The music characteristic parameters include music rhythm; The determining, based on the user characteristic parameters of the audio data, music characteristic parameters corresponding to the audio data further includes: The speech rate and the music style are input into a first mapping function, where the first mapping function is a mapping function of the music rhythm, the music style and the speech rate, and the music rhythm is determined by the first mapping function.

7. The method according to claim 6, characterized in that The music characteristic parameters include music tonality; The determining, based on the user characteristic parameters of the audio data, music characteristic parameters corresponding to the audio data further includes: The pitch and the music style are input into a second mapping function, where the second mapping function is a mapping function of the music tonality, the music style and the pitch, and the music tonality is determined by the second mapping function.

8. The method according to claim 1, characterized in that The obtaining of music data corresponding to the audio data based on the music feature parameters comprises: The music style, music emotion, music rhythm, music tonality and the audio text corresponding to the audio data are input into a first composition model, and the first composition model obtains first music data corresponding to the audio data based on the music style, music emotion, music rhythm, music tonality and the audio text corresponding to the audio data.

9. The method according to claim 3 or 8, characterized in that: The obtaining of music data corresponding to the audio data based on the music feature parameters also includes: The first music data and the user characteristic parameters are synthesized by speech synthesis technology to obtain second music data with the user voice characteristics.

10. The method according to claim 1, characterized in that The method further comprises: Acquire music data including user voice; Determining user characteristic parameters corresponding to the music data; Based on the music data, determining audio feature parameters corresponding to the music data; Audio data corresponding to the music data is obtained based on the audio feature parameters.

11. The method according to claim 10, characterized in that The step of determining, based on the music data, audio feature parameters corresponding to the music data includes: The audio feature parameters corresponding to the music data include audio emotions, an audio text corresponding to the music data is obtained through a pre-trained speech processing algorithm, and based on the audio text, the audio emotion corresponding to the music data is determined through a pre-trained emotion recognition algorithm; The obtaining of audio data corresponding to the music data based on the audio feature parameters comprises: The audio text, the audio emotion and the user characteristic parameters are synthesized by speech synthesis technology to obtain audio data with user voice characteristics, and the audio data is audio simulating the user reading the audio text.

12. An electronic device, characterized in that: include: A memory for storing instructions; A processor, configured to execute the instructions to implement the audio processing method according to any one of claims 1 to 11.

13. A readable storage medium, characterized in that: The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the audio processing method according to any one of claims 1 to 11.

Images