Audio data processing method and apparatus, electronic device, and medium

Abstract

The present disclosure provides an audio data processing method and device, electronic equipment, computer readable storage medium and computer program product, relates to the field of artificial intelligence, and particularly relates to the fields of audio processing and deep learning technology. The implementation scheme is: obtaining to-be-processed audio data; obtaining first pitch data corresponding to the to-be-processed audio data; obtaining second pitch data corresponding to reference audio data; determining a first scale difference value of a first audio frame sequence and a second audio frame sequence based on the first pitch data and the second pitch data, the first audio frame sequence being an audio frame sequence in the first pitch data, and the second frame sequence being an audio frame sequence in the second pitch data; transposing the second pitch data based on the first scale difference value; and correcting the first pitch data based on the transposed second pitch data, so as to convert the corrected first pitch data into audio data, to obtain corrected to-be-processed audio data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to the fields of audio processing and deep learning technology, specifically to an audio data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] In work and daily life, people often encounter situations where they need to record audio. Taking karaoke as an example, users frequently encounter problems such as not being able to reach high notes, not being able to reach low notes, or singing off-key due to limitations in their vocal range or inaccurate pitch judgment, affecting the aesthetics of the song. Therefore, there is an urgent need for a refined karaoke pitch adjustment algorithm to correct users' off-key singing. Summary of the Invention

[0004] This disclosure provides an audio data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of this disclosure, an audio data processing method is provided, comprising: acquiring audio data to be processed; acquiring first pitch data corresponding to the audio data to be processed, wherein the first pitch data includes pitches corresponding to multiple audio frames in the audio data to be processed; acquiring second pitch data corresponding to reference audio data, wherein the reference audio data is audio data corresponding to the audio data to be processed, wherein the second pitch data includes pitches corresponding to multiple audio frames in the reference audio data; determining a first scale difference between a first audio frame sequence and a second audio frame sequence based on the first pitch data and the second pitch data, wherein the first audio frame sequence is a sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second frame sequence is a sequence corresponding to at least a portion of the corresponding audio frames in the second pitch data; adjusting the pitch of each audio frame in the second pitch data based on the first scale difference; correcting the first pitch data based on the adjusted second pitch data to obtain corrected first pitch data; and converting the corrected first pitch data into audio data to obtain corrected audio data to be processed.

[0006] According to another aspect of this disclosure, an audio data processing apparatus is provided, comprising: a first acquisition unit configured to acquire audio data to be processed; a second acquisition unit configured to acquire first pitch data corresponding to the audio data to be processed, wherein the first pitch data includes pitches corresponding to multiple audio frames in the audio data to be processed; a third acquisition unit configured to acquire second pitch data corresponding to reference audio data, wherein the reference audio data is audio data corresponding to the audio data to be processed, wherein the second pitch data includes pitches corresponding to multiple audio frames in the reference audio data; and a determining unit configured to determine a pitch based on the first pitch data and the second pitch data. The system comprises: a first audio frame sequence and a second audio frame sequence, wherein the first audio frame sequence is a sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second frame sequence is a sequence corresponding to at least a portion of the audio frames in the second pitch data; a pitch shifting unit configured to shift the pitch of each audio frame in the second pitch data based on the first pitch difference; a correction unit configured to correct the first pitch data based on the pitch-shifted second pitch data to obtain corrected first pitch data; and a conversion unit configured to convert the corrected first pitch data into audio data to obtain corrected audio data to be processed.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described in this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in this disclosure.

[0010] According to one or more embodiments of this disclosure, the pitch curve of the modified user-sung song can be made to follow the same trend as the original pitch curve, making the pitch of the user-sung song closer to the original pitch; and, before correcting the pitch, the standard original audio is first raised or lowered, avoiding the problem of unnatural audio due to excessive pitch adjustment, thus improving the user experience.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0014] Figure 2 A flowchart of an audio data processing method according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A flowchart illustrating the acquisition of second pitch data according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A flowchart is shown of a training method for a neural network model for acquiring second pitch data according to an embodiment of the present disclosure;

[0017] Figure 5 A flowchart illustrating the correction of first pitch data according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A flowchart illustrating the correction of the pitch of a corresponding word in the first pitch data according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A structural block diagram of an audio data processing apparatus according to an embodiment of the present disclosure is shown; and

[0020] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0023] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0024] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0026] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of audio data processing methods.

[0027] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0028] exist Figure 1In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0029] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to acquire audio data to be processed. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0030] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0031] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0032] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0033] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0034] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0035] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0036] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0037] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0038] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0039] In examples of karaoke singing, users often encounter problems such as not being able to reach high notes or low notes, or singing off-key, due to limitations in their vocal range or inaccurate pitch control. This affects the aesthetic appeal of the song. Furthermore, singing may require certain techniques, such as vibrato, trills, or electronic elements, which can also lead to a lack of beauty in the song if the user hasn't mastered these techniques.

[0040] According to one aspect of this disclosure, an audio data processing method is provided. Figure 2 A flowchart of an audio data processing method 200 according to an embodiment of the present disclosure is shown. The method 200 can be used in... Figure 1 The execution can take place at any of the client devices 101, 102, 103, 104, 105, and 106 in the process; that is, the execution entity of each step of method 200 can be any of the client devices 101, 102, 103, 104, 105, and 106. Figure 1 Any of the client devices 101, 102, 103, 104, 105, and 106. In some embodiments, method 200 may also be executed at server 120. In some embodiments, method 200 may be executed in combination by a client device (e.g., any of the client devices 101, 102, 103, 104, 105, and 106) and a server (e.g., server 120).

[0041] like Figure 2As shown, method 200 may include the following steps: acquiring audio data to be processed (step 210); acquiring first pitch data corresponding to the audio data to be processed, the first pitch data including the pitches corresponding to multiple audio frames in the audio data to be processed (step 220); acquiring second pitch data corresponding to reference audio data, wherein the reference audio data is audio data corresponding to the audio data to be processed, and the second pitch data includes the pitches corresponding to multiple audio frames in the reference audio data (step 230); determining the first audio frame sequence and the second audio frame sequence based on the first pitch data and the second pitch data. A pitch difference is calculated, wherein the first audio frame sequence is the sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second frame sequence is the sequence corresponding to at least a portion of the audio frames in the second pitch data (step 240); based on the first pitch difference, the pitch of each audio frame in the second pitch data is raised or lowered (step 250); the first pitch data is corrected based on the second pitch data after raising or lowering the pitch to obtain the corrected first pitch data (step 260); and the corrected first pitch data is converted into audio data to obtain the corrected audio data to be processed (step 270).

[0042] According to embodiments of this disclosure, the pitch curve of the user's singing can be made consistent with the trend of the original pitch curve, making the pitch of the user's singing closer to the original pitch; and, before correcting the pitch, the standard original audio is first raised or lowered, avoiding the problem of unnatural audio due to excessive pitch adjustment, thus improving the user experience.

[0043] In step 210, the audio data to be processed is obtained.

[0044] In the embodiments of this disclosure, audio mainly refers to music-related audio, such as instrumental music or sung songs. As an example, the audio data to be processed in the embodiments of this disclosure can be a song sung by a user, and the reference audio data can be the original song corresponding to the song sung by the user; or, the audio data to be processed can be instrumental music played by a user, and the reference audio data can be a demonstration piece corresponding to the audio data to be processed.

[0045] Therefore, according to some embodiments, obtaining the audio data to be processed may include: obtaining raw human voice audio data collected by a terminal device; and preprocessing the raw human voice audio data to obtain the audio data to be processed, wherein the preprocessing may include, but is not limited to, noise reduction, reverberation removal, etc.

[0046] In some examples, raw human voice audio data can be acquired from a terminal device. For instance, the terminal device could be an amplifier, speakers, a multimedia control console, a digital mixing console, an audio sampling card, a synthesizer, mid-to-high frequency speakers, and a microphone. The raw human voice audio data is obtained through a voice acquisition system comprised of these terminal devices. In a karaoke scenario, this raw human voice audio data is the human voice audio data (excluding accompaniment audio data) acquired by the terminal device (e.g., a microphone).

[0047] In practice, during audio acquisition, microphones inevitably receive some external noise signals during karaoke, and sound reverberates due to reflections in the environment. Therefore, the audio data collected by a voice acquisition system is usually raw human voice audio data that includes both environmental noise and human voice audio data. To obtain better human voice audio data, preprocessing of the raw human voice audio data is necessary.

[0048] For example, in practice, reducing reverberation primarily reduces unwanted sound wave reflections that interfere with other audio signals. In one example, a user speaks approximately 2 to 3 words per second. Assuming the user utters the two words "hello," and the reverberation time is 3 seconds, after uttering the word "hello," although the intensity gradually decreases, it continues for a period (3 seconds). At the moment the word "hello" is uttered, the intensity of "hello" is still quite high. Therefore, the two words mix together, causing interference between the audio signals. Thus, preprocessing can yield clearer vocal audio data for easier subsequent processing.

[0049] In some examples, the method of this embodiment can be provided to users as application software (APP). After the user installs the APP on their mobile device, they can use the APP to correct or polish the user's singing before publishing the recorded user singing song (e.g., uploading it to a social platform).

[0050] In step 220, the first pitch data corresponding to the audio data to be processed is obtained. The first pitch data includes the pitches corresponding to multiple audio frames in the audio data to be processed.

[0051] Music is composed of mechanical waves of different frequencies. To better align with human hearing, pitch is used to describe the frequency of music. Pitch has a linear relationship with human hearing. In modern music, pitch can be represented using the twelve-tone equal temperament system. The notes corresponding to each pitch can be written as C, C#, D, D#, E, F, F#, G, G#, A, A#, B, where "#" represents a sharp (semitone). A scale is simply the set of notes we just discussed, arranged in a specific order, like steps, with each step higher than the last. A scale refers to the notes in a key system, starting from a certain pitch (the tonic) and arranged in ascending order of pitch. Scales and pitch can be understood as different ways of representing musical notes.

[0052] For example, CDEFGAB represents the seven tonic notes of C major: do re mi fa so la si (notated as 1 to 7 in numbered musical notation). Scientific pitch notation combines these two notations to represent a single note. For instance, A4 is middle la, with a frequency of 440Hz. Raising the pitch by an octave doubles the frequency; for example, A5 has a frequency of 880Hz, exactly twice that of A4.

[0053] During a performance, the pitch curve of the song sung by the singer should be consistent with the pitch curve represented by the corresponding note sequence in the score. Only in this way can the pitch of the song be accurate.

[0054] According to some embodiments, obtaining the first pitch data corresponding to the audio data to be processed includes: dividing the audio data to be processed into frames to obtain a first audio frame set; and determining the pitch corresponding to at least one audio frame in the first audio frame set to obtain the first pitch data.

[0055] In this disclosure, any suitable method can be used to obtain audio frames. For example, the audio data can be pre-emphasized, then framed, and then windowed after framing.

[0056] In some embodiments, after the audio data to be processed is framed, a vocoder, such as the open-source algorithm WORLD, can be used to decompose the audio data into spectral envelope (SP), aperiodic parameters (AP), and pitch data. The WORLD algorithm has low computational complexity and is more suitable for low-performance computing devices; in addition, the WORLD algorithm does not require a large amount of data for training and is directly calculated based on signal processing algorithms.

[0057] Understandably, the first pitch data can also be obtained through other algorithms, such as traditional signal processing algorithms like STRAIGHT and Griffim Lim, or vocoders based on neural networks (MelGAN, LPCNet, etc.).

[0058] In some examples, during the acquisition of the first pitch data, other signals that aid in audio generation and decomposition can also be acquired, so that after the audio data is corrected, the corresponding audio data can be reconstructed based on the corrected audio data and the other signals that aid in audio generation and decomposition.

[0059] In step 230, the second pitch data corresponding to the reference audio data is obtained, wherein the reference audio data is the audio data corresponding to the audio data to be processed, and the second pitch data includes the pitches corresponding to multiple audio frames in the reference audio data.

[0060] In this disclosure, reference audio data is the basis for correcting the audio data to be processed. For example, as mentioned above, when the audio data to be processed is a song sung by a user, the reference audio data can be the corresponding original audio.

[0061] According to some embodiments, the audio data to be processed includes human voice audio data, while the directly obtainable reference audio is typically audio data including both human voice and accompaniment. Therefore, as Figure 3 As shown, obtaining the second pitch data corresponding to the reference audio data (step 230) may include: separating the reference human voice audio data and the accompaniment audio data in the reference audio data to obtain the reference human voice audio data (step 310); dividing the reference human voice audio data into frames to obtain a second audio frame set (step 320); and determining the pitch corresponding to at least one audio frame in the second audio frame set to obtain the second pitch data (step 330).

[0062] In some examples, the Spleeter algorithm can be used to separate the human voice audio data from the accompaniment audio data in the reference audio data to obtain the reference human voice audio data.

[0063] According to some embodiments, voice separation can also be performed on reference audio data based on a trained voice separation model, which can be trained through transfer learning using preset sample data and the Spleeter algorithm. The preset sample data includes: multiple accompaniment audio data, multiple voice audio data, and multiple mixed audio data composed of multiple accompaniment audio data and multiple voice audio data. By performing transfer learning on the Spleeter algorithm based on preset sample data, not only can a voice separation model more suitable for the current scenario be obtained, but the efficiency of model training can also be improved by utilizing existing algorithms.

[0064] In one example, two stemming models (2stems) can be trained using preset sample data and the Spleeter algorithm. This model is mainly designed to separate human voice audio data from accompaniment audio data.

[0065] In some embodiments, similar to obtaining the first pitch data, a vocoder, such as the open-source algorithm WORLD, can be used to decompose the audio data into a spectral envelope (SP), aperiodic parameters (AP), and pitch data. Furthermore, the first pitch data can also be obtained using other algorithms, such as those based on traditional signal processing algorithms like STRAIGHT, Griffim Lim, DIO, Harvest, etc., or vocoders based on neural networks (MelGAN, LPCNet, etc.).

[0066] In some examples, after obtaining the pitch data, it can be smoothed by median filtering to serve as a second pitch data.

[0067] According to some embodiments, obtaining the second pitch data corresponding to the reference audio data may include: inputting the spectrogram corresponding to the reference audio data into a trained neural network model to obtain the second pitch data.

[0068] According to some embodiments, such as Figure 4 As shown, the trained neural network model can be trained based on the following steps: acquiring sample audio data and the corresponding voice labels and pitch labels, wherein the voice labels are used to mark whether the corresponding audio frame in the sample audio data has a voice, and the pitch labels are used to mark the pitch corresponding to the corresponding audio frame in the sample audio data (step 410); determining the spectrogram corresponding to the sample audio data (step 420); inputting the spectrogram into the neural network model to obtain whether each audio frame has a voice and the pitch corresponding to each audio frame (step 430); and adjusting the parameters of the neural network model based on whether each audio frame has a voice, the pitch corresponding to each audio frame, the voice labels, and the pitch labels (step 440).

[0069] In some examples, the user's vocal audio data during karaoke can be captured via microphone, along with the corresponding background music audio data. Superimposing these two data points yields the sample audio data for training the neural network model. Alternatively, audio data including both vocal and background audio can be acquired as sample audio data.

[0070] Furthermore, voice and pitch labels corresponding to the sample audio data can be obtained. For example, the aforementioned voice audio data can be processed using algorithms such as VAD (Voice Activity Detection) to determine whether each audio frame contains a voice. For example, if a voice is present in an audio frame, it can be labeled as 1; otherwise, it can be labeled as 0, serving as the voice label. For example, the aforementioned voice audio data can be processed using algorithms such as DIO and Harvest to extract the pitch, and then smoothed using median filtering, serving as the pitch label.

[0071] Since pitch features are easily observed on a spectrogram, and the spectrogram is easier to fit than the original waveform, in some embodiments, the input to the neural network model can be the spectrogram (also called Fbank feature) corresponding to the original audio data. In the field of audio and speech signal processing, the signal can be converted into the corresponding spectrogram, and the data on the spectrogram can be used as the features of the signal. The horizontal axis of the spectrogram is time, the vertical axis is frequency, and the coordinate point value is the speech data energy. Since it uses a two-dimensional plane to represent three-dimensional information, the magnitude of the energy value is represented by color, with darker colors indicating stronger speech energy at that point.

[0072] Since the input is a spectrogram, the backbone network can be any backbone network from the CV (Computer Vision) field. In some embodiments, in the network prediction part, there are two prediction heads: one for predicting whether there is a human voice (a classification task), and the other for predicting pitch (a regression task), to improve the training accuracy of the neural network model.

[0073] Once the neural network model is trained, only pitch prediction is needed during use. The standard original audio is input into the trained model, and the output of the pitch prediction head is extracted to obtain the pitch of each frame.

[0074] In this embodiment, a neural network model is used to predict the pitch in the reference audio data, eliminating the need for pre-separation of vocals and accompaniment. Furthermore, to improve the accuracy and precision of the prediction, a multi-task learning approach is designed, and a self-labeling technique is employed, significantly reducing labor costs.

[0075] In step 240, a first pitch difference between the first audio frame sequence and the second audio frame sequence is determined based on the first pitch data and the second pitch data, wherein the first audio frame sequence is the sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second frame sequence is the sequence corresponding to at least a portion of the audio frames in the second pitch data; and in step 250, each audio frame in the second pitch data is pitched up or down based on the first pitch difference.

[0076] Since everyone's vocal range is different, directly aligning the user's pitch to the target pitch may result in unnatural audio due to excessive pitch adjustment. Therefore, before correcting the pitch, it is advisable to first adjust the pitch of the standard original audio.

[0077] Before raising or lowering the pitch, the difference between the first pitch data and the second pitch data is determined to determine the magnitude of the pitch change. Therefore, according to some embodiments, determining the first pitch difference between the first audio frame sequence and the second audio frame sequence based on the first pitch data and the second pitch data includes: determining the first average pitch corresponding to the first audio frame sequence and the second average pitch of the second audio frame sequence; and mapping the first average pitch and the second average pitch to a first scale and a second scale, respectively, to determine the difference between the first scale and the second scale, as the first pitch difference between the first audio frame sequence and the second audio frame sequence.

[0078] In this embodiment, the first audio frame sequence and the second audio frame sequence can be all the audio frames in the first pitch data and the second pitch data; or, they can be a portion of the audio frames in the first pitch data and the second pitch data, such as the verse, chorus, or a verse and a chorus in a karaoke scenario, etc., without limitation.

[0079] In some exemplary embodiments, in a user's karaoke scenario, the average pitch of the entire song sung by the user can be determined first and then mapped onto a scale. Next, the average pitch of the entire song in the standard original audio is calculated and mapped onto a scale. The difference between the two scales is calculated, and this difference is used to adjust the pitch of the standard original audio, for example, by lowering it by three keys. This involves subtracting three from the scale of each audio frame of the standard original audio and then remapping it onto a pitch, thereby adjusting the standard original pitch to a new pitch and reducing the pitch difference between the user and the standard original.

[0080] In some embodiments, the conversion relationship between pitch and scale can be determined according to the twelve-tone equal temperament of audio using the following formula: where formula (1) represents converting scale n to pitch a. n Above; Formula (2) represents converting pitch v to scale s(v).

[0081]

[0082]

[0083] It is understandable that the conversion relationship between pitch and scale can also be determined according to other suitable musical patterns or conversion methods, and no restrictions are imposed here.

[0084] In step 260, the first pitch data is corrected based on the second pitch data after pitch shifting to obtain the corrected first pitch data.

[0085] According to some embodiments, the reference human voice audio data and the human voice audio data to be adjusted are obtained based on the same text information. Therefore, as Figure 5 As shown, correcting the first pitch data based on the second pitch data after pitch shifting (step 260) may include: determining the correspondence between the audio frames in the second pitch data after pitch shifting and the first pitch data and the text information respectively (step 510); determining the third average pitch corresponding to each character in the first pitch data and the second pitch data after pitch shifting based on the correspondence respectively (step 520); determining the scale of each character in the first pitch data and the second pitch data after pitch shifting based on the third average pitch respectively (step 530); and correcting the scale of the corresponding character in the first pitch data based on the scale of each character in the second pitch data after pitch shifting (step 540).

[0086] In some examples, the audio frame corresponding to each word can be determined by identifying the articulatory units, such as initials, finals, and phonemes, for each audio frame. For instance, the articulatory units corresponding to each audio frame can be identified based on trained models including Gaussian Mixture Models (GMMs) and Hidden Markov Models (HMMs).

[0087] According to some embodiments, such as Figure 6 As shown, correcting the scale of the corresponding character in the first pitch data based on the scale of each character in the second pitch data after the pitch changes (step 540) includes: determining the second scale difference between the scale of each character in the second pitch data after the pitch changes and the scale of the corresponding character in the first pitch data (step 610); in response to determining that the second scale difference is within a preset range, correcting the scale of the corresponding character in the first pitch data to the scale of the corresponding character in the second pitch data after the pitch changes (step 620); and in response to determining that the second scale difference is not within the preset range, raising or lowering the scale of the corresponding character in the first pitch data by a preset number of scales, wherein the preset number is determined based on the preset range (step 630).

[0088] In some exemplary embodiments, in a user's karaoke scenario, after pitch shifting, the user's pitch is corrected. Specifically, the pitch of each word sung by the user is iterated through, mapped to a scale, and the difference between this pitch and the standard original scale is determined. If the difference is between -m and m (e.g., m = 3), the user's pitch is directly corrected to the standard original scale; if the difference is not between -m and m, the user's pitch is corrected by m scale increments. The value of m can be determined experimentally to prevent excessive correction due to a large value, which could lead to audio distortion and an unnatural singing effect, or insufficient correction due to a small value.

[0089] After adjusting the pitch of each word sung by the user by raising or lowering the pitch accordingly, the resulting data is converted into pitch data, thus obtaining the corrected first pitch data.

[0090] In step 270, the corrected first pitch data is converted into audio data to obtain the corrected audio data to be processed.

[0091] As described above, the first audio frame set can be decomposed using a vocoder to obtain the first pitch data and other audio signals. Therefore, in some embodiments, converting the corrected first pitch data into audio data includes: obtaining corrected audio data to be processed based on the corrected first pitch data and the other audio signals.

[0092] It is understood that the corrected first pitch data can also be converted into audio data by any other suitable means, and no restrictions are imposed here.

[0093] According to some embodiments, before converting the corrected first pitch data into audio data, the method may further include: adding sound effects to the corrected first pitch data, including but not limited to: vibrato, trills, and electronic sounds. By customizing and adding sound techniques such as vibrato, trills, and electronic sounds, the audio can be modified to achieve the desired effect for the song, thus meeting the user's needs.

[0094] According to some embodiments, adding sound effects to the corrected first pitch data includes: obtaining the start time, end time, and adjustment amplitude value of the sound effect; and adjusting the pitch data between the start time and end time in the corrected first pitch data based on the adjustment amplitude value to add the corresponding sound effect.

[0095] In some examples, the start time, end time, and adjustment range of a sound effect input by the user through the user interface can be obtained. For instance, as an application (APP), after the user installs the APP on their mobile device, they can use the APP's interface to add sound effect information before publishing their recorded song.

[0096] According to some embodiments, the sound effect includes a pitch transition. Adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: performing a filtering operation on the pitch data between the start and end times, wherein the size of the filter kernel corresponding to the filtering operation is determined based on the adjustment amplitude value.

[0097] Specifically, the pitch data within the time interval can be smoothed, for example, by performing a one-dimensional filtering operation (such as median filtering) on ​​the pitch data within the time interval. The size of the filter kernel can be the adjustment value, so that the filtered pitch data can replace the original pitch data.

[0098] According to some embodiments, the sound effect includes vibrato. Adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: determining the oscillation amplitude of at least one audio frame between the start and end times using a preset function; and adjusting the pitch corresponding to the at least one audio frame between the start and end times based on the oscillation amplitude and the adjustment amplitude value.

[0099] Specifically, the pitch data within this time interval can be oscillated, for example, by using trigonometric functions (such as cosine or sine functions) to determine the pitch oscillation amplitude of each audio frame within this time interval. For instance, the pitch oscillation amplitude determined by the cosine or sine function can be within the range [0,1]. The pitch oscillation amplitudes corresponding to all audio frames within this time interval, when arranged, can form a portion of the waveform of the cosine or sine function. In some examples, the pitch adjustment amplitude of each audio frame can be determined by calculating (e.g., multiplying) the adjustment amplitude value as an adjustment ratio with the corresponding pitch oscillation amplitude.

[0100] According to some embodiments, the sound effect includes electronic music. Adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: determining a fourth average pitch corresponding to at least one audio frame between the start and end times; determining the pitch difference between the pitch of each audio frame in the at least one audio frame between the start and end times and the fourth average pitch; and adjusting the pitch corresponding to the corresponding audio frame in the at least one audio frame based on the pitch difference and the adjustment amplitude.

[0101] Specifically, the pitch data within this time interval can be compressed. For example, the average pitch of the audio frames within this time interval is determined, and then the pitch difference between the average pitch and the pitch of each audio frame within this time interval is determined. In some examples, the pitch adjustment range for each audio frame can be determined by calculating (e.g., multiplying) the adjustment value as an adjustment ratio with the corresponding pitch difference.

[0102] According to embodiments of this disclosure, audio pitch correction functions can be added to products such as karaoke or rap songs, and customizable sound effects such as vibrato, tremolo, and electronic music can be added to enrich product functions, enhance user entertainment, and improve the core competitiveness of the product.

[0103] According to embodiments of this disclosure, such as Figure 7As shown, an audio data processing apparatus 700 is also provided, comprising: a first acquisition unit 710 configured to acquire audio data to be processed; a second acquisition unit 720 configured to acquire first pitch data corresponding to the audio data to be processed, wherein the first pitch data includes pitches corresponding to multiple audio frames in the audio data to be processed; a third acquisition unit 730 configured to acquire second pitch data corresponding to reference audio data, wherein the reference audio data is audio data corresponding to the audio data to be processed, wherein the second pitch data includes pitches corresponding to multiple audio frames in the reference audio data; and a determination unit 740 configured to determine the pitch based on the first pitch data and the second pitch data. The system defines a first pitch difference between a first audio frame sequence and a second audio frame sequence, wherein the first audio frame sequence is a sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second frame sequence is a sequence corresponding to at least a portion of the audio frames in the second pitch data; a pitch shifting unit 750 is configured to shift the pitch of each audio frame in the second pitch data based on the first pitch difference; a correction unit 760 is configured to correct the first pitch data based on the second pitch data after pitch shifting to obtain corrected first pitch data; and a conversion unit 770 is configured to convert the corrected first pitch data into audio data to obtain corrected audio data to be processed.

[0104] According to some embodiments, the audio data processing apparatus according to the present disclosure may further include a sound effect unit configured to add sound effects to the corrected first pitch data, the sound effects including but not limited to: trills, vibrato, and electronic music.

[0105] Here, the operation of each of the above-mentioned units 710 to 770 of the audio data processing device 700 is similar to the operation of steps 210 to 270 described above, and will not be repeated here.

[0106] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0108] refer to Figure 8The present invention describes a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to electronic device 800. Input unit 806 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 807 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 may include, but is not limited to, a hard disk and an optical disk. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0111] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0117] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0119] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. An audio data processing method, comprising: Acquire the audio data to be processed; Obtain the first pitch data corresponding to the audio data to be processed, wherein the first pitch data includes the pitches corresponding to multiple audio frames in the audio data to be processed; Obtain second pitch data corresponding to reference audio data, wherein the reference audio data is audio data obtained based on the same text information corresponding to the audio data to be processed, and wherein the second pitch data includes the pitches corresponding to multiple audio frames in the reference audio data; A first pitch difference between a first audio frame sequence and a second audio frame sequence is determined based on the first pitch data and the second pitch data, wherein the first audio frame sequence is a sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second audio frame sequence is a sequence corresponding to at least a portion of the corresponding audio frames in the second pitch data. Based on the first pitch difference, the pitch of each audio frame in the second pitch data is raised or lowered; The first pitch data is corrected based on the second pitch data after pitch shifting, to obtain the corrected first pitch data, including: The correspondence between the audio frames in the second pitch data after pitch shifting and the first pitch data and the text information is determined respectively; Based on the correspondence, the third average pitch corresponding to each word in the first pitch data and the second pitch data after pitch shifting is determined respectively; Based on the third average pitch, the scale of each word in the first pitch data and the second pitch data after the rise and fall is determined respectively; and Based on the pitch of each word in the second pitch data after the pitch shifts, the pitch of the corresponding word in the first pitch data is corrected; and The corrected first pitch data is converted into audio data to obtain the corrected audio data to be processed.

2. The method of claim 1, wherein, Determining the first pitch difference between the first audio frame sequence and the second audio frame sequence based on the first pitch data and the second pitch data includes: Determine the first average pitch corresponding to the first audio frame sequence and the second average pitch corresponding to the second audio frame sequence; and The first average pitch and the second average pitch are mapped to a first scale and a second scale, respectively, to determine the difference between the first scale and the second scale, which is used as the first scale difference between the first audio frame sequence and the second audio frame sequence.

3. The method of claim 1, wherein, The audio data to be processed includes: Acquire raw human voice audio data collected by the terminal device; and The raw human voice audio data is preprocessed to obtain the audio data to be processed. The preprocessing includes at least one of the following: noise reduction and reverberation removal.

4. The method of claim 1, wherein, Obtaining the first pitch data corresponding to the audio data to be processed includes: The audio data to be processed is divided into frames to obtain a first audio frame set; and Determine the pitch corresponding to at least one audio frame in the first audio frame set to obtain the first pitch data.

5. The method as described in claim 1 or 4, wherein, The audio data to be processed includes human voice audio data, and obtaining the second pitch data corresponding to the reference audio data includes: The reference human voice audio data and the accompaniment audio data in the reference audio data are separated to obtain the reference human voice audio data; The reference human voice audio data is segmented into frames to obtain a second audio frame set; and Determine the pitch corresponding to at least one audio frame in the second audio frame set to obtain the second pitch data.

6. The method as described in claim 1 or 4, wherein, Obtaining the second pitch data corresponding to the reference audio data includes: inputting the spectrogram corresponding to the reference audio data into a trained neural network model to obtain the second pitch data.

7. The method of claim 6, wherein, The trained neural network model is obtained based on the following steps: Obtain sample audio data and corresponding human voice tags and pitch tags, wherein the human voice tags are used to mark whether there is a human voice in the corresponding audio frame of the sample audio data, and the pitch tags are used to mark the pitch corresponding to the corresponding audio frame of the sample audio data; Determine the spectrogram corresponding to the sample audio data; The spectrogram is input into a neural network model to obtain whether each audio frame contains human voices and the pitch corresponding to each audio frame. as well as Based on whether each audio frame predicted to contain human voice, the pitch corresponding to each audio frame, the human voice label, and the pitch label, the parameters of the neural network model are adjusted.

8. The method of claim 4, wherein, Determining the pitch corresponding to at least one audio frame in the first audio frame set includes: decomposing the first audio frame set using a vocoder to obtain the first pitch data and other audio signals; and Converting the corrected first pitch data into audio data includes: obtaining corrected audio data to be processed based on the corrected first pitch data and the other audio signals.

9. The method of claim 1, wherein, Correcting the pitch of the corresponding word in the first pitch data based on the pitch of each word in the second pitch data after the rise and fall of the tones includes: The difference between the second pitch of each character in the second pitch data after the pitch rises and falls and the second pitch of the corresponding character in the first pitch data is determined respectively. In response to determining that the second pitch difference is within a preset range, the pitch of the corresponding word in the first pitch data is corrected to the pitch of the corresponding word in the second pitch data after the pitch difference is raised and lowered; and In response to determining that the second pitch difference is not within the preset range, the pitch of the corresponding word in the first pitch data is raised or lowered by a preset number of pitches, wherein the preset number is determined based on the preset range.

10. The method of claim 1, wherein, Before converting the corrected first pitch data into audio data, the method further includes: adding sound effects to the corrected first pitch data, wherein the sound effects include at least one of the following: trill, vibrato, electronic music.

11. The method of claim 10, wherein, Adding sound effects to the corrected first pitch data includes: Obtain the start time, end time, and adjustment range value of the sound effect; and Based on the adjustment range value, the pitch data between the start time and the end time in the corrected first pitch data is adjusted to add corresponding sound effects.

12. The method of claim 11, wherein, The sound effect includes a pitch transition, and wherein adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: The pitch data between the start time and the end time is filtered, wherein the size of the filter kernel corresponding to the filtering operation is determined based on the adjustment amplitude value.

13. The method of claim 11, wherein, The sound effect includes vibrato, and wherein adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: The oscillation amplitude of at least one audio frame between the start and end times is determined by a preset function; and Based on the oscillation amplitude and the adjustment amplitude value, the pitch corresponding to at least one audio frame between the start time and the end time is adjusted.

14. The method of claim 11, wherein, The sound effect includes electronic music, and wherein adjusting the pitch data between the start and end times in the corrected first pitch data based on the adjustment amplitude value includes: Determine the fourth average pitch corresponding to at least one audio frame between the start time and the end time; Determine the pitch difference between the pitch of each audio frame in at least one audio frame between the start time and the end time and the fourth average pitch; and Based on the pitch difference and the adjustment range, the pitch of the corresponding audio frame in the at least one audio frame is adjusted.

15. An audio data processing apparatus, comprising: The first acquisition unit is configured to acquire audio data to be processed. The second acquisition unit is configured to acquire first pitch data corresponding to the audio data to be processed, wherein the first pitch data includes the pitches corresponding to multiple audio frames in the audio data to be processed; The third acquisition unit is configured to acquire second pitch data corresponding to reference audio data, wherein the reference audio data is audio data obtained based on the same text information corresponding to the audio data to be processed, and wherein the second pitch data includes the pitches corresponding to multiple audio frames in the reference audio data; The determining unit is configured to determine a first pitch difference between a first audio frame sequence and a second audio frame sequence based on the first pitch data and the second pitch data, wherein the first audio frame sequence is a sequence corresponding to at least a portion of the audio frames in the first pitch data, and the second audio frame sequence is a sequence corresponding to at least a portion of the corresponding audio frames in the second pitch data. The pitch shifting unit is configured to shift the pitch of each audio frame in the second pitch data based on the first pitch difference. The correction unit is configured to correct the first pitch data based on the second pitch data after pitch shifting, to obtain the corrected first pitch data, including: The correspondence between the audio frames in the second pitch data after pitch shifting and the first pitch data and the text information is determined respectively; Based on the correspondence, the third average pitch corresponding to each word in the first pitch data and the second pitch data after pitch shifting is determined respectively; Based on the third average pitch, the scale of each word in the first pitch data and the second pitch data after the rise and fall is determined respectively; and Based on the pitch of each word in the second pitch data after the pitch shifts, the pitch of the corresponding word in the first pitch data is corrected; and The conversion unit is configured to convert the corrected first pitch data into audio data to obtain the corrected audio data to be processed.

16. The apparatus of claim 15, wherein, The determining unit includes: The first determining subunit is configured to determine the first average pitch corresponding to the first audio frame sequence and the second average pitch of the second audio frame sequence; and The second determining subunit is configured to map the first average pitch and the second average pitch to a first scale and a second scale, respectively, so as to determine the difference between the first scale and the second scale as the first scale difference between the first audio frame sequence and the second audio frame sequence.

17. The apparatus of claim 15, wherein, The third acquisition unit includes: The first acquisition subunit is configured to input the spectrogram corresponding to the reference audio data into a trained neural network model to obtain the second pitch data.

18. The apparatus of claim 17, wherein, The trained neural network model is obtained based on the following steps: Obtain sample audio data and corresponding human voice tags and pitch tags, wherein the human voice tags are used to mark whether there is a human voice in the corresponding audio frame of the sample audio data, and the pitch tags are used to mark the pitch corresponding to the corresponding audio frame of the sample audio data; Determine the spectrogram corresponding to the sample audio data; The spectrogram is input into a neural network model to obtain whether each audio frame contains human voices and the pitch corresponding to each audio frame. as well as Based on whether each audio frame predicted to contain human voice, the pitch corresponding to each audio frame, the human voice label, and the pitch label, the parameters of the neural network model are adjusted.

19. The apparatus of claim 15, wherein, The correction subunit includes: A unit used to determine the difference between the second pitch of each word in the second pitch data after the pitch changes and the second pitch of the corresponding word in the first pitch data. A unit for correcting the pitch of the corresponding word in the first pitch data to the pitch of the corresponding word in the second pitch data after the pitch difference is determined to be within a preset range; and A unit for raising or lowering the pitch of the corresponding word in the first pitch data by a preset number of pitches in response to determining that the second pitch difference is not within the preset range, wherein the preset number is determined based on the preset range.

20. The apparatus of claim 15, further comprising a sound effects unit configured to add sound effects to the modified first pitch data, wherein the sound effects include at least one of the following: trills, vibrato, and electronic sounds.

21. The apparatus of claim 20, wherein, The sound effect unit includes: The second acquisition subunit is configured to acquire the start time, end time, and adjustment amplitude value of the sound effect; and The sound effects subunit is configured to adjust the pitch data between the start time and the end time in the corrected first pitch data based on the adjustment amplitude value, so as to add corresponding sound effects.

22. The apparatus of claim 21, wherein, The sound effect includes a tone transition, and wherein the sound effect subunit includes: A unit for filtering pitch data between the start and end times, wherein the size of the filter kernel corresponding to the filtering operation is determined based on the adjustment amplitude value.

23. The apparatus of claim 21, wherein, The sound effect includes vibrato, and wherein the sound effect subunit includes: A unit for determining the oscillation amplitude of at least one audio frame between the start and end times using a preset function; and A unit for adjusting the pitch of at least one audio frame between the start and end times based on the oscillation amplitude and the adjustment amplitude value.

24. The apparatus of claim 21, wherein, The sound effects include electronic music, and wherein the sound effect subunit includes: A unit for determining the fourth average pitch corresponding to at least one audio frame between the start time and the end time; A unit for determining the pitch difference between the pitch of each audio frame in at least one audio frame between the start time and the end time and the fourth average pitch; and A unit for adjusting the pitch of a corresponding audio frame in at least one audio frame based on the pitch difference and the adjustment range.

25. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

27. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-14.

Images