Audio processing method and device

Abstract

The present disclosure relates to an audio processing method and device. The audio processing method includes: based on the tonality information and mode information of the original audio corresponding to the recorded audio, obtaining the pitch of the scale corresponding to the original audio; for each signal frame of the human voice signal in the recorded audio, performing the following operations: based on the pitch of the scale corresponding to the original audio, determining the reference pitch of the current signal frame; based on the reference pitch of the current signal frame, adjusting the pitch corresponding to the current signal frame to obtain the target pitch of the current signal frame; based on the target pitch, processing the current signal frame.

Description

Technical Field

[0001] The present disclosure relates to the field of audio and video processing, and in particular to an audio processing method and device. Background Art

[0002] Pitch correction mainly refers to correcting the pitch of the human voice in the collected audio, so that the pitch of the processed human voice is more accurate than that of the human voice before processing, and it can also cover up some singing defects such as broken voice, unstable breath, and off-tune. Usually, in order to obtain accurate human voice pitch, mixers can manually correct the human voice pitch through some audio plug-ins such as wave-tune and Melodyne to achieve the above purpose. However, manual adjustment is time-consuming and laborious, and it is impossible to batch process human voices. Therefore, automatic pitch correction has become a problem of concern to audio workers.

[0003] Currently, automatic tuning strongly relies on MIDI reference information. However, not every recorded audio has corresponding MIDI reference information. For example, the adapted song does not have corresponding MIDI reference information. In this case, if the MIDI reference information of the original song is used, it is difficult to obtain accurate tuning results. Summary of the invention

[0004] The present disclosure provides an audio processing method and device to at least solve the problem of inaccurate audio correction in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an audio processing method, comprising: obtaining the pitch of a musical scale corresponding to the original audio based on the tonality information and mode information of the original audio corresponding to the recorded audio; performing the following operations for each signal frame of a vocal signal in the recorded audio: determining a reference pitch of a current signal frame based on the pitch of the musical scale corresponding to the original audio; adjusting the pitch corresponding to the current signal frame based on the reference pitch of the current signal frame to obtain a target pitch of the current signal frame; and processing the current signal frame based on the target pitch.

[0006] Optionally, based on the pitch of the scale corresponding to the original audio, determining the reference pitch of the current signal frame includes: taking the pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame as the reference pitch of the current signal frame.

[0007] Optionally, based on the reference pitch of the current signal frame, the pitch corresponding to the current signal frame is adjusted to obtain the target pitch of the current signal frame, including: obtaining the difference between the pitch corresponding to the current signal frame and the reference pitch; when the difference exceeds a predetermined range, adjusting the pitch corresponding to the current signal frame to be in a predetermined proportion to the reference pitch; and using the adjusted pitch as the target pitch of the current signal frame.

[0008] Optionally, based on the target pitch, the current signal frame is processed, including: obtaining a target frequency set corresponding to the target pitch of each signal frame of the human voice signal in the recorded audio; smoothing the target frequency set; obtaining the frequency of the pitch of the current signal frame; and adjusting the frequency to the target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

[0009] Optionally, the frequency is adjusted to a target frequency corresponding to a target pitch of a current signal frame in the target frequency set after smoothing, including: obtaining a ratio of the frequency to a target frequency corresponding to a target pitch of a current signal frame in the target frequency set after smoothing; inputting the ratio into a phase vocoder or a fundamental frequency synchronization superposition unit, and adjusting the frequency to a target frequency corresponding to a target pitch of a current signal frame in the target frequency set after smoothing.

[0010] Optionally, the pitch corresponding to each signal frame of the human voice signal is obtained by the following operations: obtaining the human voice signal in the recorded audio through voice endpoint detection; and obtaining the pitch corresponding to each signal frame of the human voice signal based on the fundamental frequency of each signal frame in the human voice signal.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided an audio processing device, comprising: a pitch acquisition unit, configured to acquire the pitch of a musical scale corresponding to the original audio based on tonality information and mode information of the original audio corresponding to the recorded audio; a processing unit, configured to perform the following operations for each signal frame of a vocal signal in the recorded audio: determine a reference pitch of a current signal frame based on the pitch of the musical scale corresponding to the original audio; adjust the pitch corresponding to the current signal frame based on the reference pitch of the current signal frame to obtain a target pitch of the current signal frame; and process the current signal frame based on the target pitch.

[0012] Optionally, the processing unit is further configured to use the pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame as the reference pitch of the current signal frame.

[0013] Optionally, the processing unit is further configured to obtain the difference between the pitch corresponding to the current signal frame and the reference pitch; when the difference exceeds a predetermined range, adjust the pitch corresponding to the current signal frame to be in a predetermined proportion to the reference pitch; and use the adjusted pitch as the target pitch of the current signal frame.

[0014] Optionally, the processing unit is also configured to obtain a target frequency set corresponding to the target pitch of each signal frame of the human voice signal in the recorded audio; smooth the target frequency set; obtain the frequency of the pitch of the current signal frame; and adjust the frequency to the target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

[0015] Optionally, the processing unit is also configured to obtain a ratio of the frequency to a target frequency corresponding to a target pitch of the current signal frame in the target frequency set after smoothing; input the ratio into a phase vocoder or a fundamental frequency synchronization superposition unit, and adjust the frequency to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

[0016] Optionally, the pitch corresponding to each signal frame of the human voice signal is obtained by the following operations: obtaining the human voice signal in the recorded audio through voice endpoint detection; and obtaining the pitch corresponding to each signal frame of the human voice signal based on the fundamental frequency of each signal frame in the human voice signal.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the audio processing method according to the present disclosure.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the audio processing method according to the present disclosure as described above.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the audio processing method according to the present disclosure when executed by a processor.

[0020] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0021] According to the audio processing method and device disclosed in the present invention, based on the tonality information and mode information of the original audio corresponding to the recorded audio, the pitch of the scale corresponding to the original audio is obtained, and then the reference pitch determined based on the pitch of the scale corresponding to the original audio is combined with the pitch of the vocal signal of the recorded audio to obtain the target pitch for correcting the vocal signal, so that the MIDI pitch of the music corresponding to the audio is no longer used as the target pitch for correcting the audio, so that a target pitch with a good match with the audio can be obtained, so that the recorded audio can be corrected according to the accurate target pitch to obtain an accurate tuning result. Therefore, the present invention solves the problem of inaccurate tuning in the related art.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0024] Figure 1 is a schematic diagram showing an implementation scenario of an audio processing method according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a system architecture diagram of an audio processing method according to an exemplary embodiment;

[0026] Figure 3 is a flowchart of an audio processing method according to an exemplary embodiment;

[0027] Figure 4 is a schematic diagram of a compression process according to an exemplary embodiment;

[0028] Figure 5 is a flow chart of a phase vocoder according to an exemplary embodiment;

[0029] Figure 6 is a block diagram of an audio processing device according to an exemplary embodiment;

[0030] Figure 7 is a block diagram of an electronic device 700 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0033] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0034] The current automatic pitch correction function strongly relies on the MIDI reference signal. When the song does not have a MIDI reference signal, the problem of automatic pitch correction cannot be solved. In addition, although the electronic sound effects can correct the pitch of the human voice to a certain extent, it will bring many unnatural processing traces (artifacts). The main reasons for the unnatural pitch correction caused by the current electronic sound effects are as follows: the human voice will have natural pitch jitter and strong jitter with the airflow, and the pitch is too closely connected, resulting in a fast glissando effect. In addition, with the emergence of karaoke software, the public can experience karaoke at home without leaving home, and share songs on the platform for others to appreciate. However, not every user has the singing skills of a professional singer, so there will be problems such as out of tune, unstable breath causing pitch drift, vibrato, broken voice, etc. However, for users who lack professional knowledge, manual pitch correction is not very practical.

[0035] In view of the above problems, the present disclosure provides an audio processing method, which can improve the problems of out-of-tune, broken voice, vibrato caused by unstable breath, etc. while ensuring the naturalness of human voice. The following is an example of performing audio correction on a recorded adapted song A.

[0036] Figure 1 is a schematic diagram showing an implementation scenario of an audio processing method according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can install an application for recording sound. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.

[0037] The user terminal 110 or the user terminal 120 records the adapted song A sung by the singer, and uploads the recorded adapted song A to the server 100. The server 100 obtains the tonality information and mode information of the original song A, and based on this, obtains the pitch of the scale corresponding to the original song A. For each signal frame of the vocal signal in the adapted song A, the following operations are performed: based on the pitch of the scale corresponding to the original song A, the reference pitch of the current signal frame is determined; based on the reference pitch of the current signal frame, the pitch corresponding to the current signal frame is adjusted to obtain the target pitch of the current signal frame; based on the target pitch, the current signal frame is processed. After each signal frame of the vocal signal in the adapted song A is processed, the revised adapted song A is obtained.

[0038] Figure 2 is a system architecture diagram of an audio processing method according to an exemplary embodiment, still taking singing adapted song A as an example. Figure 2As shown, first, the recorded audio (Vocal) is input into the VAD module to divide the audio into a vocal signal part and a silent part, and then the pitch (pitch) of the vocal signal of the vocal signal part is detected to obtain a vocal frequency sequence that changes over time, and convert it into a corresponding midi pitch, that is, the midi pitch corresponding to the vocal signal. In addition, the mode and tonality (BGM Tonality) of the original song A are obtained from the server, the pitch of the scale corresponding to the song is determined, and the pitch of the scale closest to the pitch of each signal frame in the vocal signal is found as the reference pitch. Combined with the reference pitch, the predetermined range and the midi pitch corresponding to the vocal signal obtained above, the target frequency is obtained. Specifically, in order to restore the naturalness of the vocal, an error tolerance value (equivalent to the predetermined range) is set. The pitch of the vocal is not adjusted within the allowable pitch deviation range. If it exceeds the tolerance range, the vocal pitch is compressed to reduce the gap between the vocal pitch and the reference pitch, so that the vocal naturally shakes within the allowable range, and the frequency corresponding to the pitch of each signal frame in the compressed vocal signal is used as the target frequency. After obtaining the target frequency, smoothing processing can also be performed, that is, a pitch transition range can be set to make the change connection more natural and solve the problem of portamento caused by sudden pitch switching. Then calculate the ratio of the target frequency to the frequency of the corresponding signal frame in the midi pitch corresponding to the vocal signal, and then adjust the vocal pitch according to the ratio using a phase vocoder or PSOLA to achieve natural sound correction. Simply put, the target MIDI pitch is obtained based on the tonal information of different songs and the MIDI pitch of the vocal signal in the recorded audio to adjust the vocal pitch in real time. In order to ensure the authenticity and naturalness of the vocal, an allowable error range is set, and the target MIDI pitch is smoothed, thereby achieving the effect of improving sound defects such as out-of-tune, broken voice, and unstable breath, and is compatible with people with musical literacy to adapt songs.

[0039] Next, we will refer to Figures 3 to 6 An audio processing method and apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0040] Figure 3 is a flowchart of an audio processing method according to an exemplary embodiment. Figure 3 As shown, the training method of the transfer learning model includes the following steps:

[0041] In step S301, based on the tonality information and mode information of the original audio corresponding to the recorded audio, the pitch of the scale corresponding to the original audio is obtained.

[0042] In step S302, for each signal frame of the human voice signal in the recorded audio, the following operations are performed: based on the pitch of the scale corresponding to the original audio, a reference pitch of the current signal frame is determined; based on the reference pitch of the current signal frame, the pitch corresponding to the current signal frame is adjusted to obtain a target pitch of the current signal frame; based on the target pitch, the current signal frame is processed.

[0043] According to an exemplary embodiment of the present disclosure, based on the pitch of the scale corresponding to the original audio, determining the reference pitch of the current signal frame includes: taking the pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame as the reference pitch of the current signal frame. Through this embodiment, a more accurate reference pitch can be obtained.

[0044] Specifically, the tonality information of the music (e.g., the original song) corresponding to the audio can be obtained from the server. Generally speaking, tonality is 24 major and minor keys. The twelve notes in an octave can each become the tonic of a key, so there will be twelve major keys and twelve minor keys, a total of which are the so-called twenty-four major and minor keys. Each tonality has its fixed scale, such as the scale arrangement of C natural major is: C, D, E, F, G, A, B, the scale arrangement of a natural minor is A, B, C, D, E, F, G, and the scale of a natural minor is consistent with that of C natural major. However, usually in composition, the minor key is usually a harmonic minor, and the last note needs to be raised by a semitone, so the scale arrangement of a harmonic minor is A, B, C, D, E, F, #G. Therefore, for songs in minor keys, the target scale is set to include both G and #G, taking into account both natural minor and harmonic minor. Then, the midi pitches of the target scale from C1 to C6 are obtained as the pitches of the scale corresponding to the original audio. The pitch of the scale corresponding to the original audio is then compared with the pitch of the vocal signal, and the pitch of the scale corresponding to the original audio that is closest to each signal frame of the vocal signal (i.e., the absolute value of the difference is the smallest) is found as the reference pitch of the current frame.

[0045] According to an exemplary embodiment of the present disclosure, based on the reference pitch of the current signal frame, the pitch corresponding to the current signal frame is adjusted to obtain the target pitch of the current signal frame, including: obtaining the difference between the pitch corresponding to the current signal frame and the reference pitch; when the difference exceeds a predetermined range, adjusting the pitch corresponding to the current signal frame to be in a predetermined proportion to the reference pitch; and using the adjusted pitch as the target pitch of the current signal frame. The above-mentioned predetermined proportion is a preset setting. Through this embodiment, the target pitch can be obtained conveniently and quickly.

[0046] For example, in order to maintain the authenticity of the human voice and reduce portamento and processing traces, a pitch deviation threshold range (i.e., the above-mentioned predetermined range) can be set, such as plus or minus 30 cents. When the human voice pitch exceeds the error range allowed by the current frame, i.e., exceeds the predetermined range, the human voice pitch is compressed. For example, the human voice pitch can be compressed to a predetermined ratio with the corresponding reference target pitch. The predetermined ratio can be preset, such as 4:1. Figure 4 As shown, the actual vocal pitch and the compressed vocal pitch are given, wherein the middle horizontal line represents the reference target pitch, and the two dotted lines are the upper and lower limits of the predetermined range. The compressed vocal pitch is obtained by calculation, and the compressed vocal pitch is used as the target MIDI pitch. In this way, the pitch deviation can be reduced while ensuring the normal jitter of the vocal pitch.

[0047] According to an exemplary embodiment of the present disclosure, based on the target pitch, the current signal frame is processed, including: obtaining a target frequency set corresponding to the target pitch of each signal frame of the human voice signal in the recorded audio; smoothing the target frequency set; obtaining the frequency of the pitch of the current signal frame; and adjusting the frequency to the target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing. Through this embodiment, the target frequency set is smoothed, so that the pitch of the adjusted current signal frame changes more naturally in the human voice signal, solving the problem of glissando caused by sudden pitch switching.

[0048] For example, after obtaining the target pitch, in order to ensure that the pitch connection is not unnatural, the target frequency corresponding to the target pitch of each signal frame of the human voice signal can be obtained, and all the target frequencies can be merged to obtain a target frequency set, and then the target frequency set is smoothed, that is, the transition time is lengthened. Then, based on the smoothed target frequency set, the frequency of the human voice signal is corrected, thereby achieving the effect of correcting the human voice signal.

[0049] According to an exemplary embodiment of the present disclosure, adjusting the frequency to a target frequency corresponding to a target pitch of a current signal frame in a target frequency set after smoothing includes: obtaining a ratio of the frequency to a target frequency corresponding to a target pitch of a current signal frame in a target frequency set after smoothing; inputting the ratio into a phase vocoder or a pitch synchronization superposition unit, and adjusting the frequency to a target frequency corresponding to a target pitch of a current signal frame in a target frequency set after smoothing. Through this embodiment, a phase vocoder or PSOLA can be used to quickly and automatically correct the pitch.

[0050] For example, the ratio of the target frequency in the target frequency set to the frequency of the frame corresponding to the human voice frequency sequence can be calculated. According to the obtained ratio, each frame frequency is processed, and each frame frequency is moved to the target frequency to obtain a human voice signal after pitch shift. Optionally, the processing of changing pitch without changing speed can be achieved by a phase vocoder or PSOLA. Both are currently existing technical solutions. The phase vocoder is briefly discussed below, and PSOLA is not discussed in detail.

[0051] The core of the phase vocoder is the short-time Fourier transform (STFT), which is usually encoded using the fast Fourier transform. The STFT converts the time domain representation of the sound into a frequency representation, allowing the amplitude or phase of a specific frequency component of the sound to be modified in the frequency domain. Finally, the modified time domain audio file is obtained again through the inverse STFT. The pitch of the human voice after the pitch shift is more accurate than before and there is no obvious processing effect. Figure 5 This is the flow chart of the phase vocoder.

[0052] According to an exemplary embodiment of the present disclosure, the pitch corresponding to each signal frame of the human voice signal is obtained by the following operations: obtaining the human voice signal in the recorded audio through voice endpoint detection; and obtaining the pitch corresponding to each signal frame of the human voice signal based on the fundamental frequency of each signal frame in the human voice signal. Through this embodiment, the human voice information in the audio can be obtained, thereby obtaining the corresponding MIDI pitch, and subsequently only the human voice part can be corrected without correcting other parts, thereby reducing the workload of correction.

[0053] For example, the audio recorded in real time can be passed through a voice activity detection (VAD) module to determine whether it is a human voice part (i.e., the human voice signal mentioned above) or a silent part without a human voice, such as a pure music part. An optional method: The VAD module can be implemented by calculating the short-time average energy and the short-time zero-crossing rate by a double threshold method. Since the energy of voiced sounds is higher than that of unvoiced sounds, and the zero-crossing rate of unvoiced sounds is higher than that of the silent part, the energy can be used to distinguish the voiced sound part first, and then the zero-crossing rate can be used to extract the unvoiced sound as well, thus completing the endpoint detection, that is, extracting the human voice part.

[0054] The calculation formula of short-time average energy is as follows:

[0055]

[0056] Among them, χ(m) represents the audio recorded in real time, ω(nm) represents the window function corresponding to the m-frame audio signal, N represents the window length, and n represents the sampling point.

[0057] The calculation of short-time zero-crossing rate is shown as follows:

[0058]

[0059] According to an exemplary embodiment of the present disclosure, based on the fundamental frequency of each signal frame in the human voice signal, the pitch corresponding to each signal frame of the human voice signal is obtained, including: obtaining the fundamental frequency of each signal frame in the human voice signal; merging the fundamental frequencies that meet a predetermined range to obtain a human voice frequency sequence corresponding to the human voice signal; and converting the human voice frequency sequence into the pitch corresponding to each signal frame of the human voice signal. Through this embodiment, the pitch corresponding to each signal frame of the human voice signal can be obtained conveniently and quickly.

[0060] For example, after determining the vocal part, the fundamental frequency of the vocal part is detected. An optional method is to obtain the fundamental frequency of the vocal pitch through the autocorrelation coefficient. The calculation of the autocorrelation coefficient is shown as follows:

[0061]

[0062] Among them, τ is the delay index, acf is the corresponding autocorrelation coefficient, and N is the length of the vocal sequence χ(n) in the audio. Since its waveform is periodic, when the delay index is a multiple of the period, the autocorrelation coefficient can reach its maximum value. Therefore, by calculating the autocorrelation coefficient, the period of the waveform can be obtained. Finally, the frequency of this waveform can be obtained through the relationship between the period P and the sampling rate fs.

[0063]

[0064] In order to obtain the fundamental frequency sequence of the entire vocal part, the fundamental frequency of each frame of the vocal part can be detected. The fundamental frequency range of the vocal part is usually within 70Hz to 1000Hz. Frequencies outside this range are regarded as noise. The non-noise fundamental frequencies are combined together to obtain the vocal frequency sequence. Then, the vocal frequency sequence is converted into midi pitch according to the following formula. The conversion is shown as follows:

[0065] P = 69 + 12 × log2 (f Hz / 440) (6)

[0066] Figure 6 is a block diagram of an audio processing device according to an exemplary embodiment. Figure 6 The device includes a pitch acquisition unit 60 and a processing unit 62.

[0067] The pitch acquisition unit 60 is configured to acquire the pitch of the scale corresponding to the original audio based on the tonality information and mode information of the original audio corresponding to the recorded audio; the processing unit 62 is configured to perform the following operations for each signal frame of the vocal signal in the recorded audio: determine the reference pitch of the current signal frame based on the pitch of the scale corresponding to the original audio; adjust the pitch corresponding to the current signal frame based on the reference pitch of the current signal frame to obtain the target pitch of the current signal frame; and process the current signal frame based on the target pitch.

[0068] According to an embodiment of the present disclosure, the processing unit 62 is further configured to use the pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame as the reference pitch of the current signal frame.

[0069] According to an embodiment of the present disclosure, the processing unit 62 is also configured to obtain the difference between the pitch corresponding to the current signal frame and the reference pitch; when the difference exceeds a predetermined range, the pitch corresponding to the current signal frame is adjusted to be in a predetermined proportion to the reference pitch; and the adjusted pitch is used as the target pitch of the current signal frame.

[0070] According to an embodiment of the present disclosure, the processing unit 62 is also configured to obtain a target frequency set corresponding to a target pitch of each signal frame of a human voice signal in the recorded audio; perform smoothing on the target frequency set; obtain the frequency of the pitch of the current signal frame; and adjust the frequency to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

[0071] According to an embodiment of the present disclosure, the processing unit 62 is also configured to obtain a ratio of the frequency to a target frequency corresponding to a target pitch of the current signal frame in the target frequency set after smoothing; input the ratio into a phase vocoder or a fundamental frequency synchronization superposition unit, and adjust the frequency to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

[0072] According to an embodiment of the present disclosure, the pitch corresponding to each signal frame of the human voice signal is obtained by the following operations: obtaining the human voice signal in the recorded audio through voice endpoint detection; based on the fundamental frequency of each signal frame in the human voice signal, obtaining the pitch corresponding to each signal frame of the human voice signal.

[0073] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 7 It is a block diagram of an electronic device 700 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 701 and at least one processor 702, wherein a set of computer executable instructions is stored in the at least one memory, and when the computer executable instruction set is executed by the at least one processor, an audio processing method according to an embodiment of the present disclosure is executed.

[0074] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0075] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 702 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0076] The processor 702 may execute instructions or codes stored in the memory, wherein the memory 701 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0077] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory 701.

[0078] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0079] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the audio processing method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0080] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, and when the computer instructions are executed by a processor, the audio processing method of the embodiment of the present disclosure is implemented.

[0081] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0082] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: include: Based on the tonality information and mode information of the original audio corresponding to the recorded audio, obtaining the pitch of the scale corresponding to the original audio; For each signal frame of the human voice signal in the recorded audio, the following operations are performed: Determining a reference pitch of a current signal frame based on the pitch of a musical scale corresponding to the original audio; Based on the reference pitch of the current signal frame, adjusting the pitch corresponding to the current signal frame to obtain the target pitch of the current signal frame; Based on the target pitch, processing the current signal frame; The step of adjusting the pitch corresponding to the current signal frame based on the reference pitch of the current signal frame to obtain the target pitch of the current signal frame includes: Obtaining a difference between the pitch corresponding to the current signal frame and the reference pitch; When the difference exceeds a predetermined range, adjusting the pitch corresponding to the current signal frame to be in a predetermined proportion to the reference pitch; The adjusted pitch is used as the target pitch of the current signal frame.

2. The audio processing method according to claim 1, characterized in that: The step of determining a reference pitch of a current signal frame based on the pitch of a musical scale corresponding to the original audio comprises: The pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame is used as the reference pitch of the current signal frame.

3. The audio processing method according to claim 1, characterized in that: The processing of the current signal frame based on the target pitch includes: Obtaining a target frequency set corresponding to a target pitch of each signal frame of a human voice signal in the recorded audio; Performing smoothing processing on the target frequency set; Obtaining the frequency of the pitch of the current signal frame; The frequency is adjusted to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

4. The audio processing method according to claim 3, characterized in that: The step of adjusting the frequency to a target frequency corresponding to a target pitch of the current signal frame in the target frequency set after smoothing includes: Acquire a ratio of the frequency to a target frequency corresponding to a target pitch of the current signal frame in the target frequency set after smoothing; The ratio is input into a phase vocoder or a pitch synchronization superposition unit, and the frequency is adjusted to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

5. The audio processing method according to claim 1, characterized in that: The pitch corresponding to each signal frame of the human voice signal is obtained by the following operation: Acquire the human voice signal in the recorded audio through voice endpoint detection; Based on the fundamental frequency of each signal frame in the human voice signal, the pitch corresponding to each signal frame of the human voice signal is obtained.

6. An audio processing device, characterized in that: include: A pitch acquisition unit, configured to acquire the pitch of the scale corresponding to the original audio based on the tonality information and mode information of the original audio corresponding to the recorded audio; The processing unit is configured to perform the following operations for each signal frame of the human voice signal in the recorded audio: determining a reference pitch of a current signal frame based on the pitch of the scale corresponding to the original audio; Based on the reference pitch of the current signal frame, adjusting the pitch corresponding to the current signal frame to obtain the target pitch of the current signal frame; Based on the target pitch, processing the current signal frame; The processing unit is further configured to obtain a difference between the pitch corresponding to the current signal frame and the reference pitch; When the difference exceeds a predetermined range, the pitch corresponding to the current signal frame is adjusted to be in a predetermined proportion to the reference pitch; and the adjusted pitch is used as the target pitch of the current signal frame.

7. The audio processing device according to claim 6, characterized in that: The processing unit is further configured to use the pitch with the smallest absolute difference between the pitch of the scale corresponding to the original audio and the pitch of the current signal frame as the reference pitch of the current signal frame.

8. The audio processing device according to claim 6, characterized in that: The processing unit is further configured to obtain a target frequency set corresponding to a target pitch of each signal frame of a human voice signal in the recorded audio; and perform smoothing processing on the target frequency set; Acquire the frequency of the pitch of the current signal frame; and adjust the frequency to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

9. The audio processing device according to claim 8, characterized in that: The processing unit is further configured to obtain a ratio of the frequency to a target frequency corresponding to a target pitch of the current signal frame in the target frequency set after smoothing; input the ratio into a phase vocoder or a fundamental frequency synchronization superposition unit, and adjust the frequency to a target frequency corresponding to the target pitch of the current signal frame in the target frequency set after smoothing.

10. The audio processing device according to claim 6, characterized in that: The pitch corresponding to each signal frame of the human voice signal is obtained by the following operations: acquiring the human voice signal in the recorded audio through voice endpoint detection; and obtaining the pitch corresponding to each signal frame of the human voice signal based on the fundamental frequency of each signal frame in the human voice signal.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio processing method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the audio processing method according to any one of claims 1 to 5.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the audio processing method according to any one of claims 1 to 5 is implemented.

Images