Audio pitch changing method and device

Abstract

The present application provides an audio pitch-shifting method and device, wherein the audio pitch-shifting method comprises: obtaining an audio to be processed and a target pitch; dividing the audio to be processed into a plurality of audio segments, determining the pitch period of each audio segment, and determining the initial pitch of each audio segment based on the pitch period of each audio segment; determining the pitch-shifting coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment; processing the audio to be processed based on the pitch-shifting coefficients of the plurality of audio segments to obtain a pitch-shifted audio. In this way, the user only needs to input the audio to be processed and the target pitch to be obtained, and the pitch-shifting processing of the audio to be processed can be automatically realized without user participation, and the interactive process is relatively simple, which improves the user experience, and does not need to perform a large number of calculation operations, which reduces the amount of calculation and the complexity of operation, thereby reducing the time consumption of the audio pitch-shifting processing and improving the efficiency of the audio pitch-shifting processing.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio pitch shifting method, an audio pitch shifting device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of audio processing technology, people can perform various processing on audio to achieve different audio effects. For example, audio can be processed by tone changing, changing the pronunciation of each Chinese character in the audio to a specific tone, such as changing the pronunciation of the yinping tone to the qusheng tone, so as to obtain audio that meets user needs.

[0003] Most existing audio editing software has a built-in pitch shifting function, which usually uses frequency domain analysis algorithms such as phase vocoder and FDPSOLA to perform pitch shifting on the audio. However, such algorithms are not only computationally intensive but also complex to operate, resulting in a long time spent on pitch shifting the audio, which reduces the efficiency of audio pitch shifting processing. In addition, the user needs to participate in adjusting the pitch shift coefficient during the pitch shifting process, and the interactive process is relatively complicated. Summary of the invention

[0004] In view of this, an embodiment of the present application provides an audio pitch shifting method. The present application also relates to an audio pitch shifting device, a computing device, and a computer-readable storage medium to solve the problems of low audio pitch shifting processing efficiency and complex interaction process in the prior art.

[0005] According to a first aspect of an embodiment of the present application, there is provided an audio pitch shifting method, comprising:

[0006] Get the audio to be processed and the target tone;

[0007] Dividing the audio to be processed into a plurality of audio segments, determining a pitch period of each audio segment, and determining an initial pitch of each audio segment based on the pitch period of each audio segment;

[0008] Determining a pitch change coefficient corresponding to each audio segment based on the target pitch and an initial pitch of each audio segment;

[0009] The audio to be processed is processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch shifted audio.

[0010] According to a second aspect of an embodiment of the present application, there is provided an audio pitch shifting device, comprising:

[0011] An acquisition module, configured to acquire the audio to be processed and the target tone;

[0012] A first determination module is configured to divide the audio to be processed into a plurality of audio segments, determine a pitch period of each audio segment, and determine an initial pitch of each audio segment based on the pitch period of each audio segment;

[0013] A second determination module is configured to determine a pitch change coefficient corresponding to each audio segment based on the target pitch and an initial pitch of each audio segment;

[0014] The processing module is configured to process the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch shifted audio.

[0015] According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the audio pitch shifting method when executing the instructions.

[0016] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the audio pitch shifting method are implemented.

[0017] The audio pitch shifting method provided by the present application obtains the audio to be processed and the target pitch, divides the audio to be processed into multiple audio segments, determines the pitch period of each audio segment, and determines the initial pitch of each audio segment based on the pitch period of each audio segment; determines the pitch shift coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment; processes the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch-shifted audio. By performing audio pitch shifting processing using the above method, the user only needs to input the audio to be processed and the desired target pitch, and the pitch shifting processing of the audio to be processed can be automatically realized without user participation, and the interactive process is relatively simple, which improves the user experience, and does not require the execution of a large number of calculation operations, which reduces the amount of calculation and operation complexity, thereby reducing the time consumption of audio pitch shifting processing and improving the efficiency of audio pitch shifting processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of an audio pitch shifting method provided by an embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of a tone change when a tone is pronounced provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an audio signal segment in a discrete audio signal provided by an embodiment of the present application;

[0021] Figure 4is a schematic diagram of an audio pitch shifting process provided by an embodiment of the present application;

[0022] Figure 5 is a processing flow chart of another audio pitch shifting method provided by an embodiment of the present application;

[0023] Figure 6 It is a structural schematic diagram of an audio pitch shifting device provided in one embodiment of the present application;

[0024] Figure 7 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of the present application are explained.

[0029] Pitch shifting: Adjusting the frequency of a sound to change it.

[0030] Pitch: The frequency of a sound is called pitch, which indicates the degree to which human hearing can distinguish the pitch of a sound.

[0031] Tone: specifically refers to the four pronunciations of Yinping, Yangping, Shangsheng and Qusheng in modern Chinese. Its essential feature is the continuous change of tone in time sequence.

[0032] Pitch period: When a person pronounces a voiced sound, the airflow through the glottis causes the vocal cords to vibrate in a tension-relaxation oscillatory manner, generating a quasi-periodic pulsed airflow. This airflow excites the vocal tract to produce voiced sounds, also known as voiced speech, which carries most of the energy in the speech. The frequency of this vocal cord vibration is called the fundamental frequency, and the corresponding period is called the pitch period.

[0033] Normalize: Adjust the pitch of several audio clips to the same value.

[0034] Ghost video: A common type of original video on video sites. This type of video uses highly synchronized, quickly repeated material with the rhythm of background music to achieve a funny effect.

[0035] Chinese syllable tone change algorithm: an algorithm that changes the input Chinese pronunciation into a specific tone, such as changing the pronunciation of the yinping tone into the qusheng tone.

[0036] Phoneme: The smallest speech unit divided according to the natural properties of speech.

[0037] Next, the application scenarios of the audio pitch shifting method provided in this application are briefly introduced.

[0038] In the field of ghost video creation, creators need to select appropriate clips from the materials to form a coherent audio and video. However, due to the limitation of the materials, it is necessary to change the tones of the existing Chinese character audio to expand the material library. Usually, the Chinese syllable tone change algorithm is used to expand the audio of a specific tone into four tones of audio, which is equivalent to expanding the original material library by four times, providing more possibilities for creation.

[0039] Most audio editing software on the market currently has built-in tone-shifting functions. However, due to the special linguistic characteristics of Chinese itself, the pronunciation of a syllable implies continuous changes in pitch, which are divided into four tones: yinping, yangpi, shangsheng and qusheng. This tone-shifting function usually uses frequency domain analysis algorithms for tone-shifting processing. This type of algorithm often uses fast Fourier transform, inverse Fourier transform and some complex multiplication operations in the analysis. The calculation amount is large and the operation is complex, and the processing speed is slow, resulting in reduced efficiency of audio tone-shifting processing. In addition, during the operation, the user needs to participate in adjusting the tone-shifting coefficient, which requires the user to master relevant knowledge of Chinese pronunciation, tone, and intonation. Not only is the interaction process complicated, but the learning cost is also high for the user, which increases the threshold for users to use audio editing software and reduces the user experience.

[0040] To this end, the present application provides a method for audio pitch shifting, which can solve the above-mentioned problem. The specific implementation of the method can be found in the description of the following embodiments.

[0041] In this application, an audio pitch-changing method is provided. This application also relates to an audio pitch-changing device, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0042] Figure 1 FIG. 4 shows a flowchart of an audio pitch-changing method according to an embodiment of the present application, which specifically includes the following steps:

[0043] Step 102: Obtain the audio to be processed and the target pitch.

[0044] In the embodiments of the present application, the pitch-changing process of the audio takes the pronunciation audio of a single character as the processing object, that is, the audio to be processed is the audio of a single character. The target pitch is the desired pitch, that is, after the pitch-changing process of the audio to be processed of a certain Chinese character, the new pitch of the Chinese character is expected.

[0045] For example, for the character "我" (I), its initial pitch is "上声ˇ" (the third tone in general), and now the user wants to adjust its pitch to "阳平ˊ" (the second tone in general), then "阳平ˊ" is the target pitch of the Chinese character "我".

[0046] It should be noted that the target pitch can be any pitch in Chinese, which can be set according to the actual needs of the user, or can also be set by default by the device. The embodiments of the present application do not limit this. For example, the target pitch can be any one of "阴平", "阳平", "上声", and "去声".

[0047] In the field of video creation, the creator usually obtains a piece of audio of the material to be processed. In this case, the audio of a certain character in the audio of the material to be processed can be pitch-changed, or the entire audio of the material to be processed can be pitch-changed. However, in either case, it is necessary to extract the audio of a single character from the audio of the material to be processed, and then perform pitch-changing processing on the audio of the single character.

[0048] Among them, the audio of the material to be processed can be an audio including the audio of at least one character. For example, the audio of the material to be processed can be the audio of a paragraph, the audio of a reading of an article, etc.

[0049] In implementation, the specific implementation of obtaining the audio to be processed may include:

[0050] Obtain the audio of the material to be processed, where the audio of the material to be processed includes the audio of the character to be processed; determine the pronunciation start position and pronunciation end position of the character to be processed in the audio of the material to be processed; and determine the audio segment between the pronunciation start position and the pronunciation end position in the audio of the material to be processed as the audio to be processed.

[0051] Among them, the text to be processed is the text for which the audio is to be pitch-shifted. The pronunciation start position is the position where the pronunciation audio of the text to be processed starts, and the pronunciation end position is the position where the pronunciation audio of the text to be processed ends.

[0052] In practical applications, the material to be pitch-shifted can be obtained from the material library as the audio of the material to be processed, and the audio of the text to be processed that needs to be pitch-shifted is included in the audio of the material to be processed. Since the audio of the material to be processed may include the audio of multiple texts, the specific implementation process of pitch-shifting may be different for the audio of different texts. Therefore, it is necessary to extract the audio of the text to be processed from the audio of the material to be processed according to the pronunciation start position and the pronunciation end position to obtain the audio to be processed.

[0053] In some embodiments, a phoneme splitting algorithm based on deep learning can be used to determine the pronunciation start position and the pronunciation end position of the text to be processed by analyzing the phoneme changes between audio frames in the audio of the material to be processed. As an example, the position of the audio frame where the first phoneme of the pronunciation of the text to be processed is located can be determined as the pronunciation start position, and the position of the audio frame where the last phoneme of the pronunciation of the text to be processed is located can be determined as the pronunciation end position.

[0054] For example, assume that the audio of the material to be processed is the audio of a passage "School starts today", and the text to be processed is "tian". Then the phonemes in the audio of the material to be processed include "jintiankaixue". Assume that the time length of this section of audio is 1 s, and the phoneme of "tian" is "tian". Then the position of the audio frame where the phoneme "t" is located in the audio of the material to be processed can be determined as the pronunciation start position, and the position of the audio frame where the phoneme "n" that is located after the phoneme "t" and is the closest to the phoneme "t" in the audio of the material to be processed can be determined as the pronunciation end position.

[0055] In the embodiments of the present application, the audio to be processed of a single text is used as the processing object for pitch-shifting, and the target pitch is set according to needs. Then, a single text can be pitch-shifted to any desired pitch, making the pitch-shifted audio more in line with the user's needs and improving the user experience.

[0056] Step 104: Divide the audio to be processed into multiple audio segments, determine the fundamental period of each audio segment, and determine the initial pitch of each audio segment based on the fundamental period of each audio segment.

[0057] In the embodiments of the present application, refer to Figure 2 , Figure 2 is a schematic diagram of the pitch change during the pronunciation of a pitch in the embodiments of the present application. In Figure 2Medium, high, half high, medium, half low and low represent the pitch, the numbers represent the pitch coefficient, and the straight line / broken line with arrows represents the change of pitch. Figure 2 It can be seen that the tone "Yinping" includes one tone, and the tone remains at a high pitch during its pronunciation. The tone "Yangping" includes three tones, and the changes of the tones during its pronunciation are medium, half-high and high, respectively. The tone "Shangsheng" includes five tones, and the changes of the tones during its pronunciation are half-low, low, half-low, medium and half-high, respectively. The tone "Qusheng" includes five tones, and the changes of the tones during its pronunciation are high, half-high, medium, half-low and low, respectively.

[0058] It should be noted that the embodiments of the present application are only Figure 2 Taking the example of the change of the tone when the tone is pronounced as an example, in other embodiments, a tone may include more tones, and the change of its tone may also be more complicated, which is not limited in the embodiment of the present application. For example, a tone may include 7 tones or 10 tones, etc.

[0059] According to Figure 2 From the analysis, we know that the tone is the continuous change of frequency in time sequence, and the audio to be processed is the audio of the text to be processed, which represents the continuous change of frequency in time sequence when the tone of the text to be processed is pronounced. If the audio is not divided into segments and the audio to be processed is directly processed, a prerequisite is required, which is that the frequency of the audio to be processed is constant. Figure 2 It can be known that, when the tone of the text to be processed is other three tones except Yinping, the frequency of the audio to be processed can not be constant, that is, when the tone of the text to be processed is other tones except Yinping, this prerequisite is not established, then the audio to be processed can not be directly subjected to tone change processing, or in other words, if the audio segment is not divided, the tone change audio obtained based on the audio to be processed will not only be seriously distorted, but also the tone may be wrong, causing the tone change audio obtained to have problems, and the effect that the user wants cannot be reached. Therefore, when the tone of the text to be processed is other three tones except Yinping, it is necessary to segment the audio to be processed, and then carry out subsequent processing based on the audio segmentation. When the tone of the text to be processed is Yinping, the frequency of the audio to be processed is constant, and therefore subsequent processing can also be performed without dividing the audio segment. However, human pronunciation may not completely conform to the theoretical changes in pitch when pronouncing tones, so even if the tone of the text to be processed is yinping, the frequency of the audio to be processed may also change. In this case, in order to avoid distortion of the pitch-shifted audio or turning it into a more mechanical sound, resulting in the pitch-shifting processing result not meeting user needs, the audio to be processed can be segmented, and then subsequent processing can be performed based on the audio segmentation.

[0060] In one implementation, the audio to be processed may be divided into a plurality of audio segments according to a preset analysis frame length. For example, assuming that the preset analysis frame length is 40 ms, the audio to be processed is divided into an audio segment every 40 ms. If the duration of the audio to be processed is 200 ms, five audio segments may be obtained.

[0061] It should be noted that the preset analysis frame length can be determined based on big data for analyzing the frequency of the audio, or based on a large amount of experience, or by the default setting of the device, and the embodiments of the present application do not limit this.

[0062] In another implementation, the audio to be processed may be divided according to the number of tones included in the tones of the to-be-processed text and the number of tones included in the target tones. As an example, the audio to be processed may be first divided into the number of tones segments according to the number of tones included in the tones of the to-be-processed text, and then the quotient of the number of tones included in the tones of the to-be-processed text and the number of tones included in the target tones is determined, and if the quotient is greater than or equal to 1, the division is stopped, and if the quotient is less than 1, each segment is continuously divided until the number of audio segments obtained is greater than or equal to 1 times the number of tones included in the target tones.

[0063] For example, assuming that the tone of the text to be processed is Yangping, which includes 3 tones, and the target tone is Qusheng, which includes 5 tones, the audio to be processed can be divided into 3 segments first, and then each segment is divided into two, so that 6 audio segments can be obtained, and the number of audio segments 6 is more than 1 times the number of tones 5 included in the target tone, then the division can be stopped, that is, the audio to be processed is divided into 6 audio segments.

[0064] In implementation, after multiple audio segments are obtained through division, the pitch period of the audio segments can be determined by using algorithms such as autocorrelation method, parallel processing method, average amplitude difference method, data reduction method, etc.

[0065] Taking the autocorrelation method as an example, the algorithm can determine the value of the autocorrelation function through the following formula, and determine the offset m corresponding to the maximum value as the fundamental pitch period of the audio segment.

[0066]

[0067] Where x(n) represents the audio segment, x(n+m) represents the audio segment shifted by m, R(m) is the value of the autocorrelation function, and m represents the offset.

[0068] After determining the pitch period of each audio segment by the above method, since there is a reciprocal relationship between the pitch period and the frequency, and the frequency and the pitch can be considered to be equivalent in this application, the pitch period of each audio segment has a reciprocal relationship with the initial pitch. Therefore, the reciprocal of the pitch period of each audio segment can be determined as the initial pitch of each audio segment.

[0069] In the embodiment of the present application, the audio to be processed is divided into multiple audio segments, which can avoid distortion of the subsequently obtained pitch-shifted audio, and the more audio segments are divided, the more accurate the audio pitch shifting processing is, and the better the pitch-shifted audio effect is; and, if the frequency in the same audio segment is relatively constant, then the period is also relatively constant, therefore, the fundamental pitch period of each audio segment can be determined, and the initial pitch can be determined based on the fundamental pitch period, and the accuracy of the initial pitch determined in this way is higher.

[0070] Step 106: Determine a pitch change coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment.

[0071] In the embodiment of the present application, the audio pitch change is actually to adjust the pitch of the text to be processed to the target pitch. Since the pitch is actually a continuous change of the pitch in time sequence, the target pitch may include multiple target pitches. In the above steps, the audio to be processed is divided into multiple audio segments, and the initial pitch of each audio segment may be different, and the adjusted target pitch of each audio segment may be different. Therefore, the pitch change coefficient of the pitch change processing on each audio segment may also be different. Therefore, after determining the initial pitch of each audio segment, it is necessary to determine the pitch change coefficient corresponding to each audio segment according to the target pitch and the initial pitch of each audio segment.

[0072] In implementation, based on the target tone and the initial tone of each audio segment, determining the pitch change coefficient corresponding to each audio segment may include:

[0073] The target tone included in the target tone is obtained, and the pitch coefficient of the target tone is determined based on the corresponding relationship between the preset pitch and the pitch coefficient; the pitch coefficient of each audio segment is determined based on the pitch coefficient of the target tone; the normalization coefficient of each audio segment is determined based on the initial pitch of each audio segment; and the pitch change coefficient of each audio segment is determined based on the pitch coefficient of each audio segment and the normalization coefficient of each audio segment.

[0074] In practical applications, the change of the pitch of the target tone during the pronunciation process can be determined first, that is, the target tone included can be determined. For the convenience of calculation, the corresponding relationship between the pitch and the pitch coefficient can be set in advance, and then the pitch coefficient of the target tone can be determined according to the corresponding relationship. Since the target tone is the tone that needs to be achieved after the audio segment is adjusted, and the number of target tones and the tone segments are not necessarily corresponding, it is necessary to determine the pitch coefficient of each audio segment according to the pitch coefficient of the target tone, and then normalize the tones of multiple audio segments to the same tone according to the initial pitch of each audio segment, that is, determine the normalization coefficient of each audio segment, and then determine the pitch change coefficient of each audio segment for pitch change processing based on the normalization coefficient and the pitch coefficient.

[0075] In an embodiment of the present application, multiple audio clips are normalized and a normalization coefficient is determined so that each audio clip is adjusted from the same pitch to the target pitch that it should reach. The pitch coefficient of each audio clip is determined, and the degree to which different audio clips are adjusted from the same pitch can be determined. Based on the normalization coefficient and the pitch coefficient, the pitch coefficient of each audio clip is determined. The processed audio can then be processed based on different pitch coefficients. The pitch coefficients of audio clips with different frequencies are different, so that the pitch processing is more in line with the actual situation, thereby making the processed pitch-shifted audio effect better.

[0076] In some embodiments, the corresponding relationship between audio and pitch coefficients can be preset. Figure 2 For example, assume that the preset correspondence between pitch and pitch coefficient includes: high pitch-pitch coefficient 5, half-high pitch-pitch coefficient 4, middle pitch-pitch coefficient 3, half-low pitch-pitch coefficient 2 and low pitch-pitch coefficient 1.

[0077] It should be noted that this application is only Figure 2 The correspondence between pitch and pitch coefficient is described by taking as an example. In other embodiments, a tone may include more pitches, and the correspondence between pitch and pitch coefficient may also include more groups. For example, a tone may include 7 tones or 10 tones, and the correspondence between pitch and pitch coefficient may include 7 groups or 10 groups of correspondence.

[0078] Exemplarily, assuming that the target tone is Yangping, and the target tone includes 3 tones, the target tones are middle, semi-high and high respectively. According to the above correspondence, the pitch coefficient of the middle tone is 3, the pitch coefficient of the semi-high tone is 4, and the pitch coefficient of the high tone is 5.

[0079] In practice, since the pitch of the yin-ping tone does not change during pronunciation, if the target tone is the yin-ping tone, the pitch to be adjusted is the same for each audio clip, but for other tones except the yin-ping tone, the pitch is constantly changing during pronunciation, so if the target tone is the yang-ping tone, the rising tone or the falling tone, the pitch to be adjusted is different for each audio clip. Therefore, for the two situations of adjusting each audio clip to the same pitch and adjusting to different pitches, the pitch coefficient can be determined in the following two ways respectively.

[0080] In a possible implementation, if the number of the target pitch is one, the specific implementation of determining the pitch coefficient of each audio segment based on the pitch coefficient of the target pitch may include:

[0081] The pitch coefficient of the target pitch is determined as the pitch coefficient of each audio segment.

[0082] In practical applications, if the number of target tones is one, it means that the target tone remains unchanged during the pronunciation process, and the target tone may be yinping. In this case, the pitch change processing needs to ensure that the adjusted pitch of each audio clip is the same. Therefore, the pitch coefficients of multiple audio clips are also the same and are all the pitch coefficients of the target tone.

[0083] by Figure 2 For example, when the number of target tones is one, the target tone is yinping, the target tone is a high tone, and the pitch coefficient of the high tone is 5, then it can be determined that the pitch coefficient of each audio segment is 5.

[0084] In an embodiment of the present application, if there is only one target tone, the tone coefficient of the target tone is determined as the tone coefficient of each audio segment, which can ensure that each audio segment is adjusted to the same tone, thereby ensuring that the frequency of the final pitch-shifted audio remains unchanged, thereby achieving the desired pronunciation effect of the target tone.

[0085] In another possible implementation, if there are multiple target tones, the specific implementation of determining the pitch coefficient of each audio segment based on the pitch coefficient of the target tone may include:

[0086] Based on the number of multiple audio segments and the number of multiple pitch coefficients, an audio segment group corresponding to each pitch coefficient is determined; and the pitch coefficient corresponding to the audio segment group to which each audio segment belongs is determined as the pitch coefficient of the audio segment.

[0087] In practical applications, if there are multiple target tones, it means that the target tones are constantly changing during the pronunciation process. In this case, the pitch change processing cannot adjust each audio clip to the same pitch. Each audio clip needs to be adjusted according to the changes after the target tones included in the target tones. Therefore, the pitch coefficients of multiple audio clips are different.

[0088] In some embodiments, the number of audio segments and pitch coefficients are both multiple, and here it is necessary to determine the pitch coefficient of each audio segment. Since more than two pitch segments can be adjusted to the same target pitch, but a single pitch segment is difficult to be adjusted to more than two different tones, therefore, in this solution, the number of audio segments is greater than the number of target tones.

[0089] As an example, the quotient of the number of audio clips and the number of pitch coefficients can be determined. If the quotient is an integer, the multiple audio clips are divided into groups equal to the number of pitch coefficients using the quotient as the number of each group of audio clips, and each audio clip group corresponds to a pitch coefficient. The pitch coefficient of each audio clip in the same audio clip group is the pitch coefficient corresponding to the audio clip group.

[0090] Exemplarily, assuming that the number of audio segments is 9 and the number of pitch coefficients is 3, the quotient value is determined to be an integer 3, and every 3 audio segments can be determined as an audio segment group. The first audio segment group corresponds to the first pitch coefficient, the second audio segment group corresponds to the second pitch coefficient, and the third audio segment group corresponds to the third pitch coefficient. Therefore, the pitch coefficients of the audio segments included in the first audio segment are all the first pitch coefficient, the pitch coefficients of the audio segments included in the second audio segment group are all the second pitch coefficient, and the pitch coefficients of the audio segments included in the third audio segment group are all the third pitch coefficient.

[0091] For example, assuming that the audio segments are A1, A2, A3, A4, A5, A6, A7, A8 and A9 in chronological order, and the pitch coefficients are m1, m2 and m3 in the order of change of the target pitch, then the pitch coefficients of the audio segments A1, A2 and A3 are all m1, the pitch coefficients of the audio segments A4, A5 and A6 are all m2, and the pitch coefficients of the audio segments A7, A8 and A9 are all m3.

[0092] As another example, the quotient of the number of audio segments and the number of pitch coefficients can be determined. If the quotient is not an integer, the number of pitch coefficients is determined with the quotient as a unit value, and the value of each non-integer value after rounding is determined. Each value is used as the sequence number of the last audio segment in each group in multiple audio segments, and the multiple audio segments are divided into the number of groups of pitch coefficients, and each audio segment group corresponds to a pitch coefficient. The pitch coefficient of each audio segment in the same audio segment group is the pitch coefficient corresponding to the audio segment group.

[0093] Exemplarily, assuming that the number of audio clips is 12 and the number of pitch coefficients is 5, the quotient is determined to be a non-integer 2.4, and then the five values ​​can be determined to be 2.4, 4.8, 7.2, 9.6 and 12, respectively, and the non-integer values ​​in the five values ​​are rounded off to obtain five values ​​of 2, 5, 7, 10 and 12, respectively. Therefore, it can be determined that the last audio clip in the first audio clip group is the audio clip with a sequence number of 2, that is, the first audio clip group includes the first audio clip and the second audio clip, and the first audio clip group corresponds to the first pitch coefficient; it is determined that the last audio clip in the second audio clip group is the audio clip with a sequence number of 5, that is, the second audio clip group includes the third audio clip, the fourth audio clip and the fifth audio clip, and the second audio clip group corresponds to the second pitch coefficient; and so on, it is determined that the last audio clip in the fifth audio clip group is the audio clip with a sequence number of 12, that is, the fifth audio clip group includes the eleventh audio clip and the twelfth audio clip, and the fifth audio clip group corresponds to the fifth pitch coefficient.

[0094] For example, assuming that the audio clips are A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, and A12 in chronological order, and the pitch coefficients are m1, m2, m3, m4, and m5 in the order of change of the target pitch, then audio clips A1 and A2 are a group and their pitch coefficients are all m1, audio clips A3, A4, and A5 are a group and their pitch coefficients are all m2, audio clips A6 and A7 are a group and their pitch coefficients are all m3, audio clips A8, A9, and A10 are a group and their pitch coefficients are all m4, and tone clips A11 and A12 are a group and their pitch coefficients are all m5.

[0095] Furthermore, when determining the pitch coefficient of each audio segment based on the pitch coefficient of the target pitch, the multiple audio segments can be sorted first according to the time sequence of the audio segments in the audio to be processed, and then the multiple audio segments are grouped in order based on the number of the multiple audio segments and the number of the multiple pitch coefficients, so that the number of audio segment groups obtained by the pitch coefficients are also arranged in time sequence, and the multiple pitch coefficients are also respectively corresponding to the audio segment groups according to the order of change of the target pitch before and after the target tone is pronounced. In this way, confusion can be avoided when performing the pitch change processing, and the pitch of the adjusted audio segment is not the desired pitch, so that the tone of the finally obtained pitch-changed audio is different from the target tone, affecting the effect of the pitch change processing.

[0096] In some embodiments, the specific implementation of determining the normalization coefficient of each audio segment based on the initial pitch of each audio segment may include: determining the average pitch of the audio to be processed based on the initial pitch of each audio segment, and determining the quotient of the initial pitch of each audio segment and the average pitch as the normalization coefficient of each audio segment.

[0097] In practical applications, for ease of processing, the pitch of each audio clip can be normalized to the same pitch. Therefore, the initial pitches of multiple audio clips can be added and the average value can be determined to obtain the average pitch of the audio to be processed. Then, the normalization coefficient of each audio clip can be determined according to the formula: normalization coefficient = average pitch of the audio to be processed ÷ initial pitch of each audio clip. For example, assuming that the initial pitch of audio clip A is 10 and the average pitch of the audio to be processed is 8, the normalization coefficient of audio clip A can be determined to be 0.8.

[0098] In some embodiments, after determining the pitch coefficient and normalization coefficient of each audio segment, since the pitch coefficient is used to adjust the pitch of the audio segment based on the same pitch, but in fact, the pitch of each audio segment is not necessarily the same, the pitch coefficient and the normalization coefficient can be multiplied, and the product can be determined as the pitch coefficient of each audio segment. The pitch shifting processing based on the pitch shifting coefficient is more in line with the actual situation of each audio segment, and can improve the accuracy of the pitch obtained after the pitch shifting processing, thereby improving the effect of the final pitch-shifted audio.

[0099] Exemplarily, assume that the audio to be processed includes 6 audio segments, namely A1, A2, A3, A4, A5 and A6, and the normalization coefficients of the 6 audio segments are n1, n2, n3, n4, n5 and n6, respectively, and the pitch coefficient of audio segment A1 is m1, the pitch coefficient of audio segment A2 is m2, the pitch coefficients of audio segments A3 and A4 are both m3, the pitch coefficient of audio segment A5 is m4, and the pitch coefficient of audio segment A6 is m5. Therefore, it can be determined that the pitch coefficient of audio segment A1 is n1×m1, the pitch coefficient of audio segment A2 is n2×m2, the pitch coefficient of audio segment A3 is n3×m3, the pitch coefficient of audio segment A4 is n4×m3, the pitch coefficient of audio segment A5 is n5×m4, and the pitch coefficient of audio segment A6 is n6×m5.

[0100] In an embodiment of the present application, the pitch coefficient of each audio segment is determined according to the pitch coefficient of the target pitch, and the initial pitch is normalized to the same pitch, and then the pitch change coefficient is determined based on the pitch coefficient of each audio segment and the normalization coefficient, so that better pitch-changed audio can be obtained.

[0101] Step 108: Process the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch shifted audio.

[0102] When performing pitch shifting processing, the embodiment of the present application uses a period as a unit and processes the audio to be processed based on the pitch shift coefficient of the audio segment where the period is located, so as to obtain pitch shifted audio.

[0103] In implementation, the specific implementation of processing the audio to be processed based on the pitch shift coefficients of multiple audio segments to obtain the pitch shifted audio may include:

[0104] The audio to be processed is sampled to obtain a discrete audio signal; an initial position corresponding to the maximum amplitude in each fundamental frequency period in the discrete audio signal is determined to obtain a plurality of initial positions; based on the pitch coefficient of the audio segment at each initial position, the sub-audio segment corresponding to each initial position is processed in sequence to obtain a plurality of pitch-shifted sub-audio segments, wherein the sub-audio segment corresponding to each initial position is an audio segment between a previous initial position and a next initial position adjacent to the initial position; a weight coefficient set of each pitch-shifted sub-audio segment is obtained, and based on the weight coefficient set of each pitch-shifted sub-audio segment, a plurality of pitch-shifted sub-audio segments are superimposed to obtain pitch-shifted audio.

[0105] As an example, the audio to be processed can be a continuous audio waveform, the horizontal axis of the audio waveform is time, and the vertical axis is amplitude. After discretizing the audio waveform, a discrete audio signal is obtained. In the discrete audio signal, the amplitude can be called the sampling value of the sampling point, and the initial position corresponding to the maximum amplitude in each fundamental frequency period is the sampling point with the largest sampling value in each fundamental frequency period.

[0106] In practical applications, for the convenience of processing, the audio to be processed can be sampled to obtain a discrete audio signal, and the pitch period of each audio segment is determined in step 104. Here, the initial position corresponding to the maximum amplitude in each pitch period can be determined, and then based on the pitch coefficient of the audio segment at each initial position, the audio segment between the previous initial position and the next initial position of each initial position (i.e., the pitch-shifted sub-audio segment) is processed to obtain multiple pitch-shifted sub-audio segments. Since there are overlapping parts between the multiple pitch-shifted sub-audio segments, if they are directly added, the pitch-shifted audio obtained may be distorted or not smooth. Therefore, the multiple pitch-shifted sub-audio segments can be weighted and superimposed, and the audio obtained after the weighted superposition is determined as the pitch-shifted audio. In addition, since the pitch-shifted sub-audio segment includes multiple sampling points, and each sampling point is provided with a weight coefficient, a weight coefficient set of each pitch-shifted sub-audio segment can be obtained, and the multiple pitch-shifted sub-audio segments are superimposed based on the weight coefficient set.

[0107] In some embodiments, the audio to be processed may be sampled at a preset sampling frequency to obtain a discrete audio signal. Figure 3 , Figure 3 It is a schematic diagram of an audio signal segment in a discrete audio signal provided by an embodiment of the present application.

[0108] It should be noted that the preset sampling frequency can be set by the user according to actual needs, or can be set by the device by default, and the embodiments of the present application do not limit this. For example, the preset sampling frequency can be 11025Hz, 22050Hz, 24000Hz, 44100Hz, 48000Hz, etc.

[0109] In some embodiments, after sampling to obtain a discrete audio signal, the discrete audio signal can be divided into multiple audio segments according to multiple audio segments of the audio to be processed, and then each audio segment can be periodically divided according to the pitch period of each audio segment, so that the discrete audio signal can be divided into multiple periodic signals, and then the sampling point with the largest sampling value in each periodic signal is determined, and the sampling point is determined as the initial position corresponding to the maximum amplitude. For example, continue to refer to Figure 3 , it can be determined Figure 3 It includes three fundamental frequency periods, and the initial position corresponding to the maximum amplitude in the first fundamental frequency period is the p1 sampling point, the initial position corresponding to the maximum amplitude in the second fundamental frequency period is the p2 sampling point, and the initial position corresponding to the maximum amplitude in the third fundamental frequency period is the p3 sampling point.

[0110] As an example, if the pitch shifting processing is performed on the audio segments in the pitch cycle of each initial position, the processing of audio segments in different pitch cycles is independent of each other, and there may be problems that the processed audio cannot be connected or the audio connection is not smooth. Therefore, the audio segment between the previous initial position and the next initial position of each initial position can be used as the sub-audio segment corresponding to each initial position. For example, see Figure 3 , assuming that the audio to be processed includes two audio segments, 0-X is one audio segment, XY is another audio segment, p2 is taken as the initial position to be processed, then the sub-audio segment corresponding to p2 is the audio segment between p1-p3, and p3 is taken as the initial position to be processed, then the sub-audio segment corresponding to p3 is the audio segment between p2-Y.

[0111] Furthermore, since the audio segment at each initial position is different, the pitch coefficient of the sub-audio segment corresponding to each initial position is different. Therefore, the sub-audio segment corresponding to each initial position can be pitch-shifted according to the pitch coefficient of the audio segment at each initial position to obtain multiple pitch-shifted sub-audio segments. Since there are overlapping parts between the multiple pitch-shifted sub-audio segments, and each pitch-shifted sub-audio segment includes multiple sampling points, the weight of the sampling value of each sampling point can be set, and then the multiple pitch-shifted sub-audio segments are weightedly superimposed to obtain pitch-shifted audio. The pitch-shifted audio is the audio obtained by changing the tone of the text to be processed.

[0112] In an embodiment of the present application, the audio to be processed is sampled and processed, and the audio to be processed is converted into a discrete audio signal. Then, the initial position corresponding to the maximum amplitude in each fundamental frequency period is determined, and the sub-audio segment corresponding to each initial position is subjected to pitch shifting processing according to the pitch shifting coefficient of the audio segment at each initial position to obtain a plurality of pitch-shifted sub-audio segments. The plurality of pitch-shifted sub-audio segments are weightedly superimposed to obtain pitch-shifted audio of the text to be processed after pitch shifting, thereby realizing the processing of pitch shifting of the text.

[0113] In some embodiments, based on the pitch shift coefficient of the audio segment at each initial position, the sub-audio segment corresponding to each initial position is processed in sequence to obtain multiple pitch-shifted sub-audio segments, and a specific implementation may include:

[0114] Determine an initial spacing between an initial position to be processed and a previous initial position, wherein the initial position to be processed is any one of a plurality of initial positions except the first initial position; determine a target spacing based on the initial spacing and a pitch coefficient of an audio segment at which the initial position to be processed is located; determine a target position to be processed based on the target spacing and a previous target position, wherein the previous target position is a new position corresponding to a maximum amplitude of a sub-audio segment corresponding to the previous initial position after processing; and move a sub-audio segment corresponding to the initial position to be processed based on a difference between the target position to be processed and the initial position to be processed to obtain a pitch-shifted sub-audio segment corresponding to the initial position to be processed.

[0115] In practical applications, since the spacing between the two initial positions can be understood as the fundamental pitch period of the audio segment in the audio to be processed, the pitch shifting processing of the audio can be considered as adjusting the frequency of the audio, and there is an inverse relationship between the frequency and the period. Therefore, the present application adjusts the fundamental pitch period by adjusting the spacing between the two initial positions, and can adjust the frequency of the sub-audio segment, thereby adjusting the pitch, and realizing the audio pitch shifting processing, and by adjusting the pitch of multiple sub-audio segments, the continuous adjustment of the pitch of the audio to be processed can be achieved, and the effect of changing the tone of the text to be processed can be achieved. In addition, in an embodiment of the present application, the pronunciation audio of a text may include a period of time, and the frequency of the pronunciation audio is usually unchanged at the beginning of this period of time, that is, no matter what the tone of the text is, the frequency of the first small part of its pronunciation audio may be the same, therefore, the initial position to be processed is any one of the multiple initial positions except the first initial position, that is, the sub-audio segment corresponding to the first initial position may not be adjusted.

[0116] In the embodiment of the present application, since the target position of the previous initial position is needed to determine the target position to be processed of the initial position to be processed, it is necessary to first process the sub-audio segment corresponding to the previous initial position, and then process the sub-audio segment corresponding to the initial position to be processed. That is, the present application processes the sub-audio segments in chronological order of the initial positions, starting from the sub-audio segment corresponding to the second initial position, and processing them in sequence until the sub-audio segment corresponding to the last initial position is adjusted, indicating that the tone change processing of the audio to be processed has been completed, and the tone change processing of the text to be processed is realized.

[0117] That is to say, the initial spacing between the ith initial position and the i-1th initial position can be determined, and then the target spacing can be determined based on the initial spacing and the pitch coefficient of the audio segment at the ith initial position. The ith target position can be determined based on the target spacing and the i-1th target position. Based on the difference between the ith target position and the ith initial position, the sub-audio segment corresponding to the ith initial position is moved to obtain the pitch-shifted sub-audio segment corresponding to the ith initial position. Wherein, i is a positive integer greater than 1, and when i is 2, the i-1th target position is the 1st initial position. In the embodiment of the present application, i starts to take values ​​from 2, and the sub-audio segment corresponding to each initial position is processed in turn to obtain the pitch-shifted sub-audio segment of each initial position.

[0118] As an example, the number of sampling points between the initial position to be processed and the last initial position can be determined as the initial spacing, and then the quotient of the initial spacing and the pitch coefficient of the audio segment where the initial position to be processed is located is determined as the target spacing, the sum of the last target position and the target spacing is determined as the target position to be processed, and the difference between the target position to be processed and the initial position to be processed is determined as the movement parameter of the sub-audio segment corresponding to the initial position to be processed, and the movement parameter includes a moving distance and a moving direction, and the sub-audio segment corresponding to the initial position to be processed is moved according to the movement parameter to obtain the pitch-shifted sub-audio segment corresponding to the initial position to be processed. Wherein, if the difference between the target position to be processed and the initial position to be processed is a negative number, the moving direction is the reverse direction of the time sequence, if the difference between the target position to be processed and the initial position to be processed is an integer, the moving direction is the positive direction of the time sequence, and if the difference between the target position to be processed and the initial position to be processed is 0, it means that the sub-audio segment corresponding to the initial position to be processed does not need to be moved.

[0119] For example, see Figure 4 , Figure 4 It is a schematic diagram of an audio pitch-shifting process provided by an embodiment of the present application. The waveform at the top in the figure is the audio to be processed, and the audio at the bottom is the pitch-shifted audio. With p5 as the initial position to be processed, p4 as the previous initial position, and p6 as the next initial position, the sub-audio segment corresponding to p5 is the audio segment between p4-p6 in the figure. The initial spacing between p5 and p4 can be determined first, and then the initial spacing is divided by the pitch-shifting coefficient of the audio segment where p5 is located to obtain a target spacing, and the target spacing is the spacing between p4' and p5', and then based on the target spacing and p4', the target position p5' to be processed is determined, the difference between p5' and p5 is determined as the moving distance of the sub-audio segment corresponding to p5, and the sub-audio segment is moved to the right until the new position corresponding to the maximum amplitude of the sub-audio segment is position p5'.

[0120] For example, assuming that the initial position to be processed is sampling point 180 and the previous initial position is 120, the initial spacing can be determined to be 60. Assuming that the pitch coefficient of the audio segment at the initial position to be processed is 1.2, the target spacing can be determined to be 60÷1.2=50. Assuming that the previous target position is 110, the target position to be processed can be determined to be 110+50=160. Further, the movement parameter of the sub-audio segment corresponding to the initial position to be processed can be determined to be 160-180=-20. The sub-audio segment corresponding to the initial position to be processed needs to be moved in the direction of decreasing time sequence by 20 sampling points. After the movement, the new position corresponding to the maximum amplitude in the sub-audio segment corresponding to the initial position to be processed is sampling point 160.

[0121] In the embodiment of the present application, processing the sub-audio segment according to the pitch coefficient actually moves the sub-audio segment left and right, reduces or increases the distance between the two maximum amplitudes, and then reduces or increases the audio period, thereby achieving the effect of increasing or decreasing the frequency of the audio, thereby achieving a change in the pitch of the audio. By changing the pitch of multiple sub-audio segments, the tone of the text to be processed can be changed.

[0122] In some embodiments, the specific implementation of obtaining the weight coefficient set of each pitch-shifted sub-audio segment may include:

[0123] Determine that the weight coefficient of the amplitude at the target position to be processed in each pitch-shifted sub-audio segment is 1, wherein the target position to be processed is a new position corresponding to the maximum amplitude of the sub-audio segment after processing corresponding to the initial position to be processed, and the initial position to be processed is any one of the multiple initial positions except the first initial position; determine that the weight coefficients of the amplitude corresponding to the multiple sampling points between the target position to be processed and the previous target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence; determine that the weight coefficients of the amplitude corresponding to the multiple sampling points between the target position to be processed and the next target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence.

[0124] As an example, in order to make the obtained pitch-shifted audio smoother, a weight coefficient can be set for the amplitude of each sampling point in the pitch-shifted sub-audio segment, and the weight coefficient of the amplitude at the position corresponding to the maximum amplitude in each pitch-shifted sub-audio segment (target position to be processed) can be set to 1, the weight coefficients of the amplitude at multiple sampling points from the target position to be processed to the previous target position are linearly decayed to 0 in sequence, and the weight coefficients of the amplitude at multiple sampling points from the target position to be processed to the next target position are linearly decayed to 0 in sequence. In this way, since each previous target position is also the next target position of other target positions, for the overlapping audio segments, the weight coefficients of the amplitude at each sampling point are added up to 1 when superimposed, and therefore, a relatively smooth pitch-shifted audio can be obtained.

[0125] Exemplarily, assume that there are two adjacent pitch-shifted sub-audio segments A and B, and the two pitch-shifted sub-audio segments each include 19 sampling points, the pitch-shifted sub-audio segment A is located before the pitch-shifted sub-audio segment B, the sampling points of the pitch-shifted sub-audio segment A are A1-A21, and A1, A11 and A21 are three target positions, the sampling points of the pitch-shifted sub-audio segment B are B1-B21, and B1, B11 and B21 are three target positions, and the sampling points A11-A21 overlap with the sampling points B1-B11. For the pitch-shifted audio segment A, A11 is the target position to be processed, A1 is the previous target position, A21 is the next target position, it can be determined that the weight coefficient of the amplitude corresponding to the sampling point A11 is 1, the weight coefficient of the amplitude corresponding to the sampling points A1 and A21 is 0, the weight coefficients of the amplitude corresponding to the sampling points A2, A3, A4, A5, A6, A7, A8, A9, A10 are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 respectively, and similarly, the weight coefficients of the amplitude corresponding to the sampling points A12, A13, A14, A15, A16, A17, A18, A19, A20 are 0. The weight coefficients of the amplitudes are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8 and 0.9 respectively; for the pitch-shifted sub-audio segment B, taking B11 as the target position to be processed, B1 is the previous target position, and B21 is the next target position. It can be determined that the weight coefficient of the amplitude corresponding to the sampling point B11 is 1, the weight coefficients of the amplitudes corresponding to the sampling points B1 and B21 are 0, and the weight coefficients of the amplitudes corresponding to the sampling points B2, B3, B4, B5, B6, B7, B8, B9 and B10 are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8 and 0.9 respectively.

[0126] In the embodiment of the present application, a weight is set for the amplitude of each sampling point in the pitch-shifted sub-audio segment, and then the pitch-shifted sub-audio segments are superimposed based on the weight, so that smoother and more realistic pitch-shifted audio can be obtained.

[0127] In some embodiments, a specific implementation of superimposing multiple pitch-shifted sub-audio segments based on a weight coefficient set of each pitch-shifted sub-audio segment to obtain the pitch-shifted audio may include:

[0128] Based on the weight coefficient set of each pitch-shifted sub-audio segment, the amplitudes at the same sampling points in multiple pitch-shifted sub-audio segments are superimposed to obtain pitch-shifted audio.

[0129] In practical applications, for each pitch-shifted sub-audio segment, the weight coefficient of the amplitude corresponding to each sampling point in the pitch-shifted sub-audio segment can be obtained, and then the amplitude at each sampling point in each pitch-shifted sub-audio segment is multiplied by the corresponding weight coefficient to obtain the weighted amplitude value at each sampling point in each pitch-shifted sub-audio segment. The weighted amplitude values ​​at the same sampling points in multiple pitch-shifted audio segments are superimposed to obtain the amplitude at each sampling point, so that pitch-shifted audio can be obtained.

[0130] By placing the above method in audio editing software, the user only needs to input the audio to be processed and the desired target pitch, and the software can use the above method to perform pitch shifting on the audio and output the desired pitch shifted audio, without the user having to adjust the pitch shift coefficient, thus improving the user experience. In addition, the user can perform pitch shifting on the audio of the material to be processed in the material library through the above method, which can enrich the material library and provide sufficient material for generating ghost animal videos.

[0131] The audio pitch shifting method provided by the present application obtains the audio to be processed and the target pitch, divides the audio to be processed into multiple audio segments, determines the pitch period of each audio segment, and determines the initial pitch of each audio segment based on the pitch period of each audio segment; determines the pitch shift coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment; processes the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch-shifted audio. By performing audio pitch shifting processing using the above method, the user only needs to input the audio to be processed and the desired target pitch, and the pitch shifting processing of the audio to be processed can be automatically realized without user participation, and the interactive process is relatively simple, which improves the user experience, and does not require the execution of a large number of calculation operations, which reduces the amount of calculation and operation complexity, thereby reducing the time consumption of audio pitch shifting processing and improving the efficiency of audio pitch shifting processing.

[0132] Figure 5 A processing flow chart of another audio pitch shifting method provided by an embodiment of the present application is shown, which specifically includes the following steps:

[0133] Step 502: Acquire the audio material to be processed and the target tone, wherein the audio material to be processed includes the audio of the text to be processed.

[0134] Step 504: Determine the pronunciation start position and pronunciation end position of the text to be processed in the audio material to be processed.

[0135] Step 506: The audio segment between the pronunciation start position and the pronunciation end position in the audio material to be processed is determined as the audio to be processed.

[0136] It should be noted that the specific implementation of steps 502 to 506 can refer to the relevant description of step 102 above, which will not be repeated in this embodiment.

[0137] Step 508: Divide the audio to be processed into a plurality of audio segments, determine the pitch period of each audio segment, and determine the initial pitch of each audio segment based on the pitch period of each audio segment.

[0138] It should be noted that the specific implementation of step 508 can refer to the relevant description of the above step 104, which will not be repeated here in this embodiment.

[0139] Step 510: Acquire a target tone included in the target tone, and determine the pitch coefficient of the target tone based on a preset correspondence between the tone and the pitch coefficient.

[0140] Step 512: If there are multiple target pitches, determine the audio segment group corresponding to each pitch coefficient based on the number of multiple audio segments and the number of multiple pitch coefficients.

[0141] Step 514: Determine the pitch coefficient corresponding to the audio segment group to which each audio segment belongs as the pitch coefficient of the audio segment.

[0142] Step 516: If the number of the target pitch is one, determine the pitch coefficient of the target pitch as the pitch coefficient of each audio segment.

[0143] Step 518: Determine a normalization coefficient for each audio segment based on the initial pitch of each audio segment.

[0144] Step 520: Determine the pitch coefficient of each audio segment based on the pitch coefficient of each audio segment and the normalization coefficient of each audio segment.

[0145] It should be noted that the specific implementation of steps 510 to 520 can refer to the relevant description of the above step 106, which will not be repeated in this embodiment.

[0146] Step 522: Sampling the audio to be processed to obtain a discrete audio signal.

[0147] Step 524: Determine the initial position corresponding to the maximum amplitude in each pitch period in the discrete audio signal to obtain multiple initial positions.

[0148] Step 526: Determine the initial distance between the initial position to be processed and the previous initial position.

[0149] The to-be-processed initial position is any one of the multiple initial positions except the first initial position.

[0150] Step 528: Determine a target spacing based on the initial spacing and the pitch coefficient of the audio segment at the initial position to be processed.

[0151] Step 530: Determine the target position to be processed based on the target distance and the previous target position.

[0152] The previous target position is a new position corresponding to the maximum amplitude of the sub-audio segment corresponding to the previous initial position after processing.

[0153] Step 532: Based on the difference between the target position to be processed and the initial position to be processed, the sub-audio segment corresponding to the initial position to be processed is moved to obtain the pitch-shifted sub-audio segment corresponding to the initial position to be processed.

[0154] Step 534: Determine that the weight coefficient of the amplitude at the target position to be processed in each pitch-shifted sub-audio segment is 1.

[0155] The target position to be processed is a new position corresponding to the maximum amplitude of the sub-audio segment after processing corresponding to the initial position to be processed, and the initial position to be processed is any one of the multiple initial positions except the first initial position.

[0156] Step 536: Determine that the weight coefficients of the amplitudes corresponding to multiple sampling points between the target position to be processed and the previous target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence, and determine that the weight coefficients of the amplitudes corresponding to multiple sampling points between the target position to be processed and the next target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence.

[0157] Step 538: Based on the weight coefficient set of each pitch-shifted sub-audio segment, the amplitudes at the same sampling point in multiple pitch-shifted sub-audio segments are superimposed to obtain pitch-shifted audio.

[0158] It should be noted that the specific implementation of steps 521 to 538 can refer to the relevant description of the above step 108, which will not be repeated in this embodiment.

[0159] The audio pitch shifting method provided by the present application obtains the audio to be processed and the target pitch, divides the audio to be processed into multiple audio segments, determines the pitch period of each audio segment, and determines the initial pitch of each audio segment based on the pitch period of each audio segment; determines the pitch shift coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment; processes the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch-shifted audio. By performing audio pitch shifting processing using the above method, the user only needs to input the audio to be processed and the desired target pitch, and the pitch shifting processing of the audio to be processed can be automatically realized without user participation, and the interactive process is relatively simple, which improves the user experience, and does not require the execution of a large number of calculation operations, which reduces the amount of calculation and operation complexity, thereby reducing the time consumption of audio pitch shifting processing and improving the efficiency of audio pitch shifting processing.

[0160] Corresponding to the above method embodiment, the present application also provides an audio pitch shifting device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of an audio pitch-shifting device provided by an embodiment of the present application. Figure 6 As shown, the device comprises:

[0161] An acquisition module 602 is configured to acquire the audio to be processed and the target tone;

[0162] A first determination module 604 is configured to divide the audio to be processed into a plurality of audio segments, determine a pitch period of each audio segment, and determine an initial pitch of each audio segment based on the pitch period of each audio segment;

[0163] A second determination module 606 is configured to determine a pitch change coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment;

[0164] The processing module 608 is configured to process the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain the pitch shifted audio.

[0165] Optionally, the second determining module 606 is configured to:

[0166] Acquire a target tone included in the target tone, and determine the pitch coefficient of the target tone based on a preset correspondence between the tone and the pitch coefficient;

[0167] Determining a pitch coefficient for each audio segment based on the pitch coefficient of the target pitch;

[0168] determining a normalization coefficient for each audio segment based on an initial pitch of each audio segment;

[0169] A pitch shift coefficient of each audio segment is determined based on the pitch coefficient of each audio segment and the normalization coefficient of each audio segment.

[0170] Optionally, if the number of target tones is one, the second determining module 606 is configured to:

[0171] The pitch coefficient of the target pitch is determined as the pitch coefficient of each audio segment.

[0172] Optionally, if there are multiple target tones, the second determination module 606 is configured to:

[0173] Determine, based on the number of the plurality of audio segments and the number of the plurality of pitch coefficients, a grouping of the audio segments corresponding to each pitch coefficient;

[0174] The pitch coefficient corresponding to the audio segment group to which each audio segment belongs is determined as the pitch coefficient of the audio segment.

[0175] Optionally, the processing module 608 is configured to:

[0176] Sampling the audio to be processed to obtain a discrete audio signal;

[0177] Determine an initial position corresponding to a maximum amplitude in each pitch period of the discrete audio signal to obtain a plurality of initial positions;

[0178] Based on the pitch shift coefficient of the audio segment at each initial position, the sub-audio segment corresponding to each initial position is processed in sequence to obtain multiple pitch shifted sub-audio segments, wherein the sub-audio segment corresponding to each initial position is an audio segment between a previous initial position and a next initial position adjacent to the initial position;

[0179] A weight coefficient set of each pitch-shifted sub-audio segment is obtained, and multiple pitch-shifted sub-audio segments are superimposed based on the weight coefficient set of each pitch-shifted sub-audio segment to obtain pitch-shifted audio.

[0180] Optionally, the processing module 608 is configured to:

[0181] Determine an initial distance between the to-be-processed initial position and the last initial position, wherein the to-be-processed initial position is any one of the multiple initial positions except the first initial position;

[0182] Determining a target spacing based on the initial spacing and the pitch coefficient of the audio segment at the initial position to be processed;

[0183] Determine the target position to be processed based on the target distance and the previous target position, wherein the previous target position is a new position corresponding to the maximum amplitude of the sub-audio segment corresponding to the previous initial position after processing;

[0184] Based on the difference between the target position to be processed and the initial position to be processed, the sub-audio segment corresponding to the initial position to be processed is moved to obtain the pitch-shifted sub-audio segment corresponding to the initial position to be processed.

[0185] Optionally, the processing module 608 is configured to:

[0186] Determine that the weight coefficient of the amplitude at the target position to be processed in each pitch-shifted sub-audio segment is 1, wherein the target position to be processed is a new position corresponding to the maximum amplitude of the sub-audio segment after processing corresponding to the initial position to be processed, and the initial position to be processed is any one of the multiple initial positions except the first initial position;

[0187] Determine that the weight coefficients of the amplitudes corresponding to the plurality of sampling points between the target position to be processed and the previous target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence;

[0188] It is determined that the weight coefficients of the amplitudes corresponding to the plurality of sampling points between the target position to be processed and the next target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence.

[0189] Optionally, the processing module 608 is configured to:

[0190] Based on the weight coefficient set of each pitch-shifted sub-audio segment, the amplitudes at the same sampling points in multiple pitch-shifted sub-audio segments are superimposed to obtain pitch-shifted audio.

[0191] Optionally, the acquisition module 602 is configured to:

[0192] Obtaining the audio material to be processed, wherein the audio material to be processed includes the audio of the text to be processed;

[0193] Determine the pronunciation start position and pronunciation end position of the to-be-processed text in the to-be-processed material audio;

[0194] An audio segment between a pronunciation start position and a pronunciation end position in the audio material to be processed is determined as the audio to be processed.

[0195] The audio pitch shifting device provided by the present application obtains the audio to be processed and the target pitch, divides the audio to be processed into multiple audio segments, determines the pitch period of each audio segment, and determines the initial pitch of each audio segment based on the pitch period of each audio segment; determines the pitch shift coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment; processes the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch-shifted audio. By performing audio pitch shifting processing using the above method, the user only needs to input the audio to be processed and the target pitch to be obtained, and the pitch shifting processing of the audio to be processed can be automatically realized without user participation, and the interactive process is relatively simple, which improves the user experience, and does not require the execution of a large number of calculation operations, which reduces the amount of calculation and operation complexity, thereby reducing the time consumption of audio pitch shifting processing and improving the efficiency of audio pitch shifting processing.

[0196] The above is a schematic scheme of an audio pitch shifting device of this embodiment. It should be noted that the technical scheme of the audio pitch shifting device and the technical scheme of the above audio pitch shifting method belong to the same concept, and the details of the technical scheme of the audio pitch shifting device that are not described in detail can all be referred to the description of the technical scheme of the above audio pitch shifting method.

[0197] Figure 7 The structure block diagram of a computing device 700 provided according to an embodiment of the present application is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0198] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0199] In one embodiment of the present application, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0200] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0201] When the processor 720 executes the instruction, the steps of the audio pitch shifting method are implemented.

[0202] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned audio pitch shifting method belong to the same concept, and the details of the technical scheme of the computing device not described in detail can all be referred to the description of the technical scheme of the above-mentioned audio pitch shifting method.

[0203] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the audio pitch shifting method as described above.

[0204] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned audio pitch shifting method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned audio pitch shifting method.

[0205] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0206] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0207] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0208] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0209] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. An audio pitch shifting method, characterized in that: include: Get the audio to be processed and the target tone; Dividing the audio to be processed into a plurality of audio segments, determining a pitch period of each audio segment, and determining an initial pitch of each audio segment based on the pitch period of each audio segment; Determining a pitch change coefficient corresponding to each audio segment based on the target pitch and an initial pitch of each audio segment; Processing the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch shifted audio; The step of determining the pitch change coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment includes: Acquire a target tone included in the target tone, and determine a pitch coefficient of the target tone based on a preset correspondence between the tone and the pitch coefficient; Determining a pitch coefficient of each audio segment based on the pitch coefficient of the target pitch; determining a normalization coefficient for each audio segment based on an initial pitch of each audio segment; A pitch shift coefficient of each audio segment is determined based on the pitch coefficient of each audio segment and the normalization coefficient of each audio segment.

2. The method according to claim 1, characterized in that If the number of the target pitch is one, determining the pitch coefficient of each audio segment based on the pitch coefficient of the target pitch includes: The pitch coefficient of the target pitch is determined as the pitch coefficient of each audio segment.

3. The method according to claim 1, characterized in that If there are multiple target tones, determining a pitch coefficient of each audio segment based on the pitch coefficient of the target tones includes: Determining, based on the number of the plurality of audio segments and the number of the plurality of pitch coefficients, a grouping of audio segments corresponding to each pitch coefficient; The pitch coefficient corresponding to the audio segment group to which each audio segment belongs is determined as the pitch coefficient of the audio segment.

4. The method according to claim 1, characterized in that The method further comprises: processing the audio to be processed based on the pitch-shift coefficients of the plurality of audio segments to obtain pitch-shifted audio. Sampling the audio to be processed to obtain a discrete audio signal; Determine an initial position corresponding to a maximum amplitude in each pitch period of the discrete audio signal to obtain a plurality of initial positions; Based on the pitch shift coefficient of the audio segment at each initial position, the sub-audio segment corresponding to each initial position is processed in sequence to obtain a plurality of pitch shifted sub-audio segments, wherein the sub-audio segment corresponding to each initial position is an audio segment between a previous initial position and a next initial position adjacent to the initial position; A weight coefficient set of each pitch-shifted sub-audio segment is obtained, and the multiple pitch-shifted sub-audio segments are superimposed based on the weight coefficient set of each pitch-shifted sub-audio segment to obtain the pitch-shifted audio.

5. The method according to claim 4, characterized in that Based on the pitch shift coefficient of the audio segment at each initial position, the sub-audio segment corresponding to each initial position is processed in sequence to obtain multiple pitch shifted sub-audio segments, including: Determine an initial distance between an initial position to be processed and a previous initial position, wherein the initial position to be processed is any one of the multiple initial positions except the first initial position; Determining a target spacing based on the initial spacing and the pitch coefficient of the audio segment at the initial position to be processed; Determining a target position to be processed based on the target distance and a previous target position, wherein the previous target position is a new position corresponding to the maximum amplitude of the sub-audio segment corresponding to the previous initial position after processing; Based on the difference between the target position to be processed and the initial position to be processed, the sub-audio segment corresponding to the initial position to be processed is moved to obtain the pitch-shifted sub-audio segment corresponding to the initial position to be processed.

6. The method according to claim 4, characterized in that Get the weight coefficient set of each pitch-shifted audio clip, including: Determine that the weight coefficient of the amplitude at the target position to be processed in each pitch-shifted sub-audio segment is 1, wherein the target position to be processed is a new position corresponding to the maximum amplitude of the sub-audio segment after processing corresponding to the initial position to be processed, and the initial position to be processed is any one of the multiple initial positions except the first initial position; Determine that the weight coefficients of the amplitudes corresponding to the plurality of sampling points between the target position to be processed and the previous target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence; Determine that the weight coefficients of the amplitudes corresponding to the plurality of sampling points between the target position to be processed and the next target position in each pitch-shifted sub-audio segment are linearly decayed to 0 in sequence.

7. The method according to claim 4, characterized in that The method of superimposing the plurality of pitch-shifted sub-audio segments based on a weight coefficient set of each pitch-shifted sub-audio segment to obtain the pitch-shifted audio comprises: Based on the weight coefficient set of each pitch-shifted sub-audio segment, the amplitudes at the same sampling points in the multiple pitch-shifted sub-audio segments are superimposed to obtain the pitch-shifted audio.

8. The method according to any one of claims 1 to 7, characterized in that Get the audio to be processed, including: Acquire the audio material to be processed, wherein the audio material to be processed includes the audio of the text to be processed; Determine the pronunciation start position and pronunciation end position of the to-be-processed text in the to-be-processed material audio; An audio segment between the pronunciation start position and the pronunciation end position in the material audio to be processed is determined as the audio to be processed.

9. An audio pitch-shifting device, characterized in that: include: An acquisition module, configured to acquire the audio to be processed and the target tone; A first determination module is configured to divide the audio to be processed into a plurality of audio segments, determine a pitch period of each audio segment, and determine an initial pitch of each audio segment based on the pitch period of each audio segment; A second determination module is configured to determine a pitch change coefficient corresponding to each audio segment based on the target pitch and an initial pitch of each audio segment; A processing module, configured to process the audio to be processed based on the pitch shift coefficients of the multiple audio segments to obtain pitch shifted audio; The step of determining the pitch change coefficient corresponding to each audio segment based on the target pitch and the initial pitch of each audio segment includes: Acquire a target tone included in the target tone, and determine a pitch coefficient of the target tone based on a preset correspondence between the tone and the pitch coefficient; Determining a pitch coefficient of each audio segment based on the pitch coefficient of the target pitch; determining a normalization coefficient for each audio segment based on an initial pitch of each audio segment; A pitch shift coefficient of each audio segment is determined based on the pitch coefficient of each audio segment and the normalization coefficient of each audio segment.

10. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer program product comprising computer instructions, characterized in that: When the computer instruction is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images