Audio synthesis method, computer equipment, readable storage medium and program product

Abstract

The invention relates to an audio synthesis method, computer equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring a target audio signal of a target tone, and acquiring melody feature information of a target melody; according to the melody duration and the duration of the target audio signals, splicing the plurality of target audio signals to obtain a spliced timbre audio signal, and according to the melody tone, performing tone modification processing on the spliced timbre audio signal to obtain an adjusted timbre audio signal; according to the melody duration, adjusting the duration included in the at least one wave seal characteristic parameter of the target audio signal to obtain an adjusted wave seal characteristic parameter; the wave seal characteristic parameter of the target audio signal is used for reflecting the change of the amplitude of the target audio signal within the duration of the wave seal characteristic parameter; and generating a target audio according to the adjusted tone audio signal and the adjusted wave seal characteristic parameter. By adopting the method, the flexibility of audio synthesis can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to an audio synthesis method, a computer device, a readable storage medium and a program product. Background Art

[0002] With the development of audio technology, audio synthesis has been widely used in many fields such as modern music creation, film soundtracks, and speech synthesis of smart devices. Audio synthesis technology mainly simulates and generates sounds through computers to meet the needs of audio in different scenarios. Especially in music production, audio synthesis can provide music creators with a rich selection of timbres without relying on the performance of real instruments. Therefore, audio synthesis technology can greatly expand the possibilities of creation, reduce production costs, and improve work efficiency.

[0003] At present, audio synthesis methods mainly include soft sound source rendering methods and sampler rendering methods. The soft sound source rendering method performs audio synthesis through a software-based audio synthesizer. Soft sound source rendering can synthesize audio through digital signal processing technologies such as frequency modulation synthesis, wavetable synthesis, and subtractive synthesis. The sampler rendering method performs audio synthesis through an audio synthesizer based on recorded samples, generates sound by pre-playing recorded audio samples, and adjusts the pitch, time stretches, filters, and other processing of the samples. However, these current audio synthesis methods all rely on huge sound source sampling data to complete the synthesis of audio, resulting in low flexibility in audio synthesis. Summary of the invention

[0004] Based on this, it is necessary to provide an audio synthesis method, a computer device, a readable storage medium and a program product that can improve the flexibility of audio synthesis in response to the above technical problems.

[0005] In a first aspect, the present application provides an audio synthesis method, comprising:

[0006] Acquire a target audio signal of a target timbre, and acquire melody feature information of a target melody, wherein the melody feature information includes melody duration and melody pitch;

[0007] According to the duration of the melody and the duration of the target audio signal, a plurality of the target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, and according to the melody pitch, the spliced ​​timbre audio signal is subjected to pitch shifting processing to obtain an adjusted timbre audio signal;

[0008] According to the melody duration, adjusting the duration included in at least one envelope characteristic parameter of the target audio signal to obtain an adjusted envelope characteristic parameter; the envelope characteristic parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope characteristic parameter;

[0009] A target audio signal is generated according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters; the target audio signal is a sound audio signal having the target timbre and conforming to the target melody.

[0010] In one embodiment, the step of splicing a plurality of the target audio signals to obtain a spliced ​​timbre audio signal includes:

[0011] Splicing the target audio signals head to tail and partially overlapping the two target audio signals before and after the splicing to obtain a spliced ​​audio signal with the same melody duration as the target melody;

[0012] For the overlapping part in the spliced ​​audio signal, the end part of the overlapping part belonging to the target audio signal is faded out, and the beginning part of the overlapping part belonging to the target audio signal is faded in to obtain the spliced ​​timbre audio signal.

[0013] In one embodiment, the pitch-changing processing is performed on the spliced ​​timbre audio signal according to the melody pitch to obtain the adjusted timbre audio signal, including:

[0014] Determining a resampling coefficient based on a pitch difference between the melody pitch and the pitch of the concatenated timbre audio signal;

[0015] The spliced ​​timbre audio signal is resampled according to the resampling coefficient to obtain the adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch, and the playback rate of the adjusted timbre audio signal is consistent with that of the spliced ​​timbre audio signal.

[0016] In one embodiment, the step of obtaining a target audio signal of a target timbre includes:

[0017] Acquire an original audio signal of the target timbre and a plurality of envelope characteristic parameters corresponding to the original audio signal; the envelope characteristic parameters of the original audio signal are used to reflect the change of the amplitude of the original audio signal within the duration of the envelope characteristic parameters;

[0018] According to the ratio between the amplitude change and the duration corresponding to each of the envelope characteristic parameters, determine, from the plurality of envelope characteristic parameters, an envelope characteristic parameter whose ratio is less than a preset ratio threshold as a target envelope characteristic parameter;

[0019] A target audio signal of the target timbre is generated according to the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal.

[0020] In one of the embodiments, generating the target audio signal of the target timbre according to the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal includes:

[0021] If the average amplitude of the audio signal segment is less than a preset amplitude threshold, amplifying the audio signal segment to obtain a target audio signal of the target timbre;

[0022] If the amplitude fluctuation frequency of the audio signal segment is greater than a preset frequency threshold, the audio signal segment is compressed to obtain a target audio signal of the target timbre.

[0023] In one of the embodiments, adjusting the duration of at least one envelope characteristic parameter of the target audio signal according to the melody duration to obtain an adjusted envelope characteristic parameter includes:

[0024] Determining a sum of durations included in the at least one envelope characteristic parameter;

[0025] According to the duration ratio between the melody duration and the sum of the duration durations, the duration duration included in the at least one envelope characteristic parameter is scaled so that the sum of the duration durations included in the at least one envelope characteristic parameter after scaling is equal to the melody duration, thereby obtaining the adjusted envelope characteristic parameter.

[0026] In one embodiment, generating the target audio according to the adjusted timbre audio signal and the adjusted envelope characteristic parameter comprises:

[0027] The amplitude of the adjusted timbre audio signal and the amplitude included in the adjusted envelope characteristic parameter are multiplied point by point at each time point to generate the target audio.

[0028] In a second aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0029] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0030] In a fourth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0031] The above-mentioned audio synthesis method, device, computer equipment, computer-readable storage medium and computer program product obtain a target audio signal of a target timbre, and obtain melody feature information of a target melody, wherein the melody feature information includes melody duration and melody pitch; according to the melody duration and the duration of the target audio signal, a plurality of target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, and the spliced ​​timbre audio signal is subjected to pitch-shifting processing according to the melody pitch to obtain an adjusted timbre audio signal; according to the melody duration, the duration included in at least one envelope feature parameter of the target audio signal is adjusted to obtain an adjusted envelope feature parameter, wherein the envelope feature parameter of the target audio signal is used to reflect the change in the amplitude of the target audio signal within the duration of the envelope feature parameter; according to the adjusted timbre audio signal and the adjusted envelope feature parameter, a target audio is generated, wherein the target audio is sound audio having a target timbre and conforming to the target melody. By splicing and pitch-shifting the target audio signal, the target audio signal can be flexibly adjusted according to the melody feature information of the target melody. At the same time, the duration included in at least one envelope feature parameter of the target audio signal can be flexibly adjusted according to the melody duration of the target melody. Finally, based on the adjusted timbre audio and the adjusted envelope feature parameters, the target audio with the target timbre and conforming to the target melody can be synthesized. There is no need to rely on huge sound source sampling data. Only a small amount of target audio signal of the target timbre is needed to synthesize the target audio that meets any duration and pitch requirements, thereby improving the flexibility of audio synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A diagram of an application environment of an audio synthesis method in an embodiment;

[0034] Figure 2 is a schematic flow chart of an audio synthesis method in an embodiment;

[0035] Figure 3 is a schematic diagram of an envelope model in one embodiment;

[0036] Figure 4 is a schematic diagram of an envelope line in another embodiment;

[0037] Figure 5 is a flow chart of another audio synthesis method in one embodiment;

[0038] Figure 6 is a structural block diagram of an audio synthesis device in one embodiment;

[0039] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The audio synthesis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains the target audio signal of the target timbre, and obtains the melody feature information of the target melody, and the melody feature information includes the melody duration and the melody pitch; the terminal 102 splices several target audio signals according to the melody duration and the duration of the target audio signal to obtain a spliced ​​timbre audio signal, and performs pitch change processing on the spliced ​​timbre audio signal according to the melody pitch to obtain an adjusted timbre audio signal; the terminal 102 adjusts the duration included in at least one envelope feature parameter of the target audio signal according to the melody duration to obtain an adjusted envelope feature parameter; the envelope feature parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope feature parameter; the terminal 102 generates the target audio according to the adjusted timbre audio signal and the adjusted envelope feature parameter; the target audio is a sound audio with a target timbre and conforming to the target melody. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0042] In an exemplary embodiment, Figure 2As shown, an audio synthesis method is provided, which is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation, including:

[0043] Step S202: acquiring a target audio signal of a target timbre, and acquiring melody feature information of a target melody.

[0044] The melody feature information includes melody duration and melody pitch.

[0045] The target timbre can refer to the timbre of any sound-producing body, such as the timbre of any musical instrument such as a violin or piano, or the timbre of any animal such as a dog or cat, or the timbre of a certain person. The target timbre is the specific timbre that is expected to be simulated or synthesized during the audio synthesis process.

[0046] The target audio signal of the target timbre may be an audio signal synthesized by a synthesizer or an actually recorded audio signal, for example, an audio signal of a violin timbre directly synthesized by a synthesizer or an audio signal recorded while a violin is being played.

[0047] The embodiments of the present application can be applied to synthesize high-quality target audio in scenarios where sound sources are lacking. Therefore, in a specific implementation, the duration of the target audio signal of the target timbre can be a shorter duration such as 1 second, 2 seconds, or 5 seconds.

[0048] Taking the target timbre as the timbre of a musical instrument as an example, the target audio signal may be an audio signal generated by the instrument playing any note for several seconds. In other words, the target audio signal may only include the sound of one tone emitted by any sound-emitting body.

[0049] The target melody may refer to the expected melody structure in the audio to be synthesized. The melody feature information may refer to detailed description information of the melody, including pitch (frequency of notes), melody pitch, melody duration, duration (duration of each note), rhythm (time interval between notes), and other information.

[0050] Step S204, according to the duration of the melody and the duration of the target audio signal, a plurality of target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, and the spliced ​​timbre audio signal is subjected to pitch shifting processing according to the melody pitch to obtain an adjusted timbre audio signal.

[0051] In a specific implementation, the terminal can splice several target audio signals according to the melody duration to obtain a spliced ​​timbre audio signal, so that the audio duration of the spliced ​​timbre audio signal is consistent with the melody duration of the target melody. Optionally, when splicing several target audio signals, the several target audio signals may not overlap each other, or may overlap to a certain extent at the beginning and end splicing. For example, if the melody duration of the target melody is 15 seconds and the duration of the target audio signal is equal to 5 seconds, at least 3 target audio signals are required for splicing.

[0052] In a specific implementation, the terminal performs pitch-shifting processing on the spliced ​​timbre audio signal according to the melody pitch, which may be by changing the frequency of the spliced ​​timbre audio signal to adjust the pitch of the spliced ​​timbre audio signal to obtain an adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch of the target melody. Optionally, the pitch-shifting processing may include time domain method, frequency domain method and parametric method, among which the time method may be to change the pitch by resampling technology without changing the playback speed; the frequency domain method may be to analyze the frequency components of the audio by Fourier transform, adjust the pitch and then perform an inverse transform.

[0053] Step S206: adjusting the duration included in at least one envelope characteristic parameter of the target audio signal according to the duration of the melody to obtain an adjusted envelope characteristic parameter.

[0054] The envelope characteristic parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope characteristic parameter.

[0055] The envelope can refer to the amplitude profile of an audio signal, which is composed of a series of peaks and valleys. The envelope can be regarded as an envelope, which is the outer contour of the amplitude of an audio signal changing over time. Envelope characteristic parameters are parameters that can quantify the characteristics of the envelope, and may include amplitude, duration, etc.

[0056] For example, for a specific target timbre, a corresponding envelope model (ADSR model) can be constructed. Different timbres and different fundamental frequencies have different harmonic laws in the frequency domain, which is the basic basis for human hearing to distinguish different musical instruments. When synthesizing sound, if the fundamental frequency and harmonic laws of the instrument in the frequency domain and the curve change law of the envelope model in the time domain can be restored, then audio that simulates the timbre of the instrument can be synthesized.

[0057] In order to facilitate the understanding of those skilled in the art, Figure 3An exemplary schematic diagram of an envelope model is provided. The envelope model is used to describe the characteristics of the amplitude of an audio signal changing over time, and may include four stages of the sound produced by the sound-producing body: attack (A), decay (D), sustain (S), and release (R). When a musical instrument produces sound, its characteristics change over time, including periodic and non-periodic components. In the initial stage, the energy usually increases, i.e. Figure 3 In the A phase, the sound gradually increases, and this part contains a large number of non-periodic components distributed throughout the frequency range. After the attack (A) phase, the sound will stabilize (D phase) and reach a stable phase (S phase) with a more or less periodic pattern. The S phase occupies most of the duration, and the audio energy may remain constant or gradually decrease during this phase. The last phase of the instrument's pronunciation is the R phase, when the sound gradually disappears. The envelope model can provide a highly consistent fit to the amplitude envelope of the tone produced by the instrument.

[0058] The envelope characteristic parameters of the target audio signal may include characteristic parameters of any pronunciation stage in the envelope model of the target audio signal. Since the change characteristics of the amplitude of the target audio signal within the duration of the envelope characteristic parameters are different in different pronunciation stages, the envelope characteristic parameters corresponding to different pronunciation stages are different. Exemplarily, the envelope characteristic parameters of the target audio signal may include the amplitude and duration of any pronunciation stage in the envelope model of the target audio signal. For example, at least one envelope characteristic parameter of the target audio signal may include the amplitude and duration of the A stage in the ADSR model, the amplitude and duration of the D stage in the ADSR model, the amplitude and duration of the S stage in the ADSR model, and the amplitude and duration of at least one of the amplitude and duration of the R stage in the ADSR model.

[0059] As an example, the upper envelope, lower envelope and amplitude envelope of the target audio signal can be calculated by using a sliding window on the waveform of the target audio signal and finding extreme values ​​in each sliding window to determine the envelope model of the target audio signal. The upper envelope is the upper boundary of the target audio signal; the lower envelope is the lower boundary of the target audio signal; and the amplitude envelope is the change curve of the amplitude of the target audio signal, which is the maximum value of the amplitude of the target audio signal.

[0060] Specifically, each sampling point of the target audio signal can be traversed, and based on a sliding window with a preset window length, the maximum amplitude, the maximum upper envelope and the minimum lower envelope in the current sliding window can be calculated, so that three envelopes can be obtained, namely, the amplitude envelope, the upper envelope and the lower envelope.

[0061] Further optionally, the four pronunciation stages of the envelope model can be represented by w and f(x) respectively, where w represents the proportion of the duration of the pronunciation stage to the total duration, and f(x) can be used to represent the function fitting relationship of the pronunciation stage. Exemplarily, the amplitude range recorded in the envelope model is between 0 and 1, or between -1 and 1.

[0062] In a specific implementation, adjusting the duration included in at least one envelope characteristic parameter of the target audio signal to obtain the adjusted envelope characteristic parameter may include adjusting the duration included in each envelope characteristic parameter of the target audio signal, and using the adjusted envelope characteristic parameters as a whole as the adjusted envelope characteristic parameter; or, may include adjusting the duration included in one envelope characteristic parameter of the target audio signal, and using the individually adjusted envelope characteristic parameter as the adjusted envelope characteristic parameter. The duration included in the adjusted envelope characteristic parameter is equal to the melody duration of the target melody.

[0063] Optionally, the duration included in each envelope characteristic parameter may be adjusted in proportion so that the sum of the durations included in each envelope characteristic parameter after adjustment is equal to the melody duration of the target melody; or, the duration included in one of the envelope characteristic parameters may be adjusted so that the duration included in the adjusted envelope characteristic parameter is equal to the melody duration of the target melody.

[0064] The adjusted envelope characteristic parameters can match the duration requirement of the melody and reflect the natural change process of the sound, ensuring that the sound not only has pitch and timbre, but also has real dynamic performance.

[0065] Step S208: generating target audio according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters.

[0066] The target audio is a sound audio having a target timbre and conforming to a target melody.

[0067] The adjusted timbre audio signal provides frequency information (pitch, timbre), and the adjusted envelope characteristic parameters provide amplitude change information (dynamic characteristics). The combination of the two can ensure that the generated target audio not only has the correct timbre and pitch, but also has real and natural dynamic fluctuations. In a specific implementation, the terminal generates the target audio according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters. The target audio can be generated by multiplying the adjusted timbre audio signal and the adjusted envelope characteristic parameters point by point, so that the amplitude of the adjusted timbre audio signal is adjusted by the adjusted envelope characteristic parameters at each time point, thereby generating the target audio with natural dynamic performance.

[0068] The embodiment of the present application is based on a small amount of sound source sampling data, simulates the time domain characteristics of the target timbre through envelope characteristic parameters, and through splicing and pitch shifting processing, proposes a lightweight and freely variable timbre synthesis method that combines pitch migration and time scaling. It only needs the sound-emitting body of the target timbre to emit a sound or play a sound. By collecting this sound, the target timbre can be used to render target audio of different durations and natural listening.

[0069] In the above-mentioned audio synthesis method, a target audio signal of a target timbre is obtained, and melody feature information of a target melody is obtained, wherein the melody feature information includes a melody duration and a melody pitch; according to the melody duration and the duration of the target audio signal, a plurality of target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, and the spliced ​​timbre audio signal is subjected to pitch-shifting processing according to the melody pitch to obtain an adjusted timbre audio signal; according to the melody duration, the duration included in at least one envelope feature parameter of the target audio signal is adjusted to obtain an adjusted envelope feature parameter, wherein the envelope feature parameter of the target audio signal is used to reflect the change in the amplitude of the target audio signal within the duration of the envelope feature parameter; according to the adjusted timbre audio signal and the adjusted envelope feature parameter, a target audio is generated, wherein the target audio is sound audio having a target timbre and conforming to the target melody. By splicing and pitch-shifting the target audio signal, the target audio signal can be flexibly adjusted according to the melody feature information of the target melody. At the same time, the duration included in at least one envelope feature parameter of the target audio signal can be flexibly adjusted according to the melody duration of the target melody. Finally, based on the adjusted timbre audio and the adjusted envelope feature parameters, the target audio with the target timbre and conforming to the target melody can be synthesized. There is no need to rely on huge sound source sampling data. Only a small amount of target audio signal of the target timbre is needed to synthesize the target audio that meets any duration and pitch requirements, thereby improving the flexibility of audio synthesis.

[0070] In another embodiment, a plurality of target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, including: splicing the plurality of target audio signals head to tail and partially overlapping the two target audio signals before and after the splicing to obtain a spliced ​​audio signal having the same melody duration as the target melody; for the overlapping part in the spliced ​​audio signal, fading out the end part of the overlapping part belonging to the target audio signal, and fading in the beginning part of the overlapping part belonging to the target audio signal, to obtain the spliced ​​timbre audio signal.

[0071] The terminal splices the multiple target audio signals head to tail, and the two target audio signals before and after the splicing have a certain degree of overlap at the beginning and the end.

[0072] Assuming that the melody duration of the target melody is greater than the duration of a single target audio signal, multiple target audio signals can be spliced ​​together, and then the beginning and ending parts can be faded in and out to make a smooth transition so that their duration is extended to the melody duration of the target melody. In this way, the audio duration of the generated spliced ​​audio signal can be consistent with the melody duration of the target melody, and the sound quality is natural, avoiding the audio fragmentation caused by direct splicing.

[0073] For the overlapping part generated when the beginning and the end of the spliced ​​audio signal are spliced, the overlapping part can be subjected to a fade-in and fade-out transition process. Specifically, the end part of the target audio signal in the overlapping part is subjected to a fade-out process, and the fade-out process makes the amplitude gradually decrease and eventually approach zero, ensuring that a smooth audio transition effect can be produced when the end part of any target audio signal transitions to the next target audio signal; the beginning part of the target audio signal in the overlapping part is subjected to a fade-in process, and the fade-in process makes the amplitude gradually increase from zero to a normal level, ensuring that the transition of the beginning part of any target audio signal when it is connected to the previous target audio signal is smoother and more natural.

[0074] The technical solution of this embodiment generates a spliced ​​timbre audio signal with a smooth transition by splicing several target audio signals head to tail and fading-in and -out the overlapping parts, thereby ensuring a natural connection between the target audio signals and avoiding abrupt volume changes. The audio duration of the finally generated spliced ​​timbre audio signal is consistent with the duration of the target melody, so that the audio length of the target audio signal can be adaptively changed to ensure that the finally synthesized target audio is consistent with the melody structure of the target melody, and fading-in and -out are performed on the overlapping parts head to tail, so that the finally synthesized target audio can have a more natural auditory effect.

[0075] In another embodiment, the spliced ​​timbre audio signal is pitch-shifted according to the melody pitch to obtain an adjusted timbre audio signal, including: determining a resampling coefficient according to the pitch difference between the melody pitch and the pitch of the spliced ​​timbre audio signal; resampling the spliced ​​timbre audio signal according to the resampling coefficient to obtain an adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch, and the playback rate of the adjusted timbre audio signal is consistent with that of the spliced ​​timbre audio signal.

[0076] The resampling coefficient is determined based on the pitch difference between the melody pitch and the pitch of the concatenated timbre audio signal. ,in, is the semitone difference between two tones.

[0077] Resampling is a technique that changes the sampling rate of an audio signal, by upsampling or downsampling the audio signal to change its pitch without affecting the playback speed. During the resampling process, the density of the sampling points of the audio signal changes, which directly affects the frequency (i.e. pitch) of the signal.

[0078] The resampling factor is the proportional coefficient used during resampling, which determines the change in the sampling rate of the audio signal. In the specific implementation, it is assumed that the resampling factor is , where P is the upsampling coefficient and Q is the downsampling coefficient. The terminal resamples the superimposed timbre audio according to the resampling coefficient. During the upsampling process, it inserts sampling points, which will make the fundamental frequency period become P times of the original one, and the spectrum is compressed to the original times, the duration becomes P times of the original. Therefore, the fundamental frequency will become times, the pitch will be reduced to the original times, the speech speed of the audio signal will be slowed down to the original Similarly, during downsampling, every This will cause the pitch cycle length to become the original times, the spectrum is expanded to Q times of the original, and the duration is shortened to the original Therefore, the fundamental frequency will increase to Q times of the original, the pitch will increase to Q times of the original, and the speech speed of the audio signal will increase to Q times of the original. Combining the above two processes, the speech speed of the audio signal can be changed to the original times, and get the output signal y(n), and then perform This way, the speech speed is normal, but the pitch is changed to the original times the final output speech z(n). In summary, after the audio passes After resampling by multiple times, the playback rate can be kept unchanged, but the speech speed and pitch of the signal will become the original times.

[0079] The technical solution of this embodiment resamples the spliced ​​timbre audio signal according to the resampling coefficient, and while adjusting the pitch of the spliced ​​timbre audio signal to be consistent with the melody pitch of the target melody, keeps the playback rate unchanged, thereby achieving pitch change without changing speed through resampling, thereby improving the naturalness, flexibility and efficiency of audio synthesis, and ensuring the accurate match of the generated audio with the expected melody.

[0080] In another embodiment, obtaining a target audio signal of a target timbre includes: obtaining an original audio signal of the target timbre and a plurality of envelope characteristic parameters corresponding to the original audio signal; the envelope characteristic parameters of the original audio signal are used to reflect the change in the amplitude of the original audio signal within the duration of the envelope characteristic parameters; according to the ratio between the amplitude change and the duration corresponding to each envelope characteristic parameter, determining from the plurality of envelope characteristic parameters an envelope characteristic parameter having a ratio less than a preset ratio threshold value as a target envelope characteristic parameter; and generating a target audio signal of the target timbre according to an audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal.

[0081] Optionally, the multiple envelope characteristic parameters corresponding to the original audio signal may include characteristic parameters of each pronunciation stage in the envelope model of the original audio signal, such as characteristic parameters of each pronunciation stage such as attack (A), decay (D), hold (S) and release (R) stages. The multiple envelope characteristic parameters corresponding to the original audio signal reflect the change pattern of the amplitude of the original audio signal in different time periods. For example, the amplitude changes dramatically in one pronunciation stage, while the amplitude is stable in another pronunciation stage. Therefore, the ratio between the amplitude change and the duration corresponding to each envelope characteristic parameter can reflect the severity of the amplitude change. A large ratio indicates that the amplitude changes dramatically and the sound fluctuates greatly; a small ratio indicates that the amplitude is stable and the sound is relatively stable. Optionally, the ratio between the amplitude change and the duration corresponding to the envelope characteristic parameter may refer to the ratio between the difference between the maximum amplitude and the minimum amplitude and the duration.

[0082] If the ratio is less than the preset ratio threshold, it means that the amplitude change reflected by the envelope characteristic parameter is relatively gentle, that is, the sound is relatively stable. Therefore, the envelope characteristic parameter with a ratio less than the preset ratio threshold is used as the target envelope characteristic parameter, and the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal is used as the target audio signal of the target timbre. Optionally, since the S stage of the envelope model is generally a stage with stable amplitude, the envelope characteristic parameter corresponding to the S stage is usually a target envelope characteristic parameter that meets the ratio less than the preset ratio threshold. Therefore, the audio signal segment corresponding to the S stage in the original audio signal is used as the target audio signal of the target timbre.

[0083] In order to facilitate the understanding of those skilled in the art, Figure 4 An exemplary schematic diagram of an envelope is provided. It can be seen that Figure 4In the envelope of the piano sound, the amplitude variation range of stage A is large, and the duration of the stage is short, so the ratio between the amplitude variation and the duration is large, and the envelope characteristic parameters of this stage are not suitable as target envelope characteristic parameters; while the amplitude variation range of stage S is medium, and the duration of the stage is long, so the ratio between the amplitude variation and the duration is small, and the envelope characteristic parameters of this stage are suitable as target envelope characteristic parameters. Therefore, the audio signal segment corresponding to the S stage can be intercepted from the envelope of the piano sound as the target audio signal of the target timbre, that is, the audio signal segment corresponding to the target envelope characteristic parameters can be used as the target audio signal of the target timbre.

[0084] For another example, in the envelope of the violin sound, the duration of the A stage is relatively long. If the ratio between the amplitude variation range of the A stage and the duration of the stage is less than the preset ratio threshold, then the envelope characteristic parameters of the A stage can also be used as the target envelope characteristic parameters; the duration of the S stage is relatively long, and the amplitude variation range is relatively small, so the envelope characteristic parameters of the S stage can also be used as the target envelope characteristic parameters. Therefore, the audio signal segments corresponding to the S stage and the A stage can be intercepted from the envelope of the violin as the target audio signal of the target timbre, that is, the audio signal segments corresponding to the target envelope characteristic parameters are used as the target audio signal of the target timbre.

[0085] The technical solution of this embodiment determines the audio signal segment corresponding to the target envelope characteristic parameter from the original timbre audio as the target audio signal of the target timbre, which is used in the subsequent audio processing steps (splicing and pitch shifting processing), and can ensure that the target audio signal processed subsequently is stable in sound and has the performance that meets the expected dynamic characteristics. The audio signal segment corresponding to the target envelope characteristic parameter has a relatively stable amplitude and a relatively stable sound, so it can provide a stable audio foundation in the audio synthesis process, which helps to generate a smoother and more coherent audio effect.

[0086] In another embodiment, a target audio signal of a target timbre is generated based on an audio signal segment corresponding to a target envelope characteristic parameter in an original audio signal, including: if the average amplitude of the audio signal segment is less than a preset amplitude threshold, amplifying the audio signal segment to obtain the target audio signal of the target timbre; if the amplitude fluctuation frequency of the audio signal segment is greater than a preset frequency threshold, compressing the audio signal segment to obtain the target audio signal of the target timbre.

[0087] If the average amplitude is lower than the amplitude threshold, it means that the average amplitude of the audio signal segment is not large enough and needs to be amplified. The amplification process can be to increase the amplitude of the audio signal segment as a whole so that the average amplitude can be higher than the preset amplitude threshold or as close to the maximum amplitude allowed as possible.

[0088] Among them, the amplitude fluctuation frequency can refer to the frequency of amplitude changes in a short period of time. If the amplitude fluctuation frequency of an audio signal segment is high, it means that the amplitude changes rapidly and the fluctuations are large, and the audio can clearly feel that the sound is unstable. Therefore, if the amplitude fluctuation frequency exceeds the preset frequency threshold, it means that the audio signal segment changes rapidly and the fluctuations are large, and compression processing is required. Compression processing can compress the amplitude of the audio signal segment with a high amplitude fluctuation frequency, making the audio signal more stable, reducing high-frequency amplitude fluctuations, and avoiding instability in the audio sense.

[0089] The technical solution of this embodiment, through amplification and compression processing, the amplitude of the audio signal segment is fully amplified and the dynamic range is controlled, so that the target audio signal of the target timbre can be full-amplitude audio, that is, the amplitude of the target audio signal fully utilizes the available dynamic range and can be close to the allowed maximum value, ensuring the loudness improvement and quality optimization of the target audio signal, and can have a smooth dynamic performance, avoiding the situation of unstable sound, which is helpful to improve the quality, clarity and stability of the subsequently generated target audio, and has a high-quality auditory effect.

[0090] In another embodiment, the duration included in at least one envelope characteristic parameter of the target audio signal is adjusted according to the melody duration to obtain the adjusted envelope characteristic parameter, including: determining the sum of the duration included in the at least one envelope characteristic parameter; scaling the duration included in the at least one envelope characteristic parameter according to the duration ratio between the melody duration and the sum of the duration, so that the sum of the duration included in the at least one envelope characteristic parameter after scaling is equal to the melody duration, thereby obtaining the adjusted envelope characteristic parameter.

[0091] Optionally, the sum of the durations included in at least one envelope characteristic parameter may refer to the time length from the beginning to the end of the ADSR model, that is, the time experienced by the complete amplitude change process of the target audio signal. The sum of the durations included in at least one envelope characteristic parameter may be equal to the sum of the durations of each stage of the envelope model (such as attack, decay, hold, and release).

[0092] Exemplarily, the sum of the durations included in at least one envelope characteristic parameter may be equal to the sum of the durations of the four stages, namely, the duration of the A stage, the duration of the D stage, the duration of the S stage, and the duration of the R stage in the ADSR model of the target audio signal. Assuming that the duration of the A stage is 1 second, the duration of the D stage is 2 seconds, the duration of the S stage is 5 seconds, and the duration of the R stage is 2 seconds, then the sum of the durations included in at least one envelope characteristic parameter of the target audio signal is equal to 1+2+5+2=10 seconds. If the melody duration of the target melody is 20 seconds, the duration ratio between the melody duration and the sum of the durations is 2. Then, the durations included in each envelope characteristic parameter may be scaled to twice the original value, so that the sum of the durations included in each scaled envelope characteristic parameter is equal to 2+4+10+4=20 seconds. That is, the duration of the adjusted envelope characteristic parameter is equal to 20 seconds, which is equal to the melody duration of the target melody.

[0093] Scaling the duration included in at least one envelope characteristic parameter according to the duration ratio may be performed by scaling the duration included in each envelope characteristic parameter according to the same ratio, thereby keeping the amplitude dynamic characteristics of at least one envelope characteristic parameter unchanged and only extending or compressing it in time.

[0094] At least one envelope characteristic parameter after scaling is jointly used as an adjusted envelope characteristic parameter, and the sum of the durations included in at least one envelope characteristic parameter after scaling is equal to the melody duration, that is, the duration of the adjusted envelope characteristic parameter is equal to the melody duration, ensuring that the amplitude dynamic characteristics of at least one envelope characteristic parameter can be accurately mapped to the time frame of the target melody.

[0095] In another embodiment, target audio is generated according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters, including: multiplying the amplitude of the adjusted timbre audio signal by the amplitude included in the adjusted envelope characteristic parameters point by point at each time point to generate the target audio.

[0096] Since the amplitude included in the adjusted envelope characteristic parameters can refer to the amplitude of any pronunciation stage of the envelope model, the amplitude range is usually between 0 and 1, or between -1 and 1. Therefore, the amplitude of the adjusted timbre audio signal is multiplied point by point with the amplitude included in the adjusted envelope characteristic parameters at each time point, which is equivalent to modulating the amplitude of the adjusted timbre audio signal by the amplitude included in the adjusted envelope characteristic parameters, ensuring that the amplitude of the adjusted timbre audio signal at each time point will be adjusted by the amplitude included in the adjusted envelope characteristic parameters, ensuring that the generated target audio not only has the correct pitch and duration, but also has natural amplitude dynamic changes, so that the generated target audio sounds more layered and realistic.

[0097] The technical solution of this embodiment, by multiplying the amplitude of the adjusted timbre audio signal with the amplitude included in the adjusted envelope characteristic parameters point by point at each time point, can use the target timbre to render target audio of different durations and tones with natural listening experience, thereby improving the quality of generated target audio.

[0098] In another embodiment, Figure 5 As shown, an audio synthesis method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0099] S502, obtaining a target audio signal of a target timbre, and obtaining melody feature information of a target melody.

[0100] The melody feature information includes melody duration and melody pitch.

[0101] In one of the embodiments, an original audio signal of a target timbre and a plurality of envelope characteristic parameters corresponding to the original audio signal are obtained; the envelope characteristic parameters of the original audio signal are used to reflect the change in the amplitude of the original audio signal within the duration of the envelope characteristic parameters; based on the ratio between the amplitude change and the duration corresponding to each envelope characteristic parameter, an envelope characteristic parameter having a ratio less than a preset ratio threshold is determined from the plurality of envelope characteristic parameters as a target envelope characteristic parameter; based on an audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal, a target audio signal of the target timbre is generated.

[0102] In one of the embodiments, a target audio signal of a target timbre is generated based on an audio signal segment corresponding to a target envelope characteristic parameter in an original audio signal, including: if the average amplitude of the audio signal segment is less than a preset amplitude threshold, amplifying the audio signal segment to obtain the target audio signal of the target timbre; if the amplitude fluctuation frequency of the audio signal segment is greater than a preset frequency threshold, compressing the audio signal segment to obtain the target audio signal of the target timbre.

[0103] S504, according to the melody duration and the duration of the target audio signal, multiple target audio signals are spliced ​​head to tail with the two target audio signals before and after the splicing partially overlapping to obtain a spliced ​​audio signal with the same melody duration as the target melody.

[0104] S506, for the overlapping part in the spliced ​​audio signal, fade out the end part of the overlapping part belonging to the target audio signal, and fade in the beginning part of the overlapping part belonging to the target audio signal to obtain a spliced ​​timbre audio signal.

[0105] S508, determining a resampling coefficient according to the pitch difference between the melody pitch and the pitch of the concatenated timbre audio signal.

[0106] S510, resampling the spliced ​​timbre audio signal according to the resampling coefficient to obtain an adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch, and the playback rate of the adjusted timbre audio signal is consistent with that of the spliced ​​timbre audio signal.

[0107] S512: Determine a sum of durations included in at least one envelope characteristic parameter.

[0108] The envelope characteristic parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope characteristic parameter.

[0109] S514, scaling the duration included in at least one envelope characteristic parameter according to the duration ratio between the melody duration and the sum of the duration durations, so that the sum of the duration durations included in the at least one envelope characteristic parameter after scaling is equal to the melody duration, thereby obtaining the adjusted envelope characteristic parameter.

[0110] S516: multiply the amplitude of the adjusted timbre audio signal and the amplitude included in the adjusted envelope characteristic parameters point by point at each time point to generate a target audio signal.

[0111] The target audio is a sound audio having a target timbre and conforming to a target melody.

[0112] It should be noted that the specific limitations of the above steps can refer to the specific limitations of an audio synthesis method mentioned above.

[0113] In practical applications, the embodiments of the present application can be well applied to the construction of high-quality sound source files in scenarios where sound sources are lacking. This construction can generate high-quality audio synthesis data in batches, and can be used as a data enhancement solution or the construction of the data set itself to support tasks such as timbre migration and audio domain generation tasks; moreover, it can realize the conversion from symbolic music representation to audio content, and therefore can be used as a downstream generation interface for symbolic music tasks; and it can generate high-quality audio at a lower computing cost; at the same time, it can realize fast sampling of a sound, for example, for the preservation or restoration of some endangered ethnic musical instruments that need protection, or for some interesting applications (ambient sounds, bird calls, etc.).

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides an audio synthesis device for implementing the audio synthesis method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more audio synthesis device embodiments provided below can refer to the limitations on the audio synthesis method above, and will not be repeated here.

[0116] In an exemplary embodiment, Figure 6 As shown, an audio synthesis device is provided, comprising:

[0117] The acquisition module 610 is used to acquire a target audio signal of a target timbre and acquire melody feature information of a target melody, wherein the melody feature information includes melody duration and melody pitch.

[0118] The audio adjustment module 620 is used to splice a plurality of the target audio signals according to the duration of the melody and the duration of the target audio signal to obtain a spliced ​​timbre audio signal, and to perform pitch shifting processing on the spliced ​​timbre audio signal according to the melody pitch to obtain an adjusted timbre audio signal.

[0119] The parameter adjustment module 630 is used to adjust the duration included in at least one envelope characteristic parameter of the target audio signal according to the melody duration to obtain an adjusted envelope characteristic parameter; the envelope characteristic parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope characteristic parameter.

[0120] The generating module 640 is used to generate a target audio signal according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters; the target audio signal is a sound audio signal having the target timbre and conforming to the target melody.

[0121] In one embodiment, the audio adjustment module 620 is specifically used to splice a plurality of the target audio signals end to end and partially overlap the two target audio signals before and after the splicing to obtain a spliced ​​audio signal with the same melody duration as the target melody; for the overlapping part in the spliced ​​audio signal, the ending part of the overlapping part belonging to the target audio signal is faded out, and the beginning part of the overlapping part belonging to the target audio signal is faded in to obtain the spliced ​​timbre audio signal.

[0122] In one embodiment, the audio adjustment module 620 is specifically used to determine a resampling coefficient based on the pitch difference between the melody pitch and the pitch of the spliced ​​timbre audio signal; resample the spliced ​​timbre audio signal according to the resampling coefficient to obtain the adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch, and the playback rate of the adjusted timbre audio signal is consistent with that of the spliced ​​timbre audio signal.

[0123] In one of the embodiments, the acquisition module 610 is specifically used to acquire the original audio signal of the target timbre and multiple envelope characteristic parameters corresponding to the original audio signal; the envelope characteristic parameters of the original audio signal are used to reflect the change in the amplitude of the original audio signal within the duration of the envelope characteristic parameters; according to the ratio between the amplitude change and the duration corresponding to each of the envelope characteristic parameters, the envelope characteristic parameter whose ratio is less than a preset ratio threshold is determined from the multiple envelope characteristic parameters as the target envelope characteristic parameter; according to the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal, the target audio signal of the target timbre is generated.

[0124] In one of the embodiments, the acquisition module 610 is specifically used to generate a target audio signal of the target timbre based on an audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal, including: if the average amplitude of the audio signal segment is less than a preset amplitude threshold, amplifying the audio signal segment to obtain the target audio signal of the target timbre; if the amplitude fluctuation frequency of the audio signal segment is greater than a preset frequency threshold, compressing the audio signal segment to obtain the target audio signal of the target timbre.

[0125] In one of the embodiments, the parameter adjustment module 630 is specifically used to determine the sum of the durations included in at least one envelope characteristic parameter; scale the duration included in at least one envelope characteristic parameter according to the duration ratio between the melody duration and the sum of the durations, so that the sum of the durations included in the scaled at least one envelope characteristic parameter is equal to the melody duration, thereby obtaining the adjusted envelope characteristic parameter.

[0126] In one embodiment, the generating module 640 is specifically configured to multiply the amplitude of the adjusted timbre audio signal by the amplitude included in the adjusted envelope characteristic parameter point by point at each time point to generate the target audio.

[0127] Each module in the above audio synthesis device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0128] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an audio synthesis method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0129] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0134] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0136] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An audio synthesis method, characterized in that: The method comprises: Acquire a target audio signal of a target timbre, and acquire melody feature information of a target melody, wherein the melody feature information includes melody duration and melody pitch; According to the duration of the melody and the duration of the target audio signal, a plurality of the target audio signals are spliced ​​to obtain a spliced ​​timbre audio signal, and according to the melody pitch, the spliced ​​timbre audio signal is subjected to pitch shifting processing to obtain an adjusted timbre audio signal; According to the melody duration, adjusting the duration included in at least one envelope characteristic parameter of the target audio signal to obtain an adjusted envelope characteristic parameter; the envelope characteristic parameter of the target audio signal is used to reflect the change of the amplitude of the target audio signal within the duration of the envelope characteristic parameter; A target audio signal is generated according to the adjusted timbre audio signal and the adjusted envelope characteristic parameters; the target audio signal is a sound audio signal having the target timbre and conforming to the target melody.

2. The method according to claim 1, characterized in that The step of splicing a plurality of target audio signals to obtain a spliced ​​timbre audio signal comprises: Splicing the target audio signals head to tail and partially overlapping the two target audio signals before and after the splicing to obtain a spliced ​​audio signal with the same melody duration as the target melody; For the overlapping part in the spliced ​​audio signal, the end part of the overlapping part belonging to the target audio signal is faded out, and the beginning part of the overlapping part belonging to the target audio signal is faded in to obtain the spliced ​​timbre audio signal.

3. The method according to claim 1, characterized in that The step of performing pitch-shifting processing on the spliced ​​timbre audio signal according to the melody pitch to obtain an adjusted timbre audio signal includes: Determining a resampling coefficient based on a pitch difference between the melody pitch and the pitch of the concatenated timbre audio signal; The spliced ​​timbre audio signal is resampled according to the resampling coefficient to obtain the adjusted timbre audio signal, so that the pitch of the adjusted timbre audio signal is consistent with the melody pitch, and the playback rate of the adjusted timbre audio signal is consistent with that of the spliced ​​timbre audio signal.

4. The method according to claim 1, characterized in that: The step of obtaining a target audio signal of a target timbre comprises: Acquire an original audio signal of the target timbre and a plurality of envelope characteristic parameters corresponding to the original audio signal; the envelope characteristic parameters of the original audio signal are used to reflect the change of the amplitude of the original audio signal within the duration of the envelope characteristic parameters; According to the ratio between the amplitude change and the duration corresponding to each of the envelope characteristic parameters, determine, from the plurality of envelope characteristic parameters, an envelope characteristic parameter whose ratio is less than a preset ratio threshold as a target envelope characteristic parameter; A target audio signal of the target timbre is generated according to the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal.

5. The method according to claim 4, characterized in that The step of generating the target audio signal of the target timbre according to the audio signal segment corresponding to the target envelope characteristic parameter in the original audio signal comprises: If the average amplitude of the audio signal segment is less than a preset amplitude threshold, amplifying the audio signal segment to obtain a target audio signal of the target timbre; If the amplitude fluctuation frequency of the audio signal segment is greater than a preset frequency threshold, the audio signal segment is compressed to obtain a target audio signal of the target timbre.

6. The method according to claim 1, characterized in that The step of adjusting the duration of at least one envelope characteristic parameter of the target audio signal according to the melody duration to obtain an adjusted envelope characteristic parameter comprises: Determining a sum of durations included in the at least one envelope characteristic parameter; According to the duration ratio between the melody duration and the sum of the duration durations, the duration duration included in the at least one envelope characteristic parameter is scaled so that the sum of the duration durations included in the at least one envelope characteristic parameter after scaling is equal to the melody duration, thereby obtaining the adjusted envelope characteristic parameter.

7. The method according to claim 1, characterized in that The step of generating target audio according to the adjusted timbre audio signal and the adjusted envelope characteristic parameter comprises: The amplitude of the adjusted timbre audio signal and the amplitude included in the adjusted envelope characteristic parameter are multiplied point by point at each time point to generate the target audio.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images