Audio processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an audio processing method, device, electronic device and storage medium, the audio processing method comprising: obtaining original audio and an audio segment selected from the original audio; determining a starting position in the original audio that is aligned with the starting time of the audio segment; based on the starting position, determining the first beat moment of the reference beat in the original audio that is closest to the starting position; based on the first beat moment, adjusting the starting time of the audio segment. The audio processing method, device, electronic device and storage medium according to the present disclosure can solve the problem that different standards for manually selecting audio segments and accurate selection cannot be achieved, and can achieve accurate adjustment of the segmentation moment of the audio segment selected from the original audio.

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] The audio editing process usually involves capturing a complete audio clip. Take short video production as an example. The length of a short video is generally between 10 and 60 seconds. When you need to use a song to compose music for a short video, you cannot use the entire song. You need to select a part of the song. Generally speaking, you need to select the highlight part of the audio that you understand subjectively, such as the chorus of a song.

[0003] Existing audio clip selection usually relies on manual segmentation of the complete audio. For example, in the scenario of short video soundtrack, it may be necessary to manually identify the chorus part of the song and then segment the chorus part as the soundtrack of the short video. However, in such manual selection, it is impossible to achieve accurate selection, and the intercepted audio may be discontinuous or broken. Summary of the invention

[0004] The present disclosure provides an audio processing method, device, electronic device and storage medium to at least solve the problem in the related art that the standards for manually selecting audio segments are different and accurate selection cannot be achieved. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, an audio processing method is provided, the audio processing method comprising: obtaining original audio and an audio segment selected from the original audio; determining a starting position in the original audio that is aligned with the starting time of the audio segment; based on the starting position, determining a first beat moment of a reference beat in the original audio that is closest to the starting position; and adjusting the starting time of the audio segment based on the first beat moment.

[0006] Optionally, the step of determining a starting position in the original audio that is aligned with the starting moment of the audio segment includes: determining the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain; determining the moment corresponding to the maximum correlated audio frame in the original audio as the starting position, wherein the maximum correlated audio frame is the audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

[0007] Optionally, the step of determining the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain includes: converting the original audio and the audio segment into amplitude spectra in the frequency domain, respectively, wherein the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; converting the amplitude spectrum of the original audio and the amplitude spectrum of the audio segment into one-dimensional amplitude spectra, respectively; and determining the correlation between each audio frame of the original audio and each audio frame of the audio segment based on the autocorrelation coefficient between the one-dimensional amplitude spectrum of the original audio and the one-dimensional amplitude spectrum of the audio segment.

[0008] Optionally, the audio processing method also includes: determining an end position in the original audio that is aligned with the end time of the audio segment based on the duration of the audio segment and the starting position; based on the end position, determining a second beat moment of a reference beat in the original audio that is closest to the end position; and based on the second beat moment, adjusting the end time of the audio segment, wherein the reference beat is a re-beat of the original audio.

[0009] Optionally, the audio processing method also includes: detecting the beat of the original audio; based on the interval time of the beat, weakening the audio within the interval time after the start time in the adjusted audio segment, and weakening the audio within the interval time before the end time in the adjusted audio segment.

[0010] Optionally, the audio clip is selected from the original audio in the following manner: performing track processing on the original audio to obtain at least one track audio; determining a turning point at which the audio content of the at least one track audio turns based on change information of characteristic parameters of the at least one track audio; and selecting the audio clip from the original audio based on the turning point of the at least one track audio.

[0011] Optionally, the step of determining the turning point moment when the audio content of the at least one track audio changes according to the change information of the characteristic parameters of the at least one track audio includes: determining a change curve of the characteristic parameters of each track audio in the at least one track audio, and detecting the number of beats per unit time of the track audio; smoothing the change curve according to the number of beats per unit time to obtain a smooth curve; and determining the turning point moment of the track audio based on the peak position in the smooth curve.

[0012] Optionally, based on the peak position in the smooth curve, the step of determining the turning moment of the track audio includes: based on the peak position, determining the rebeat moment in the track audio that is closest to the peak moment before the peak moment corresponding to the peak position; and using the rebeat moment as the turning moment of the track audio.

[0013] Optionally, the peak position is determined in the following manner: performing differential processing on the smooth curve, and determining the peak position based on a differential value in a differential curve obtained after the differential processing.

[0014] Optionally, based on the turning point of the at least one track audio, the step of selecting an audio segment from the original audio includes: for each turning point of each track audio in the at least one track audio, weighting the characteristic parameter value of the track audio based on a preset weight value of the track audio; for each turning point of each track audio, determining the sum of the weighted characteristic parameter values ​​of all track audios; determining the turning point with the largest sum as the turning point of the original audio in timing, and selecting an audio segment from the original audio based on the turning point, wherein the start time or end time of the audio segment is a time before or after the turning point that is spaced a predetermined number of bars from the turning point.

[0015] According to a second aspect of an embodiment of the present disclosure, an audio processing device is provided, characterized in that the audio processing device includes: an acquisition unit, configured to acquire original audio and an audio segment selected from the original audio; a first determination unit, configured to determine a starting position in the original audio that is aligned with the starting time of the audio segment; a second determination unit, configured to determine, based on the starting position, a first beat moment of a reference beat in the original audio that is closest to the starting position; and an adjustment unit, configured to adjust the starting time of the audio segment based on the first beat moment.

[0016] Optionally, the first determination unit is further configured to: determine the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain; determine the moment corresponding to the maximum correlated audio frame in the original audio as the starting position, wherein the maximum correlated audio frame is the audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

[0017] Optionally, the first determination unit is further configured to: convert the original audio and the audio segment into amplitude spectra in the frequency domain, respectively, wherein the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; convert the amplitude spectrum of the original audio and the amplitude spectrum of the audio segment into one-dimensional amplitude spectra, respectively; and determine the correlation between each audio frame of the original audio and each audio frame of the audio segment based on the autocorrelation coefficient between the one-dimensional amplitude spectrum of the original audio and the one-dimensional amplitude spectrum of the audio segment.

[0018] Optionally, the adjustment unit is further configured to: determine an end position in the original audio that is aligned with the end time of the audio segment based on the duration of the audio segment and the starting position; determine a second beat moment of a reference beat in the original audio that is closest to the end position based on the end position; and adjust the end time of the audio segment based on the second beat moment, wherein the reference beat is a re-beat of the original audio.

[0019] Optionally, the audio processing device also includes a weakening unit, which is configured to: detect the beat of the original audio; based on the interval time of the beat, weaken the audio in the adjusted audio segment within the interval time after the start time, and weaken the audio in the adjusted audio segment within the interval time before the end time.

[0020] Optionally, the acquisition unit is further configured to: perform track processing on the original audio to obtain at least one track audio; determine a turning point at which the audio content of the at least one track audio changes based on change information of characteristic parameters of the at least one track audio; and select the audio segment from the original audio based on the turning point of the at least one track audio.

[0021] Optionally, the acquisition unit is further configured to: determine a change curve of characteristic parameters of each track audio in the at least one track audio, and detect the number of beats per unit time of the track audio; smooth the change curve according to the number of beats per unit time to obtain a smooth curve; and determine the turning point of the track audio based on the peak position in the smooth curve.

[0022] Optionally, the acquisition unit is further configured to: determine, based on the peak position, a rebeat moment in the track audio that is closest to the peak moment and before the peak moment corresponding to the peak position; and use the rebeat moment as the turning moment of the track audio.

[0023] Optionally, the acquisition unit is further configured to determine the peak position in the following manner: performing differential processing on the smooth curve, and determining the peak position based on a differential value in a differential curve obtained after the differential processing.

[0024] Optionally, the acquisition unit is further configured to: for each turning moment of each track audio in the at least one track audio, weight the characteristic parameter value of the track audio based on a preset weight value of the track audio; for each turning moment of each track audio, determine the sum of the weighted characteristic parameter values ​​of all track audios; determine the turning moment with the largest sum as the turning point of the original audio in timing, and select an audio segment from the original audio based on the turning point, wherein the start time or end time of the audio segment is a moment before or after the turning point that is spaced a predetermined number of bars from the turning point.

[0025] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor executable instructions, when executed by the processor, prompt the processor to execute the audio processing method according to the present disclosure.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the audio processing method according to the present disclosure.

[0027] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the audio processing method according to the present disclosure is implemented.

[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0029] For audio clips selected from the original audio, the start time of the audio clip can be aligned with the original audio and the precise adjustment of the segmentation time of the audio clip can be achieved according to the position of the reference beat of the original audio. This solves the problem that it is impossible to achieve precise selection manually and avoids the possible discontinuity and distortion of the captured audio.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0032] Figure 1 The flowchart of an example of an audio processing method is shown according to an exemplary embodiment.

[0033] Figure 2 The present invention is a flowchart showing a step of obtaining a starting position in an audio processing method according to an exemplary embodiment.

[0034] Figure 3 The present invention is a flowchart showing a step of determining a correlation between original audio and an audio frame of an audio segment in an audio processing method according to an exemplary embodiment.

[0035] Figure 4 The present invention is a flowchart showing the step of selecting an audio segment from original audio in an audio processing method according to an exemplary embodiment.

[0036] Figure 5 The present invention is a flowchart showing a step of determining a turning point of track audio in an audio processing method according to an exemplary embodiment.

[0037] Figure 6 The present invention is a flowchart showing the steps of selecting an audio segment based on a turning point in an audio processing method according to an exemplary embodiment.

[0038] Figure 7 It is a schematic framework diagram of selecting an audio segment in an audio processing method according to an exemplary embodiment.

[0039] Figure 8 The diagram is a schematic diagram showing an example of detecting rhythm and beat in an audio processing method according to an exemplary embodiment.

[0040] Fig. 9 It is a schematic diagram showing another example of detecting rhythm and beat in an audio processing method according to an exemplary embodiment.

[0041] Fig.10 The diagram is a schematic diagram of audio separation in an audio processing method according to an exemplary embodiment.

[0042] Fig.11 The invention is a block diagram of an audio processing device according to an exemplary embodiment.

[0043] Fig.12 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0047] It should also be noted that the audio processing method according to the exemplary embodiment of the present disclosure can be applied to the server side or the user side. For example, the server side can execute the above audio processing method in response to the audio processing request of the user side, and the user side can also directly execute the above audio processing method.

[0048] Here, the server side may refer to any execution subject that can execute the audio processing method described below, such as a server. The server may include an independently running server, a distributed server, or a server cluster composed of multiple servers, or a cloud computing platform or a virtualization center. The application side may be a concept corresponding to the server side, which can provide local services to users. The application side here may be, for example, a terminal device, which may specifically include physical devices such as smart phones, tablet computers, laptops, digital assistants, wearable devices, and vehicle-mounted terminals. It may also include software running on physical devices such as web browsers, virtual machines, and application clients, which provide services to users by receiving corresponding service information from the server side.

[0049] As mentioned above, in the existing audio selection methods, manual operation is the main method. However, on the one hand, since everyone has different understandings of highlights, manual selection of audio clips such as the highlighted part may have unclear selection logic and inconsistent selection criteria, which may result in different audio clips even if the same audio is selected; on the other hand, the manually intercepted part cannot be guaranteed to be perfect. For example, the audio may be discontinuous or broken due to failure to intercept the entire bar. In addition, when intercepting the highlighted part of the audio, there may be a situation where the lyrics of the audio are not continuous due to failure to intercept according to the rebeat.

[0050] In view of the above problems, an audio processing method, an audio processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product according to exemplary embodiments of the present disclosure will be provided below with reference to the accompanying drawings.

[0051] Figure 1 is a flowchart of an audio processing method according to an exemplary embodiment. Figure 1 As shown, the audio processing method may include the following steps:

[0052] In step S11, original audio and audio segments selected from the original audio may be obtained.

[0053] Here, the original audio may be audio for selecting an audio clip, which may be in the form of music such as a song or a piece of music. Although the original audio is a song as an example for description below, the present disclosure is not limited thereto, and the original audio may be any audio with a rhythm. The audio clip may be at least a portion of the audio in the original audio.

[0054] It should be noted that, in this article, original audio and audio clip are relative concepts and do not represent the audio length and audio content of the two themselves. For example, the original audio can be a complete song or music, or a part of a complete song or music. The audio clip can be a part of the original audio. In some embodiments, it can also be the same as the original audio, that is, the entire original audio can be used as the selected audio clip.

[0055] In this step, the audio clip may be obtained in any manner and is not particularly limited herein.

[0056] In one example, the audio segment can be selected from the original audio manually. For example, the audio segment can be selected from the original audio using an existing manual method.

[0057] In another example, the audio segment can also be automatically selected by performing algorithmic analysis on the original audio. Here, the specific operation steps of automatically selecting the audio segment from the original audio will be referred to below. Figure 4Detailed description.

[0058] In step S12, the starting position aligned with the starting moment of the audio segment in the original audio can be determined.

[0059] In this step, the position of the audio segment selected manually or automatically in the original audio can be found by aligning the starting moment of the audio segment with the corresponding starting position in the original audio.

[0060] As an example, as Figure 2 shown, step S12 may include the following steps:

[0061] In step S121, the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain can be determined.

[0062] In this step S121, the audio segment can be aligned with the original audio in the frequency domain, and the correlation between each audio frame of the original audio and each audio frame of the audio segment can be determined in the frequency domain. In this way, it can be avoided that the alignment is inaccurate due to the audio quality loss caused by transcoding, filtering or other operations on the manually or automatically selected audio segment. Here, the duration of each audio frame can be equal to the number of samples divided by the sampling frequency, that is to say, each audio frame can contain multiple sampling time points.

[0063] As an example, as Figure 3 shown, step S121 may include: step S1211, converting the original audio and the audio segment into amplitude spectra in the frequency domain respectively, where the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; step S1212, converting the amplitude spectra of the original audio and the audio segment into one-dimensional amplitude spectra respectively; step S1213, determining the correlation between each audio frame of the original audio and each audio frame of the audio segment based on the autocorrelation coefficient between the one-dimensional amplitude spectra of the original audio and the audio segment.

[0064] Specifically, in step S1211, the original audio and the audio segment represented in the time domain can be respectively subjected to Fourier transform and taking the absolute value, so as to obtain the amplitude spectra of the original audio and the audio segment in the time-frequency domain.

[0065] For example, if the original audio signal x raw is represented as x raw (t) in the time domain, where t represents time and 0 < t ≤ T, then after short-time Fourier transform (STFT), x raw (t) can be represented in the time-frequency domain by the following formula (1):

[0066] X raw (n, k) = STFT(x raw (t)) (1)

[0067] where n is the frame sequence of the audio frames of the original audio, 0 < n ≤ N S , N S is the total number of frames of the original audio, k is the center frequency sequence of the audio frames, 0 < k ≤ K, K is the total number of frequency points of the original audio, and STFT() represents the short-time Fourier transform.

[0068] Similarly, if the audio segment signal x of length T′ section is represented in the time domain as x section (t), where t represents time, 0 < t ≤ T′, then after the short-time Fourier transform, x section (t) can be represented in the time-frequency domain by the following equation (2):

[0069] X section (N, k) = STFT(x section (t)) (2)

[0070] where N is the frame sequence of the audio frames of the audio segment, 0 < N ≤ N S ′, N S ′ is the total number of frames of the audio segment, k′ is the center frequency sequence of the audio frames, 0 < k′ ≤ K, K′ is the total number of frequency points of the audio segment, and STFT() represents the short-time Fourier transform.

[0071] The original audio X raw (n, k) and the audio segment X section (N, k) obtained through the above equations (1) and (2) are two-dimensional amplitude spectra, that is, the horizontal axis is the time of the frames, and the vertical axis is the amplitude value of each frame. In step S1212, the amplitude spectra of the original audio and the audio segment can be converted into one-dimensional amplitude spectra respectively. For example, the two-dimensional amplitude spectrum can be folded into the form of a one-dimensional array through the Flatten function.

[0072] In step S1213, based on the one-dimensional amplitude spectra of the original audio and the audio segment, the autocorrelation coefficient between each audio frame of the two can be calculated.

[0073] Specifically, the autocorrelation coefficient AC of the amplitude spectra of the original audio and the audio segment can be determined by the following equation (3):

[0074] AC = AutoCorrelation(Flatten(X raw (n, k)), Flatten(X section (N, k))) (3)

[0075] Among them, AutoCorrelation() represents autocorrelation operation, and Flatten() represents one-dimensional array operation.

[0076] Thus, the autocorrelation coefficient AC between each audio frame in the original audio and each audio frame in the audio segment can be obtained, so that the correlation between each audio frame in the original audio and each audio frame in the audio segment can be determined.

[0077] In step S122, the time corresponding to the maximum relevant audio frame in the original audio may be determined as the starting position.

[0078] In this step, the maximum correlated audio frame in the original audio can be determined based on the correlation between each audio frame of the original audio and each audio frame of the audio segment calculated above. Here, the maximum correlated audio frame may refer to an audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

[0079] In the original audio, the time corresponding to the maximum correlation audio frame may be a position aligned with the start time of the audio segment, that is, the above-mentioned start position.

[0080] When the starting position of the audio segment in the original audio is determined, the ending position in the original audio that is aligned with the ending time of the audio segment may be determined based on the duration and the starting position of the audio segment.

[0081] In this way, the audio segment can be positioned in the original audio by aligning the start time and end time of the audio segment with the start position and end position of the original audio, thereby allowing the start time and end time of the audio segment to be adjusted based on the original audio. It should be noted that it is also possible to only align the start time of the audio segment with the start position of the original audio without adjusting the end time of the audio segment.

[0082] In step S13, based on the starting position, the first beat time of the reference beat closest to the starting position in the original audio may be determined.

[0083] In this step, the reference beat may be a re-beat of the original audio, but is not limited thereto. The reference beat may be set according to actual application requirements, and may be any beat in the original audio, such as a weak beat.

[0084] In this way, the re-beat moment closest to the starting position in the original audio can be used as the first beat moment. Here, the first beat moment can be before or after the starting position, or can be the starting position.

[0085] Here, the time at which the reference beat of the original audio is located can be determined based on any method, and the reference beat of the original audio ... Figure 8 and Fig. 9 Two examples of detecting the rhythm and beat of the original audio are given, but the present disclosure is not particularly limited to detecting the rhythm and beat of the original audio.

[0086] In step S14, the start time of the audio segment may be adjusted based on the first beat time.

[0087] Specifically, the starting time of the audio clip can be adjusted to the first beat time. In this way, the problem that the starting time of the audio clip selected manually or automatically does not fall precisely on the beat can be solved. The starting time of the audio clip can be adjusted to the first beat time, thereby achieving fine-tuning of the audio clip selection.

[0088] Here, when the first beat moment is the starting position, the first beat moment is aligned with the starting moment of the audio segment, and therefore, adjusting the starting moment of the audio segment to the first beat moment may refer to keeping the starting moment of the audio segment at the current moment.

[0089] Similarly, when determining the end position in the original audio that is aligned with the end time of the audio segment, the second beat time of the reference beat closest to the end position in the original audio can be determined based on the end position, and the end time of the audio segment can be adjusted based on the second beat time. Specifically, the end time of the audio segment can be adjusted to the second beat time.

[0090] According to the audio processing method of the exemplary embodiment described above, after determining the starting time of the audio segment, the starting time of the audio segment can be adjusted to a reference beat such as a rebeat, taking into account the completeness of the audio rhythm. For example, the end of the beat before the reference beat can be used as the starting time of the audio segment.

[0091] In this way, according to the audio processing method of the exemplary embodiment of the present disclosure, the beat and beat moments of the original audio are found through the beat and beat detection algorithm, and the beat (for example, the beat) closest to the start and end times of the selected audio segment is found to serve as the start and end times of the new audio segment, respectively. This can ensure the rhythmic integrity of the audio segment to a certain extent, thereby avoiding the auditory confusion caused by irregular interception of the audio segment.

[0092] After adjusting the start time and / or end time of the audio segment, the audio processing method according to the exemplary embodiment of the present disclosure may also include: detecting the beat of the original audio; based on the interval time of the beats, weakening the audio in the interval time after the start time of the adjusted audio segment, and weakening the audio in the interval time before the end time of the adjusted audio segment.

[0093] Specifically, the audio between the start time and the first beat after the start time and the audio between the first beat before the end time and the end time can be weakened so that the audio segment can have a fade-in and fade-out effect, preventing explosive sounds from appearing at the beginning and end of the audio segment.

[0094] The following will refer to Figure 4 Describes in detail the steps to extract an audio clip from the original audio.

[0095] like Figure 4 As shown, the step of selecting an audio segment from the original audio may include:

[0096] In step S21, the acquired original audio may be processed into tracks to obtain at least one track audio.

[0097] Here, the original audio may be divided into tracks according to predetermined tracks, which may include vocal tracks and instrument tracks, which may include at least one of drum tracks, bass tracks, and mixed audio tracks, which are audio tracks other than vocal tracks, drum tracks, and bass tracks.

[0098] Here, the original audio can be split into tracks based on any method, which will be referred to below. Fig.10 An example of dividing the original audio into tracks is given, but the present disclosure does not particularly limit the method of dividing the original audio into tracks.

[0099] In step S22, a turning point moment at which the audio content of at least one track of audio changes can be determined based on change information of a characteristic parameter of at least one track of audio.

[0100] Here, the characteristic parameter may be a parameter characterizing the characteristics of the track audio, which may be preset. For example, the characteristic parameter corresponding to the track audio of the vocal track may be pitch, and the characteristic parameter corresponding to the track audio of the instrument track may be sound energy. For example, for the track audio of the vocal track, the pitch may be extracted from the fundamental frequency in the original audio.

[0101] As an example, Figure 5 As shown, step S22 may specifically include the following steps:

[0102] In step S221, for each track audio in at least one track audio, a change curve of a characteristic parameter of the track audio can be determined, and the number of beats per unit time of the track audio can be detected.

[0103] In this step, a change curve of the corresponding characteristic parameter can be determined for each track audio, for example, for the track audio of the vocal track, the pitch change curve of the track audio can be determined. For the track audio of the instrument track, the sound energy change curve of the track audio can be determined.

[0104] In this step, the number of beats per unit time of the original audio may be determined based on any method, which will be referred to below. Figure 8 and Fig. 9 Two examples of detecting the rhythm and beat of the original audio are given, but the present disclosure does not impose any particular limitation on detecting the number of beats per unit time of the original audio.

[0105] In step S222, the change curve may be smoothed according to the number of beats per unit time to obtain a smooth curve.

[0106] In this step, the number of beats per unit time may be, for example, the number of beats per minute (BPM). Specifically, a smoothing factor for smoothing the changing curve may be set according to the number of beats per unit time, and the smoothing factor may be inversely proportional to the number of beats per unit time. The larger the number of beats per unit time, the smaller the smoothing factor may be set; and the smaller the number of beats per unit time, the larger the smoothing factor may be set. Here, the specific value of the smoothing factor may be set according to actual needs and the selected smoothing algorithm, and the smoothing algorithm may be any curve smoothing algorithm such as median filtering, average value smoothing, etc.

[0107] In step S223, the turning point of the track audio may be determined based on the peak position in the smooth curve.

[0108] Generally speaking, it can be considered that the peak position of the characteristic parameters of the track audio is close to or at the turning point of the audio. For example, for a song, the pitch of the chorus will be significantly higher than that of the verse, and the sound energy of the chorus drum will be significantly greater than that of the verse.

[0109] In this way, the re-beat moment closest to the peak moment before the peak moment corresponding to the peak position in the track audio can be determined according to the peak position in the smooth curve, and the re-beat moment can be used as the turning moment of the track audio. For example, the re-beat moment closest to the peak moment before the peak moment can be used as the turning moment of the track audio, such as the turning moment between the main song and the chorus.

[0110] In this step, the peak position can be determined by performing differential processing such as first-order differential on the smooth curve and based on the differential value in the differential curve obtained after the differential processing, such as performing peak detection, so as to find the turning point of the track audio, for example, the time position where the pitch becomes significantly higher or the time position where the sound energy becomes significantly larger.

[0111] Here, each track audio may have one or more turning moments.

[0112] In step S23, an audio segment may be selected from the original audio based on a turning point of at least one track audio.

[0113] As an example, Figure 6 As shown, step S23 may include the following steps:

[0114] In step S231, for each turning moment of each track audio in at least one track audio, the characteristic parameter value of the track audio may be weighted based on a preset weight value of the track audio.

[0115] Here, the preset weight value may be pre-set, for example, it may be allocated according to the energy proportion of different track audios in the original audio. Specifically, in this step, at each turning point of each track audio, the values ​​of the characteristic parameters of all track audios may be normalized, and the normalized characteristic parameters may be weighted using the preset weight value of the corresponding track audio.

[0116] For example, the first track audio has a first turning moment, and the second track audio has a second turning moment. For the first turning moment, the preset weight value of the first track audio can be used to weight the characteristic parameters of the first track audio at the first turning moment, and the preset weight value of the second track audio can be used to weight the characteristic parameters of the second track audio at the first turning moment; for the second turning moment, the preset weight value of the first track audio can be used to weight the characteristic parameters of the first track audio at the second turning moment, and the preset weight value of the second track audio can be used to weight the characteristic parameters of the second track audio at the second turning moment.

[0117] In step S232, for each turning point of each track audio, the sum of the weighted characteristic parameter values ​​of all track audios may be determined.

[0118] Taking the above example, the first sum of the weighted feature parameter value of the first track audio and the weighted feature parameter value of the second track audio at the first turning point can be calculated; and the second sum of the weighted feature parameter value of the first track audio and the weighted feature parameter value of the second track audio can be calculated at the second turning point.

[0119] In step S233, the turning point with the largest sum value can be determined as the turning point of the original audio in terms of time sequence, and an audio segment can be selected from the original audio based on the turning point.

[0120] Taking the above example, the turning point corresponding to the maximum value of the first sum value and the second sum value can be used as the turning point of the original audio in terms of timing.

[0121] The start time or end time of the selected audio segment may be a time before or after the turning point that is spaced a predetermined number of bars from the turning point.

[0122] Taking the original audio as an example of a song, after obtaining the turning point between the verse and the chorus of the original audio, you can extract the highlight part of the chorus, that is, several bars after the turning point (generally selected as 4 or 8 bars); you can extract the highlight part of the transition, that is, several bars before and after the turning point (generally selected as 4 or 8 bars).

[0123] According to the audio processing method of the exemplary embodiment of the present disclosure, for an audio segment selected manually or automatically, the segmentation time accuracy of the audio segment can be adjusted based on the results of rhythm detection and stress detection.

[0124] The purpose of segmenting music segments such as the highlight part of a song can be achieved by automatically understanding and analyzing the original audio content. Specifically, for the original audio, audio segments such as the highlight part of the chorus and the highlight part of the transition can be automatically extracted by analyzing the turning moments such as the turning points of the verse and chorus, thereby realizing the automatic extraction of audio segments.

[0125] The following will refer to Figure 7 A schematic framework diagram describing the selection of an audio segment in an audio processing method according to an exemplary embodiment.

[0126] like Figure 7 As shown, the rhythm (beats), that is, the beat moment, of the original audio can be obtained by performing rhythm detection on the input original audio, the downbeat moment of the original audio can be obtained by performing beat detection on the original audio, and the track audio of different tracks can be obtained by performing audio separation on the original audio, such as vocal track audio, bass track audio, drum track audio and other track audio, and then the audio clips can be selected based on the obtained beats, downbeats and track audio, and the specific selection process has been described above. In the audio processing method of the present invention, the deep neural network's understanding of the various dimensions of the original audio can be used, combined with the knowledge of music theory and traditional signal processing, to achieve rhythm detection, downbeat detection and audio separation, thereby automatically selecting music clips in different scenarios, which will be described in detail below.

[0127] First, refer to Figure 8 and Fig. 9 A schematic diagram is provided to describe an example and another example of detecting rhythm and beat in an audio processing method according to an exemplary embodiment.

[0128] Specifically, the original audio can be input into a pre-trained first machine learning model, and the rhythm and beats in the original audio can be determined based on the output of the first machine learning model, wherein the first machine learning model is used to detect the rhythm and beats in the audio.

[0129] Here, beat is an organizational form in music that represents a fixed unit of duration and a pattern of strength and weakness, which defines the rhythmic structure of a musical work. In music, the rhythm formed by the beat is characterized by a repeated sequence of downbeats and downbeats, where downbeats refer to the strong beats in music.

[0130] The rhythm detection algorithm can automatically identify the beat location by analyzing the audio data. As an example, different neural networks can be used for rhythm and beat detection respectively, or the same neural network can be used for both rhythm and beat detection.

[0131] Figure 8 The first machine learning model shown may include a rhythm detection network and a beat detection network that are independent of each other. Fig. 9 The first machine learning model shown may include a rhythm detection subnetwork and a beat detection subnetwork connected to each other.

[0132] Figure 8 and Fig. 9 The first machine learning model shown may include a convolutional neural network (CNN), a bidirectional long short-term memory network (BLSTM), a concatenation network (CONCAT), and a fully connected network (DENSE). Figure 8 and Fig. 9 In the figure, CNN(C) is a two-dimensional convolutional layer with C convolution kernels; BLSTM(L) is a bidirectional long short-term memory recurrent neural network with L nodes; CONCAT can perform a merge operation, for example, a 5×2 matrix and a 5×4 matrix are merged in the second dimension to obtain a 5×6 matrix; DENSE(d) is a fully connected layer with d nodes.

[0133] The input of each network is the time-frequency spectrum of the signal, such as the short-time Fourier spectrum. Specifically, the short-time Fourier spectrum of the signal can be obtained by the following operations:

[0134] For example, if the original audio signal x with a length of T raw is represented in the time domain as x raw (t), where t represents time and 0 < t ≤ T, then after the short-time Fourier transform, x raw (t) can be represented in the time-frequency domain as:

[0135] X raw (n,k) = STFT(x raw (t))

[0136] where n is the frame sequence of the audio frames of the original audio, 0 < n ≤ N S , N S is the total number of frames of the original audio, k is the center frequency sequence of the audio frames, 0 < k ≤ K, K is the total number of frequency points of the original audio, and STFT() represents the short-time Fourier transform.

[0137] Alternatively, the input of each network can be the Mel spectrogram of the signal. Specifically, the Mel spectrogram of the signal can be obtained through the operation of Equation (4) below:

[0138] M(n,k) = Mel(X raw (n,m)) (4)

[0139] where n is the frame sequence of the audio frames of the original audio, k is the center frequency sequence of the audio frames, 0 < k ≤ K, m is the frequency sequence after Mel spectrogram mapping, and Mel() represents the Mel spectrogram feature.

[0140] The beat probability P1(n) and the downbeat probability P2(n) can be obtained through a rhythm detection network and a downbeat detection network respectively. Combining the pre-annotated beat annotation information B1(n) and downbeat annotation information B2(n), thus, the objective loss function l1(n) = Loss(P1(n), B1(n)) for rhythm detection and the objective loss function l2(n) = Loss(P2(n), B2(n)) for downbeat detection can be calculated. Therefore, the total loss function l(n) = l1(n) + l2(n) of the first machine learning model can be obtained by adding the objective loss functions of the two neural networks. Here, the Loss() loss function can be, for example, a weighted cross entropy loss function.

[0141] Fig.10 is a schematic diagram of audio separation in an audio processing method shown according to an exemplary embodiment.

[0142] Specifically, the original audio can be input into a pre-trained second machine learning model, and the second machine learning model can be used to separate the track audio corresponding to the preset audio track from the original audio, wherein the second machine learning model is used to separate the track audio corresponding to the preset audio track from the audio.

[0143] like Fig.10 As shown, the second machine learning model can be a deep audio separation network, which can separate the human voice and various musical instruments in the audio. For example, the input audio signal x can be noisy speech, and the second machine learning model can separate the input audio signal x into four tracks of audio according to the human voice track, drum track, bass track and mixed audio track. The input audio signal x can be converted into a mixed signal short-time Fourier spectrum via a short-time Fourier transform (STFT), and the masking process (e.g., masking matrix) estimated by the second machine learning model and the input signal are multiplied to obtain the output separation signal Short-time Fourier spectrum. Then, after the short-time Fourier inverse transform (ISTFT), the track audio in the time domain is obtained. Here, different deep audio separation networks can be used for different tracks. For example, four networks can be used to separate the mixed signal of the original audio into vocal track audio, drum track audio, bass track audio, and mixed track audio.

[0144] The second machine learning model can be constructed based on a U-shaped network (U-NET network). Fig.10 As shown in Figure 1, the U-NET network has an encoder-decoder architecture. The Encoder of the U-NET network contains multiple convolution blocks, each of which contains a two-dimensional convolution (Conv2d) layer, a batch normalization (Batch Normalization) layer, a dropout (Dropout) layer, and a LeakyRelu activation function. The Decoder of the U-NET network contains 8 reverse convolution blocks, and the only difference between each reverse convolution block and the corresponding convolution block is that the Conv2d layer is replaced by a two-dimensional convolution transpose (Conv2dTranspose) layer. Table 1 below shows the hyperparameters of each Conv2d layer in the 8-layer Encoder.

[0145] Table 1

[0146]

[0147] Fig.11 is a block diagram of an audio processing device according to an exemplary embodiment. Fig.11 The audio processing device 1000 includes an acquisition unit 100, a first determination unit 200, a second determination unit 300 and an adjustment unit 400.

[0148] The acquisition unit 100 may be configured to acquire original audio and an audio segment selected from the original audio.

[0149] The first determining unit 200 may be configured to determine a start position in the original audio that is aligned with a start time of the audio segment.

[0150] The second determining unit 300 may be configured to determine, based on the starting position, a first beat time of a reference beat closest to the starting position in the original audio.

[0151] The adjustment unit 400 may be configured to adjust the start time of the audio segment based on the first beat time.

[0152] As an example, the first determination unit 200 can also be configured to: determine the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio fragment in the frequency domain; determine the moment corresponding to the maximum correlated audio frame in the original audio as the starting position, wherein the maximum correlated audio frame is the audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

[0153] As an example, the first determination unit 200 can also be configured to: convert the original audio and the audio segment into amplitude spectra in the frequency domain, respectively, wherein the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; convert the amplitude spectrum of the original audio and the amplitude spectrum of the audio segment into one-dimensional amplitude spectra, respectively; and determine the correlation between each audio frame of the original audio and each audio frame of the audio segment based on the autocorrelation coefficient between the one-dimensional amplitude spectrum of the original audio and the one-dimensional amplitude spectrum of the audio segment.

[0154] Optionally, the adjustment unit is further configured to:

[0155] Based on the duration and the start position of the audio segment, determining an end position in the original audio that is aligned with the end time of the audio segment;

[0156] Based on the end position, determine the second beat time of the reference beat closest to the end position in the original audio;

[0157] Adjust the end time of the audio clip based on the second beat time.

[0158] The reference beat is the beat of the original audio.

[0159] As an example, the audio processing device 1000 also includes a weakening unit 500, which can be configured to: detect the beat of the original audio; based on the interval time of the beat, weaken the audio in the interval time after the start time in the adjusted audio segment, and weaken the audio in the interval time before the end time in the adjusted audio segment.

[0160] As an example, the acquisition unit 100 can also be configured to: perform track processing on the original audio to obtain at least one track audio; determine the turning point when the audio content of at least one track audio turns based on the change information of the characteristic parameters of the at least one track audio; and select an audio clip from the original audio based on the turning point of the at least one track audio.

[0161] As an example, the acquisition unit 100 can also be configured to: for each track audio in at least one track audio, determine the change curve of the characteristic parameters of the track audio, and detect the number of beats per unit time of the track audio; according to the number of beats per unit time, smooth the change curve to obtain a smooth curve; based on the peak position in the smooth curve, determine the turning point of the track audio.

[0162] As an example, the acquisition unit 100 can also be configured to: based on the peak position, determine the rebeat moment closest to the peak moment in the track audio before the peak moment corresponding to the peak position; and use the rebeat moment as the turning moment of the track audio.

[0163] As an example, the acquisition unit 100 may also be configured to determine the peak position in the following manner: performing differential processing on the smooth curve, and determining the peak position based on a differential value in the differential curve obtained after the differential processing.

[0164] As an example, the acquisition unit 100 can also be configured to: for each turning moment of each track audio in at least one track audio, weight the characteristic parameter value of the track audio based on the preset weight value of the track audio; for each turning moment of each track audio, determine the sum of the weighted characteristic parameter values ​​of all track audios; determine the turning moment with the largest sum as the turning point of the original audio in timing, and select an audio segment from the original audio based on the turning point, wherein the start time or end time of the audio segment is a moment before or after the turning point that is separated from the turning point by a predetermined number of bars.

[0165] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] Fig.12FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Fig.12 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the audio processing method as described in the above exemplary embodiment.

[0167] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as a server that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0168] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0169] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0170] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0171] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0172] In an exemplary embodiment, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by the processor of the server, the server can execute the audio processing method as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0173] In an exemplary embodiment, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the audio processing method as described in the exemplary embodiment above is implemented.

[0174] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0175] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: The audio processing method comprises: Acquire original audio and an audio segment selected from the original audio, wherein the audio segment is at least a portion of the original audio; Determining a starting position in the original audio that is aligned with a starting time of the audio segment; Based on the starting position, determining a first beat time of a reference beat in the original audio that is closest to the starting position; Adjusting the start time of the audio segment based on the first beat time; The obtaining of the original audio and the audio clip selected from the original audio includes: Performing track processing on the original audio to obtain at least one track audio; Determining, according to the change information of the characteristic parameter of the at least one track audio, a turning point moment at which the audio content of the at least one track audio turns; For each turning moment of each track audio in the at least one track audio, weighting the characteristic parameter value of the track audio based on a preset weight value of the track audio; For each turning point of each track audio, determine the sum of weighted characteristic parameter values ​​of all track audios; The turning point at which the sum value is the largest is determined as a turning point in the timing of the original audio, and an audio segment is selected from the original audio based on the turning point.

2. The audio processing method according to claim 1, characterized in that: The step of determining a starting position in the original audio that is aligned with the starting time of the audio segment comprises: Determine a correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain; The time corresponding to the maximum correlation audio frame in the original audio is determined as the starting position, wherein the maximum correlation audio frame is an audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

3. The audio processing method according to claim 2, characterized in that: The step of determining the correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain comprises: Convert the original audio and the audio segment into amplitude spectra in the frequency domain, respectively, wherein the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; Converting the amplitude spectrum of the original audio and the amplitude spectrum of the audio segment into one-dimensional amplitude spectra respectively; Based on the autocorrelation coefficient between the one-dimensional magnitude spectrum of the original audio and the one-dimensional magnitude spectrum of the audio segment, the correlation between each audio frame of the original audio and each audio frame of the audio segment is determined.

4. The audio processing method according to claim 2, characterized in that: The audio processing method further comprises: Determining an end position in the original audio that is aligned with an end time of the audio segment based on the duration of the audio segment and the start position; Based on the end position, determine a second beat time of a reference beat closest to the end position in the original audio; Based on the second beat time, adjusting the end time of the audio segment, The reference beat is a re-beat of the original audio.

5. The audio processing method according to any one of claims 1 to 4, characterized in that: The audio processing method further comprises: Detecting the beat of the original audio; Based on the interval time of the beats, the audio in the interval time after the start time of the adjusted audio segment is weakened, and the audio in the interval time before the end time of the adjusted audio segment is weakened.

6. The audio processing method according to claim 1, characterized in that: The step of determining, according to the change information of the characteristic parameter of the at least one track audio, a turning point moment at which the audio content of the at least one track audio turns includes: For each track audio of the at least one track audio, determine a change curve of a characteristic parameter of the track audio, and detect the number of beats per unit time of the track audio; According to the number of beats in the unit time, the change curve is smoothed to obtain a smooth curve; Based on the peak position in the smooth curve, the turning point of the audio track is determined.

7. The audio processing method according to claim 6, characterized in that: The step of determining the turning point of the track audio based on the peak position in the smooth curve comprises: Based on the peak position, determine a re-beat moment closest to the peak moment before the peak moment corresponding to the peak position in the track audio; The retake moment is regarded as the turning point of the audio of the track.

8. The audio processing method according to claim 6, characterized in that: The peak position is determined in the following manner: performing differential processing on the smooth curve, and determining the peak position based on a differential value in a differential curve obtained after the differential processing.

9. The audio processing method according to any one of claims 1 to 8, characterized in that: The start time or the end time of the audio segment is a time before or after the turning point that is spaced a predetermined number of bars from the turning point.

10. An audio processing device, characterized in that: The audio processing device comprises: an acquisition unit, configured to acquire original audio and an audio segment selected from the original audio, wherein the audio segment is at least a portion of the original audio; A first determining unit is configured to determine a starting position in the original audio that is aligned with a starting time of the audio segment; A second determining unit is configured to determine, based on the starting position, a first beat time of a reference beat in the original audio that is closest to the starting position; an adjusting unit, configured to adjust a start time of the audio segment based on the first beat time; Wherein, the acquisition unit is further configured to: Performing track processing on the original audio to obtain at least one track audio; Determining, according to the change information of the characteristic parameter of the at least one track audio, a turning point moment at which the audio content of the at least one track audio turns; For each turning moment of each track audio in the at least one track audio, weighting the characteristic parameter value of the track audio based on a preset weight value of the track audio; For each turning point of each track audio, determine the sum of weighted characteristic parameter values ​​of all track audios; The turning point at which the sum value is the largest is determined as a turning point in the timing of the original audio, and an audio segment is selected from the original audio based on the turning point.

11. The audio processing device according to claim 10, characterized in that: The first determining unit is further configured to: Determine a correlation between each audio frame of the original audio in the frequency domain and each audio frame of the audio segment in the frequency domain; The time corresponding to the maximum correlation audio frame in the original audio is determined as the starting position, wherein the maximum correlation audio frame is an audio frame corresponding to the maximum value of the correlation among all audio frames of the original audio.

12. The audio processing device according to claim 11, characterized in that: The first determining unit is further configured to: Convert the original audio and the audio segment into amplitude spectra in the frequency domain, respectively, wherein the amplitude spectrum represents the amplitude value at the center frequency of each frame of the audio; Converting the amplitude spectrum of the original audio and the amplitude spectrum of the audio segment into one-dimensional amplitude spectra respectively; Based on the autocorrelation coefficient between the one-dimensional magnitude spectrum of the original audio and the one-dimensional magnitude spectrum of the audio segment, the correlation between each audio frame of the original audio and each audio frame of the audio segment is determined.

13. The audio processing device according to claim 11, characterized in that: The adjustment unit is further configured to: Determining an end position in the original audio that is aligned with an end time of the audio segment based on the duration of the audio segment and the start position; Based on the end position, determine a second beat time of a reference beat closest to the end position in the original audio; Based on the second beat time, adjusting the end time of the audio segment, The reference beat is a re-beat of the original audio.

14. The audio processing device according to any one of claims 10 to 13, characterized in that: The audio processing device also includes a weakening unit, which is configured to: detect the beat of the original audio; based on the interval time of the beat, weaken the audio in the adjusted audio segment within the interval time after the start time, and weaken the audio in the adjusted audio segment within the interval time before the end time.

15. The audio processing device according to claim 10, characterized in that: The acquisition unit is further configured to: For each track audio of the at least one track audio, determine a change curve of a characteristic parameter of the track audio, and detect the number of beats per unit time of the track audio; According to the number of beats in the unit time, the change curve is smoothed to obtain a smooth curve; Based on the peak position in the smooth curve, the turning point of the audio track is determined.

16. The audio processing device according to claim 15, characterized in that: The acquisition unit is further configured to: Based on the peak position, determine a re-beat moment closest to the peak moment before the peak moment corresponding to the peak position in the track audio; The retake moment is regarded as the turning point of the audio of the track.

17. The audio processing device according to claim 15, characterized in that: The acquisition unit is further configured to determine the peak position by performing differential processing on the smooth curve, and determining the peak position based on a differential value in a differential curve obtained after the differential processing.

18. The audio processing device according to any one of claims 10 to 17, characterized in that: The start time or the end time of the audio segment is a time before or after the turning point that is spaced a predetermined number of bars from the turning point.

19. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, When the processor executable instructions are executed by the processor, the processor is prompted to execute the audio processing method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to perform the audio processing method according to any one of claims 1 to 9.

21. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the audio processing method according to any one of claims 1 to 9 is implemented.

Images