Timestamp determination method for dry audio, electronic device and storage medium

Abstract

The application discloses a timestamp determination method for dry audio, an electronic device and a storage medium. The method comprises the following steps: obtaining a target dry audio and a standard dry audio corresponding to the target dry audio; extracting a first basic audio feature of the target dry audio and a second basic audio feature of the standard dry audio; inputting the first basic audio feature and the second basic audio feature into a feature extraction model based on a neural network respectively to obtain a first enhanced audio feature corresponding to the first basic audio feature and a second enhanced audio feature corresponding to the second basic audio feature; aligning the first enhanced audio feature and the second enhanced audio feature by using a dynamic time warping method to determine a corresponding relationship between each frame of audio in the standard dry audio and each frame of audio in the target dry audio; and determining timestamp information of the target dry audio based on timestamp information of the standard dry audio and the corresponding relationship, thereby improving the accuracy of determining the timestamp of the dry audio.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, more particularly, to a timestamp determination method and device for dry audio, an electronic device and a computer readable storage medium. BACKGROUND

[0002] In recent years, the application of online karaoke platforms in intelligent technology has become increasingly prominent. The karaoke platform can support the one-key tuning function of most songs, realize the effect of automatically optimizing the user's singing works, and reduce the difficulty and threshold of online karaoke. In the one-key tuning function, automatically analyzing the timestamp information of the user's dry sound is the premise of automatic tuning.

[0003] In the related art, the automatic speech recognition technology or forced alignment technology based on HMM (Hidden Markov Model) can be used to estimate the timestamp information of the user's dry sound. The accuracy of this scheme is not high, especially when encountering long vowels, the problem of cutting off long vowels is prone to occur, and in addition, a model needs to be trained separately for each language, which is low in efficiency.

[0004] In addition, the mel-frequency cepstral coefficient of the dry audio can also be extracted, and then dynamic time warping is performed to obtain timestamp alignment information. However, since the mel-frequency cepstral coefficient is extracted separately for each frame of audio signal, there is no continuity and clustering between consecutive frames, and the adaptability to noise is poor.

[0005] Therefore, how to improve the accuracy of determining the timestamp of the dry audio is a technical problem to be solved by those skilled in the art. SUMMARY

[0006] The present application aims to provide a timestamp determination method for dry audio, an electronic device and a computer readable storage medium, which improves the accuracy of determining the timestamp of the dry audio.

[0007] To achieve the above-mentioned purpose, the first aspect of the present application provides a timestamp determination method for dry audio, comprising:

[0008] obtaining a target dry audio and a standard dry audio corresponding to the target dry audio;

[0009] extracting first basic audio features of the target dry audio and second basic audio features of the standard dry audio by using a vocoder;

[0010] inputting the first basic audio features and the second basic audio features into a feature extraction model based on a neural network respectively, to obtain first enhanced audio features corresponding to the first basic audio features and second enhanced audio features corresponding to the second basic audio features respectively;

[0011] aligning the first enhanced audio features and the second enhanced audio features by using a dynamic time warping method to determine a correspondence between each frame of audio in the standard dry audio and each frame of audio in the target dry audio;

[0012] determining timestamp information of the target dry audio based on the timestamp information of the standard dry audio and the correspondence.

[0013] To achieve the above object, a second aspect of the present application provides an electronic device, comprising:

[0014] a memory for storing a computer program;

[0015] a processor for executing the computer program to implement the steps of the dry audio timestamp determination method.

[0016] To achieve the above object, a third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the dry audio timestamp determination method.

[0017] According to the above scheme, the dry audio timestamp determination method provided by the present application comprises: obtaining a target dry audio and a standard dry audio corresponding to the target dry audio; extracting first basic audio features of the target dry audio and second basic audio features of the standard dry audio by using a vocoder; inputting the first basic audio features and the second basic audio features into a neural network-based feature extraction model respectively to obtain first enhanced audio features corresponding to the first basic audio features and second enhanced audio features corresponding to the second basic audio features respectively; aligning the first enhanced audio features and the second enhanced audio features by using a dynamic time warping method to determine a correspondence between each frame of audio in the standard dry audio and each frame of audio in the target dry audio; and determining timestamp information of the target dry audio based on the timestamp information of the standard dry audio and the correspondence.

[0018] The timestamp determination method for dry audio provided in the application extracts the robust and continuous feature representations corresponding to the target dry audio of the user and the standard dry audio of the original song respectively by using the feature extraction model based on the neural network, instead of the acoustic features such as the mel-frequency cepstral coefficient in the related art, which has more obvious advantages in overcoming noise interference and non-standard pronunciation scenes. Further, the first enhanced audio features corresponding to the target dry audio and the second enhanced audio features corresponding to the standard dry audio are dynamically time warping, so that the frame-level correspondence between the target dry audio and the standard dry audio can be obtained. Since the timestamp information of the standard dry audio is known, the timestamp information of the target dry audio can be deduced through the above correspondence. Therefore, the timestamp determination method for dry audio provided in the application improves the accuracy of determining the timestamp of the dry audio. The application also discloses an electronic device and a computer readable storage medium, which can also achieve the above technical effects.

[0019] It should be understood that the above general description and the following detailed description are exemplary and not limiting. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor. The drawings are used to provide further understanding of the present disclosure and constitute a part of the specification, and are used to explain the present disclosure together with the following specific embodiments, but do not constitute a limitation on the present disclosure. In the drawings:

[0021] Figure 1 A flowchart of a timestamp determination method for dry audio provided by an embodiment of the present application;

[0022] Figure 2 A structural schematic diagram of a feature extraction model provided by an embodiment of the present application;

[0023] Figure 3 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] The embodiment of the present application discloses a timestamp determination method of dry audio, which improves the accuracy of determining the timestamp of dry audio.

[0026] Referring to Figure 1 , the flowchart of the timestamp determination method of dry audio provided by the embodiment of the present application is shown in Figure 1 , which comprises the following steps.

[0027] S101: acquiring target dry audio and standard dry audio corresponding to the target dry audio.

[0028] The purpose of the embodiment is to determine the timestamp information of the target dry audio sung by the user, that is, to determine the start and end time of each word sung by the user, and subsequently, the corresponding dry sound segment of each word can be pitch processed according to the pitch information of each word specified in the original singing score, so as to realize the sound correction function. The execution subject of the embodiment can be a client providing karaoke, or an electronic device, that is, the client sends the target dry audio sung by the user to the electronic device for backend processing.

[0029] In this step, the target dry audio sung by the user and the corresponding standard dry audio are acquired. The standard dry audio can be understood as the dry audio of the original song.

[0030] S102: extracting first basic audio features of the target dry audio and second basic audio features of the standard dry audio by using a vocoder.

[0031] In this step, the first basic audio features of the target dry audio and the second basic audio features of the standard dry audio are extracted by using a vocoder. The basic audio features in the embodiment can include mel-frequency cepstral coefficients, spectral envelope and other acoustic features, which are not limited here.

[0032] As a preferred embodiment, this step comprises: extracting the spectral envelope of the target dry audio as the first basic audio features and extracting the spectral envelope of the standard dry audio as the second basic audio features. It can be understood that when the sound wave produced by the vibration of the vocal cord passes through the sound channel composed of the oral cavity, the nasal cavity and the like, resonance will occur. The result of the resonance will strengthen some areas of the frequency spectrum, thereby forming peaks. There are multiple peaks on the frequency spectrum, and the height of each peak on the frequency spectrum is not the same. The ratio of the heights of these peaks determines the timbre. Connecting these peaks by a smooth curve is the spectral envelope. It should be noted that the spectral envelope has low correlation with the fundamental frequency, and is only related to the pronunciation content, pronunciation timbre and pronunciation energy. Therefore, using the spectral envelope can make the accuracy of the subsequently determined timestamp information higher.

[0033] As a feasible implementation, the extracting the spectral envelope of the target dry audio as the first basis audio feature and the spectral envelope of the standard dry audio as the second basis audio feature by using the vocoder comprises: extracting a target fundamental frequency of the target dry audio by using the vocoder, performing inverse Fourier transform on a power spectrum of the target dry audio to obtain a target cepstrum, performing low-pass processing on the target cepstrum based on the target fundamental frequency to obtain the spectral envelope of the target dry audio as the first basis audio feature; and extracting a standard fundamental frequency of the standard dry audio by using the vocoder, performing inverse Fourier transform on a power spectrum of the standard dry audio to obtain a standard cepstrum, and performing low-pass processing on the standard cepstrum based on the standard fundamental frequency to obtain the spectral envelope of the standard dry audio as the second basis audio feature. In a specific implementation, the fundamental frequency of the dry audio is first extracted, and the specific manner of extracting the fundamental frequency is not limited here, and the harvest algorithm of the world vocoder, the pYin algorithm, the DIO algorithm, etc. can be used. The basic principle of the harvest algorithm is to filter the dry audio by using low-pass filters with different cutoff frequencies, analyze the standard degree of the waveform of the filtered audio, and take the cutoff frequency corresponding to the waveform most similar to the standard sinusoidal signal as the fundamental frequency. Then, the CheapTrick algorithm of the world vocoder can be used to perform inverse Fourier transform on the power spectrum of the dry audio to obtain the cepstrum, and then the cepstrum can be low-pass processed based on the fundamental frequency to obtain the spectral envelope of the dry audio. Preferably, the dimension of the spectral envelope corresponding to each frame of audio signal in the extracted dry audio can be 60.

[0034] S103: inputting the first basis audio feature and the second basis audio feature into a feature extraction model based on a neural network respectively to obtain a first enhanced audio feature corresponding to the first basis audio feature and a second enhanced audio feature corresponding to the second basis audio feature respectively;

[0035] In this step, the feature extraction model based on the neural network is used to convert the first basis audio feature and the second basis audio feature into features with stronger robustness and continuity, i.e., the first enhanced audio feature and the second enhanced audio feature.

[0036] As a feasible implementation, the feature extraction model in this embodiment is as follows: Figure 2As shown, it includes a linear transformation layer (Linear), a leaky rectified linear unit (LeakyReLU), and a dilated convolution layer (Dilated Conv), wherein the base module composed of LeakyReLU and Dilated Conv is repeated 5 times. If the first and second base audio features are both 60-dimensional spectral envelopes, after passing through the first Linear layer, the dimension changes from [T, 60] to [T, 128], the convolution kernel size of the Dilated Conv in the five base modules is all 3, the channel number is all 128, but the dilation coefficients are [2, 4, 8, 2, 4] respectively, thereby increasing the receptive field of the model, and finally passing through a Linear layer again, the obtained feature representation changes from [T, 128] to [T, 64].

[0037] The training process of the feature extraction model is: obtaining a training data set; wherein the training data set includes a plurality of training dry audio; using a vocoder to extract training base audio features of the training dry audio; and using the training base audio features to train the neural network-based feature extraction model. In specific implementation, a training data set containing a plurality of training dry audio is obtained, and a vocoder is used to extract training base audio features of the training dry audio. The specific extraction process is similar to that described in step S102, and will not be repeated here. Each time the training is performed, a predetermined number of training dry audio is selected from the training data set as a training sample, and the feature extraction model is trained using the training base audio features corresponding to the training sample.

[0038] As a preferred embodiment, the training of the neural network-based feature extraction model using the training base audio features includes: determining the sampling weight corresponding to the training dry audio; in the process of each training, the training sample for this time is selected from the training data set according to the sampling weight, and the neural network-based feature extraction model is trained using the training base audio features corresponding to the training sample.

[0039] It should be noted that the frequency of human voice is between 60Hz and 1200Hz, and the range is relatively wide. The acoustic characteristics corresponding to the different singing pitches of a person also change accordingly, so the training data set needs to cover the training dry sound frequency of 60Hz-1200Hz. However, if the training dry sound frequency is collected without special rules, the pitch is mainly concentrated between 120Hz and 480Hz, so it is easy to cause the problem of unbalanced training samples when selecting training samples in each training, which leads to the unsatisfactory adaptability of the feature extraction model obtained by training. In a specific implementation, a corresponding sampling weight is assigned to each training dry sound frequency in the training data set, that is, the probability of being selected in each training process. A lower weight is assigned to the training dry sound frequency between 120Hz and 480Hz, so that the training dry sound frequency in this range is more difficult to be selected although the number is large. A higher weight is assigned to the training dry sound frequency between 60Hz and 480Hz and between 480Hz and 1200Hz, so that the training dry sound frequency in this range is more likely to be selected although the number is small, achieving balanced data distribution of training samples.

[0040] As a feasible implementation, the determining the sampling weight corresponding to the training dry sound frequency comprises: constructing a pitch distribution vector corresponding to the training dry sound frequency according to the number of frames corresponding to each pitch contained in the training dry sound frequency; constructing a pitch distribution matrix according to the pitch distribution vectors corresponding to all the training dry sound frequencies in the training data set; calculating a weight matrix based on the pitch distribution matrix and a target matrix, and determining the sampling weight corresponding to the training dry sound frequency based on the weight matrix; wherein each element in the target matrix is a target value, and the target value is the ratio between the sum of all elements in the pitch distribution matrix and the number of training dry sound frequencies in the training data set.

[0041] In a specific implementation, a corresponding pitch distribution vector is constructed for each training dry sound frequency, the dimension n of the pitch distribution vector is the covered pitch range, for example, 1200 dimensions, and each element is the number of frames of the corresponding pitch contained in the training dry sound frequency. The length of each frame can be 16 milliseconds. Secondly, a pitch distribution matrix A is constructed according to the pitch distribution vectors corresponding to all the training dry sound frequencies in the training data set. For example, the training data set contains m training dry sound frequencies, so the size of the pitch distribution matrix is mxn. Then, the weight matrix X is calculated by the following formula:

[0042] A T X=B,s.t.X i >0;

[0043] Wherein, X is the weight matrix, the size of which is mx1, X iis the sample weight corresponding to the i-th training dry sound audio, B is a target matrix with a size of n x 1, each element in B is equal, B = sum(A) / m. i

[0044] As a preferred embodiment, in the process of each training, a target loss function is constructed based on the similarity between the enhanced audio features of adjacent frames and the similarity between the enhanced audio features of non-adjacent frames output by the feature extraction model; wherein the function value of the target loss function is positively correlated with the similarity between the enhanced audio features of adjacent frames and negatively correlated with the similarity between the enhanced audio features of non-adjacent frames. In specific implementation, a supervision function of contrast loss can be used as a loss function to train the feature extraction model, and the basic idea is that the smaller the similarity between the enhanced audio features of the same phonemes is, the better, that is, the smaller the similarity between the enhanced audio features of adjacent frames is, the better, and the larger the similarity between the enhanced audio features of different phonemes is, the better, that is, the larger the similarity between the enhanced audio features of non-adjacent frames is, the better.

[0045] Preferably, the target loss function is specifically:

[0046]

[0047] wherein z i is the enhanced audio feature of the i-th frame output by the feature extraction model, z i+1 is the enhanced audio feature of the i+1-th frame output by the feature extraction model, z j is the enhanced audio feature of the j-th frame output by the feature extraction model, sim(z i ,z i+1 ) is the cosine similarity distance between z i and z i+1 , and sim(z i ,z j ) is the cosine similarity distance between z i and z j .

[0048] Based on the above target loss function for training the feature extraction model, the similarity between the enhanced audio features of adjacent frames can be higher, that is, the enhanced audio features of the same phonemes have clustering effect, and the enhanced audio features between different phonemes have better distinguishability.

[0049] S104: aligning the first enhanced audio feature and the second enhanced audio feature by using a dynamic time warping method to determine the corresponding relationship between each frame of audio in the standard dry sound audio and each frame of audio in the target dry sound audio;

[0050] ​S105: determining the timestamp information of the target dry audio based on the timestamp information of the standard dry audio and the correspondence.

[0051] In a specific implementation, the first enhanced audio features and the second enhanced audio features are aligned at a frame level by using a method of dynamic time warping (DTW). The principle is to calculate the Euclidean distance between the first enhanced audio features corresponding to each frame of audio in the target dry audio and the second enhanced audio features corresponding to the corresponding frame of audio and the adjacent frame of audio in the standard dry audio, and to establish the correspondence between the two corresponding frames with the smallest Euclidean distance. For example, the first enhanced audio features corresponding to the target dry audio and the second enhanced audio features corresponding to the standard dry audio both include T frames. A warping matrix of T x T is obtained by using the dynamic time warping method, which records the correspondence between the i-th frame in the target dry audio and the j-th frame in the standard dry audio, as shown in Table 1:

[0052] Table 1

[0053] 1 2 3 4 5 6 7 1 (1,1) 2 (2,2) 3 (3,3) (3,4) 4 (4,5) 5 (5,5) 6 (6,6) 7 (7,7)

[0054] In Table 1, the column represents the first enhanced audio features corresponding to the target dry audio sung by the user, and the row represents the second enhanced audio features corresponding to the standard dry audio of the original song. (3, 3) and (3, 4) represent that the 3rd frame of the target dry audio corresponds to the 3rd frame and the 4th frame of the standard dry audio, (4, 5) and (5, 5) represent that the 4th frame and the 5th frame of the target dry audio correspond to the 5th frame of the standard dry audio, and so on. Further, since the timestamp information of the standard dry audio is known, the timestamp information of the target dry audio can be deduced by the above correspondence.

[0055] The method for determining the timestamp of the dry audio provided in the embodiments of the present application extracts the features with stronger robustness and continuity corresponding to the target dry audio of the user and the standard dry audio of the original song by using the feature extraction model based on the neural network, instead of the acoustic features such as the mel-frequency cepstral coefficient in the related art, which has more obvious advantages in overcoming noise interference and non-standard pronunciation scenes. Further, the dynamic time warping is performed on the first enhanced audio features corresponding to the target dry audio and the second enhanced audio features corresponding to the standard dry audio, so that the frame-level correspondence between the target dry audio and the standard dry audio can be obtained. Since the timestamp information of the standard dry audio is known, the timestamp information of the target dry audio can be deduced by the above correspondence. It can be seen that the method for determining the timestamp of the dry audio provided in the embodiments of the present application improves the accuracy of determining the timestamp of the dry audio.

[0056] The following introduces a dry audio timestamp determination device provided in an embodiment of the present application. The following description of a dry audio timestamp determination device can be mutually referred to the above description of a dry audio timestamp determination method. Specifically, the dry audio timestamp determination device comprises:

[0057] The first obtaining module is configured to obtain a target dry audio and a standard dry audio corresponding to the target dry audio.

[0058] The first extracting module is configured to extract, by using a vocoder, a first basic audio feature of the target dry audio and a second basic audio feature of the standard dry audio.

[0059] The input module is configured to input the first basic audio feature and the second basic audio feature into a neural network-based feature extraction model respectively to obtain a first enhanced audio feature corresponding to the first basic audio feature and a second enhanced audio feature corresponding to the second basic audio feature respectively.

[0060] The alignment module is configured to align the first enhanced audio feature and the second enhanced audio feature by using a dynamic time warping method to determine a corresponding relationship between each frame of audio in the standard dry audio and each frame of audio in the target dry audio.

[0061] The determination module is configured to determine timestamp information of the target dry audio based on timestamp information of the standard dry audio and the corresponding relationship.

[0062] The dry audio timestamp determination device provided in the embodiment of the present application extracts, by using a neural network-based feature extraction model, a feature representation with stronger robustness and continuity corresponding to a target dry audio of a user and a standard dry audio of an original singer respectively, instead of acoustic features such as mel-frequency cepstral coefficients in related technologies, which has more obvious advantages in overcoming noise interference and non-standard pronunciation scenarios. Further, dynamic time warping is performed on a first enhanced audio feature corresponding to the target dry audio and a second enhanced audio feature corresponding to the standard dry audio, so that a frame-level corresponding relationship between the target dry audio and the standard dry audio can be obtained. Since the timestamp information of the standard dry audio is known, the timestamp information of the target dry audio can be deduced by using the above corresponding relationship. Therefore, the dry audio timestamp determination device provided in the embodiment of the present application improves the accuracy of determining the timestamp of the dry audio.

[0063] In the above embodiment, as a preferred implementation manner, the first extracting module is specifically a module configured to extract a spectral envelope of the target dry audio as the first basic audio feature and extract a spectral envelope of the standard dry audio as the second basic audio feature by using a vocoder.

[0064] On the basis of the above-mentioned embodiments, as a preferred implementation, the first extraction module comprises:

[0065] The first extraction unit is configured to extract a target fundamental frequency of the target dry audio by using the vocoder, perform inverse Fourier transform on a power spectrum of the target dry audio to obtain a target cepstrum, and perform low-pass processing on the target cepstrum based on the target fundamental frequency to obtain a spectral envelope of the target dry audio as a first basic audio feature;

[0066] The second extraction unit is configured to extract a standard fundamental frequency of the standard dry audio by using the vocoder, perform inverse Fourier transform on a power spectrum of the standard dry audio to obtain a standard cepstrum, and perform low-pass processing on the standard cepstrum based on the standard fundamental frequency to obtain a spectral envelope of the standard dry audio as a second basic audio feature.

[0067] On the basis of the above-mentioned embodiments, as a preferred implementation, the method further comprises:

[0068] The second acquisition module is configured to acquire a training data set; wherein the training data set comprises a plurality of training dry audios;

[0069] The second extraction module is configured to extract training basic audio features of the training dry audios by using the vocoder;

[0070] The training module is configured to train the neural network-based feature extraction model by using the training basic audio features.

[0071] On the basis of the above-mentioned embodiments, as a preferred implementation, the training module comprises:

[0072] The determination unit is configured to determine a sampling weight corresponding to the training dry audio;

[0073] The training unit is configured to select, in each training process, a training sample for the current training from the training data set according to the sampling weight, and train the neural network-based feature extraction model by using the training basic audio features corresponding to the training sample.

[0074] On the basis of the above-mentioned embodiments, as a preferred implementation, the determination unit comprises:

[0075] The first construction sub-unit is configured to construct a pitch distribution vector corresponding to the training dry audio according to a number of frames corresponding to each pitch included in the training dry audio;

[0076] The second construction sub-unit is configured to construct a pitch distribution matrix according to the pitch distribution vectors corresponding to all the training dry audios in the training data set;

[0077] The determining sub-unit is configured to calculate a weight matrix based on the pitch distribution matrix and a target matrix, and determine a sampling weight corresponding to the training dry audio based on the weight matrix, wherein each element in the target matrix is a target value, and the target value is a ratio between a sum of all elements in the pitch distribution matrix and a number of training dry audios in the training data set.

[0078] In addition to the above embodiments, as a preferred implementation, the training module further comprises:

[0079] The constructing unit is configured to construct a target loss function based on similarities between the enhanced audio features of adjacent frames and similarities between the enhanced audio features of non-adjacent frames output by the feature extraction model in each training process, wherein a function value of the target loss function is positively correlated with the similarities between the enhanced audio features of adjacent frames and negatively correlated with the similarities between the enhanced audio features of non-adjacent frames.

[0080] In addition to the above embodiments, as a preferred implementation, the target loss function is specifically:

[0081]

[0082] wherein z i is the enhanced audio feature of the i-th frame output by the feature extraction model, z i+1 is the enhanced audio feature of the i+1-th frame output by the feature extraction model, z j is the enhanced audio feature of the j-th frame output by the feature extraction model, sim(z i ,z i+1 ) is a cosine similarity distance between z i and z i+1 , and sim(z i ,z j ) is a cosine similarity distance between z i and z j .

[0083] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0084] The present application also provides an electronic device, referring to Figure 3 , a structural diagram of an electronic device 30 provided by an embodiment of the present application, as shown in Figure 3 , can include a processor 31 and a memory 32.

[0085] The processor 31 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 31 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 31 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 31 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 31 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0086] The memory 32 can include one or more computer-readable storage media, which can be non-transitory. The memory 32 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In this embodiment, the memory 32 is used at least to store the following computer program 321, wherein the computer program is loaded and executed by the processor 31, and can implement the related steps in the timestamp determination method of the dry sound audio performed by the electronic device side disclosed in any of the preceding embodiments. In addition, the resources stored by the memory 32 can also include an operating system 322 and data 323, etc., and the storage mode can be temporary storage or permanent storage. The operating system 322 can include Windows, Unix, Linux, etc.

[0087] In some embodiments, the electronic device 30 can also include a display screen 33, an input / output interface 34, a communication interface 35, a sensor 36, a power supply 37, and a communication bus 38.

[0088] Of course, Figure 3 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present application. In actual applications, the electronic device can include more or fewer components than those shown, or combine some components. Figure 3

[0089] ​In another exemplary embodiment, a computer readable storage medium including program instructions is also provided, which when executed by a processor implement the steps of the timestamp determination method of the dry sound audio performed by the electronic device of any of the above embodiments.

[0090] The various embodiments described in the specification are presented by way of example only and are not intended to limit the scope of the application. The embodiments disclosed should not be interpreted as limiting the scope of the application. The description of the embodiments is merely provided for the purposes of illustration and explanation. It is provided in this regard that the embodiments disclosed should not be interpreted as limiting the scope of the application. It will be apparent to one of ordinary skill in the art that many modifications, substitutions and alterations can be made to the embodiments disclosed without departing from the spirit and scope of the application. The scope of the application is not to be limited by the embodiments disclosed in the specification, but only by the claims that will be included in the final patent.

[0091] It should also be noted that, in this specification, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action, without necessarily implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element(s).

Claims

1. A method of timestamp determination for dry audio, characterized by, The method comprises: obtaining a target dry audio and a standard dry audio corresponding to the target dry audio; extracting, by using a vocoder, first basic audio features of the target dry audio and second basic audio features of the standard dry audio; inputting the first basic audio features and the second basic audio features into a neural network-based feature extraction model respectively to obtain first enhanced audio features corresponding to the first basic audio features and second enhanced audio features corresponding to the second basic audio features respectively; aligning the first enhanced audio features and the second enhanced audio features by using a dynamic time warping method to determine a corresponding relationship between each frame of audio in the standard dry audio and each frame of audio in the target dry audio; determining timestamp information of the target dry audio based on timestamp information of the standard dry audio and the corresponding relationship; wherein the timestamp determination method further comprises: in each process of training the feature extraction model, constructing a target loss function based on similarities between adjacent frame enhanced audio features and similarities between non-adjacent frame enhanced audio features output by the feature extraction model; wherein the function value of the target loss function is positively correlated with the similarities between adjacent frame enhanced audio features and negatively correlated with the similarities between non-adjacent frame enhanced audio features.

2. The timestamp determination method of claim 1, wherein, The method further comprises: extracting, by using a vocoder, a spectral envelope of the target dry audio as the first basic audio features and a spectral envelope of the standard dry audio as the second basic audio features.

3. The timestamp determination method of claim 2, wherein, The method further comprises: extracting, by using a vocoder, a target fundamental frequency of the target dry audio, performing inverse Fourier transform on a power spectrum of the target dry audio to obtain a target cepstrum, and performing low-pass processing on the target cepstrum based on the target fundamental frequency to obtain the spectral envelope of the target dry audio as the first basic audio features; extracting, by using a vocoder, a standard fundamental frequency of the standard dry audio, performing inverse Fourier transform on a power spectrum of the standard dry audio to obtain a standard cepstrum, and performing low-pass processing on the standard cepstrum based on the standard fundamental frequency to obtain the spectral envelope of the standard dry audio as the second basic audio features.

4. The timestamp determination method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining a training data set; wherein the training data set comprises a plurality of training dry audios; extracting, by using a vocoder, training basic audio features of the training dry audios; training a neural network-based feature extraction model by using the training basic audio features.

5. The timestamp determination method of claim 4, wherein, The method further comprises: determining sampling weights corresponding to the training dry audios; During each training, training samples for the training are selected from the training data set according to the sampling weights, and a neural network-based feature extraction model is trained using training base audio features corresponding to the training samples.

6. The timestamp determination method of claim 5, wherein, The determining of the sampling weights corresponding to the training dry audio comprises: constructing a pitch distribution vector corresponding to the training dry audio according to the number of frames corresponding to each pitch included in the training dry audio; constructing a pitch distribution matrix according to the pitch distribution vectors corresponding to all the training dry audios in the training data set; calculating a weight matrix based on the pitch distribution matrix and a target matrix, and determining the sampling weights corresponding to the training dry audio based on the weight matrix, wherein each element in the target matrix is a target value, and the target value is a ratio between the sum of all elements in the pitch distribution matrix and the number of training dry audios in the training data set.

7. The timestamp determination method of claim 4, wherein, The target loss function is specifically: ； wherein, is an enhanced audio feature for the i-th frame output by the feature extraction model, is an enhanced audio feature for the i+1-th frame output by the feature extraction model, is an enhanced audio feature for the j-th frame output by the feature extraction model, is a cosine similarity distance between and is a cosine similarity distance between and 8. An electronic device, comprising: comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the dry audio timestamp determination method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and when executed by the processor, the steps of the dry audio timestamp determination method according to any one of claims 1 to 7 are implemented.

Images