A method for automatic speech editing tasks

Abstract

The present invention discloses a method for automatic speech editing and a neural network model training method, comprising: adopting a data format of first overall and then local, and adopting a training speed of first fast and then slow to train the neural network model; the present application provides a method for automatic speech editing and a neural network model training method, the method comprising: extracting audio from original audio and video and preprocessing it to obtain preprocessed audio; extracting multiple acoustic features from the preprocessed audio; inputting the acoustic features into a trained neural network model for speech detection, and outputting the results; and automatically editing the original audio and video according to the output results of the trained neural network model. According to the present invention, speech segments and non-speech segments in audio and video are quickly detected, and the original audio and video are automatically and artistically edited. The device can achieve an effect close to that of manual editing and greatly shorten the editing time.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, speech detection and audio and video editing, and in particular to a method for automatic speech editing tasks and a neural network model training method. Background Art

[0002] With the widespread adoption of the internet and the rapid development of the digital media industry, the volume of audio and video media has grown exponentially, and their post-production is inseparable from editing. Editing is an artistic and highly stylized process, with different types of media requiring varying styles and requirements. Furthermore, compared to other tasks like color grading and subtitling, editing often consumes the most manpower and time during post-production. Language-based audio and video content consistently accounts for a high proportion of broadcast and television content, and their length requires comprehensive review by editors before commencing editing. Therefore, editing language-based audio and video consumes a significant amount of manpower and time.

[0003] Because editing is closely related to contextual information, directly applying speech endpoint detection to audio editing performs poorly, resulting in abrupt transitions and low-quality editing. Current automatic editing systems primarily rely on machine learning algorithms, such as the Hidden Malcolm Model (HMM) combined with the Vibit algorithm. However, due to the limitations of the Markov property, the HMM's ability to extract long-term sequence information is limited, resulting in editing results that fail to incorporate contextual connections between audio and video. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a method for automatic speech editing and a neural network model training method, which can quickly detect speech segments and non-speech segments in audio and video, and automatically and artistically edit the original audio and video. The device can achieve an effect close to that of manual editing and significantly shorten the editing time. In order to achieve the above-mentioned purpose and other advantages of the present invention, a method for automatic speech editing is provided, comprising:

[0005] S1. Establishing a neural network model, wherein the neural network model includes a convolutional layer, a bidirectional recurrent neural network and a feedforward neural network;

[0006] S2, training the neural network model in step S1;

[0007] S3. Automatically edit audio and video based on the neural network model, wherein the automatic audio and video editing comprises the following steps:

[0008] S11, extracting audio from the original audio and video and preprocessing it to obtain preprocessed audio;

[0009] S12, extracting multiple acoustic features from the audio preprocessed in step S11;

[0010] S13: Input the acoustic features in step S12 into the trained neural network model to perform speech detection and output the results;

[0011] S14. Automatically edit the original audio and video according to the result output in step S13.

[0012] Preferably, the convolution layer includes multiple neural networks, and the convolution layer is composed of multiple convolution neural networks that perform convolution operations on different acoustic features respectively, and the convolution layer is combined with a bidirectional recurrent neural network and a feedforward neural network to obtain a neural network model for the automatic speech editing task. The bidirectional recurrent neural network includes a layer of forward recurrent neural network and a layer of backward recurrent neural network.

[0013] Preferably, the convolutional layer activates the output of each neural network using an activation function and then stacks them to obtain the final output result of the convolutional layer. The output result of the convolutional layer is then input into a bidirectional recurrent neural network to obtain the output result of the bidirectional recurrent neural network. The output result of the bidirectional recurrent neural network is then input into a feedforward neural network to obtain the output result of the feedforward neural network. The output result of the feedforward neural network is activated and classified using a Softmax activation function to obtain the final output result of the neural network model.

[0014] Preferably, in step S2, a binary cross entropy loss function is used to calculate the loss value, the gradient parameters are updated through the back propagation algorithm, and then the trained model parameters are saved. During the model training process, the model with the best performance on the validation set is selected and tested on the test set, and finally the model parameters with the best performance on the test set are saved.

[0015] Preferably, the acoustic features in step S3 include logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate, and the three acoustic features are normalized separately to facilitate subsequent neural network calculations, and the extracted logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate have the same sequence length.

[0016] Preferably, the output sequence of the neural network model and the input feature sequence have the same length, and the number of neural networks in the model convolutional layer and the number of input features remain the same, and the output result of the neural network model is a binary classified 1-dimensional time series.

[0017] A training method based on a neural network model uses a larger batch of training set data and a larger learning rate for the first round of training. When the loss function is close to convergence, the first round of training is stopped. A smaller batch of training set data and a smaller learning rate are used for the second round of training. When the loss function converges, the second round of training is stopped. This training method is used to continuously adjust the hyperparameters, and finally the model parameters with the best performance on the test set are saved.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) Compared to traditional machine learning algorithms, the method and neural network model training method for automatic speech editing in the present invention can more effectively read the contextual information of audio and video, making the editing process more intelligent. In addition, compared to traditional speech endpoint detection methods, the method of the present invention retains different degrees of margin before and after the speech segmentation, making the editing results more natural and artistic.

[0020] (2) The device of the present invention has high computational efficiency, and it takes less than 1 minute to compute 60 minutes of audio. Compared with traditional manual editing, it can save a lot of manpower and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the network structure of a neural network model according to the method for automatic speech editing and the neural network model training method of the present invention;

[0022] Figure 2 A schematic diagram of the convolutional layer structure of the method for automatic speech editing and the neural network model training method according to the present invention;

[0023] Figure 3 A schematic diagram of the bidirectional gated recurrent neural network structure of the method for automatic speech editing and the neural network model training method according to the present invention;

[0024] Figure 4 A flowchart of a method for automatic speech editing and a neural network model training method according to the present invention;

[0025] Figure 5 Schematic diagram of a labeling process of a data set in an embodiment of a method for automatic speech editing and a neural network model training method according to the present invention;

[0026] Figure 6 A flowchart of an automatic audio and video editing method based on a neural network according to the method for automatic speech editing tasks and a neural network model training method of the present invention;

[0027] Figure 7 A schematic diagram of the structure of a neural network-based automatic audio and video editing device according to the method for automatic speech editing and the neural network model training method of the present invention;

[0028] Figure 8 This is a comparison chart of the model editing and manual editing results of the method for automatic speech editing tasks and the neural network model training method according to the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] The following first introduces the relevant terms and related concepts such as neural networks involved in the embodiments of this application.

[0031] (1) Feedforward Neural Network

[0032] A feedforward neural network (FNN) is a unidirectional, multi-layered structure consisting of an initial input layer, an intermediate hidden layer, and a final output layer. The hidden layer can be single or multi-layered. Each layer contains several neurons, and the neurons between layers are fully connected. Neurons in each layer can receive signals from neurons in the previous layer and generate outputs to the next layer.

[0033] (2) Convolutional Neural Network

[0034] A convolutional neural network (CNN) is a feedforward neural network with a convolutional structure. It uses a filter (consisting of a convolution kernel and a bias vector) to continuously slide and perform a Hadamard product operation on data of the corresponding window size, then sums the result with the bias vector to obtain a new value. The calculation process is shown in Equation (1). CNN has the characteristic of weight sharing, with fewer learning parameters, which can effectively avoid overfitting and has high computational efficiency. In addition, due to the existence of a local receptive field, CNN can combine the representation information of surrounding data, and as the convolution layer deepens, it can abstract higher-dimensional features of the data.

[0035]

[0036] where ⊙ is the Hadamard product.

[0037] (3) Recurrent Neural Network

[0038] A recurrent neural network (RNN) is a type of neural network with short-term memory. It memorizes previous hidden layer information and uses this information to influence the output of the current node. In other words, the nodes in an RNN's hidden layer are connected, and the hidden layer input includes not only the output of the input layer but also the output of the previous hidden layer. While RNNs can theoretically process sequences of data of any length, traditional RNNs cannot autonomously update the hidden layer state, resulting in vanishing or exploding gradients, making them ineffective in capturing relevant information from long time series. The subsequent development of gated recurrent units (GRUs) and LSTMs incorporates the concept of control gates to update the hidden layer state, effectively avoiding these vanishing or exploding gradients and enabling them to better integrate information from long time series.

[0039] (4) Gated recurrent unit

[0040] Gated Recurrent Units (GRU)

[0041] The GRU is a variant of the RNN. It features a reset gate and an update gate to update the hidden layer state, enabling it to effectively capture long-term dependencies and, to a certain extent, avoiding the vanishing gradient issues common in classic RNNs. It is similar to the Long Short-Term Memory (LSTM) network, but differs in that it combines the LSTM's input gate and forget gate into an update gate, making it less computationally complex than the LSTM. The GRU's update gate determines how much previously hidden information is retained at the current node, while the reset gate controls how much information is forgotten at the current node.

[0042] (4) Loss function

[0043] The loss function is used to measure the difference between the model's predicted value f(x) and the true value Y. It is a non-negative function, usually represented by L(Y, f(x)). The smaller the loss function, the closer the model's predicted value is to the true value. The training process of a neural network model uses an optimizer and a backpropagation algorithm to iteratively update the model parameters, continuously reducing the loss function until the loss value fully converges, at which point the model training process ends.

[0044] (5) Backpropagation algorithm

[0045] The back propagation algorithm (BP) is a learning algorithm suitable for multilayer neural networks. It is based on the gradient descent method. The BP algorithm's learning process consists of forward propagation and backward propagation. During the forward propagation process, input information passes through the input layer, then the hidden layer, and is processed layer by layer before being transmitted to the output layer. The difference between the model output and the true value is calculated as the objective function. Backward propagation then proceeds to calculate the partial derivative of the objective function with respect to each neuron weight layer by layer. This gradient of the objective function with respect to the weight vector is then used as the basis for modifying the weights. Network training is completed during this weight modification process. Training ends when the error reaches the desired range.

[0046] Reference Figure 1-8 , a method for an automatic speech editing task, comprising: S1, establishing a neural network model, wherein the neural network model comprises a convolutional layer, a bidirectional gated recurrent neural network (Bi-GRU) and an FNN;

[0047] S2. Training the neural network model in step S1, using the data format of first global and then local to train the neural network model, while the training process is fast at first and then slow.

[0048] In the real-time example of this application, the audio of the training set was manually recorded, and the validation set and test set were each intercepted from the ChiME-5 dataset for one hour. Professional editors were invited to label the datasets. The training set was divided into longer audio as the data for each batch, and a larger learning rate was set to perform the first round of training on the neural network model. In the second round of training, the training set was divided into shorter audio as the data for each batch, and a smaller learning rate was used for training.

[0049] S3. Automatically edit audio and video based on the neural network model, wherein the automatic audio and video editing comprises the following steps:

[0050] S11. Extracting audio from the original audio and video and preprocessing it to obtain preprocessed audio. In an embodiment of the present application, the preprocessing process includes downsampling the audio in the original audio and video to obtain preprocessed audio, and if the audio in the original audio and video is multi-channel, converting it to mono;

[0051] S12, extracting multiple acoustic features from the audio preprocessed in step S11;

[0052] S13: Input the acoustic features in step S12 into the trained neural network model to perform speech detection and output the results;

[0053] S14. Automatically edit the original audio and video according to the result output in step S13.

[0054] Furthermore, the convolutional layer includes multiple neural networks, which are used to perform convolution operations on different acoustic features respectively to form a convolutional layer, and the convolutional layer is combined with Bi-GRU and a layer of FNN to obtain a neural network model for the automatic speech editing task.

[0055] Furthermore, the convolutional layer activates each convolutional neural network output using the LeakyReLU function and then stacks them to obtain the final output of the convolutional layer. The convolutional layer output is then input into the Bi-GRU to obtain the Bi-GRU output. The Bi-GRU output is then input into the FNN to obtain the FNN output. The FNN output is activated and classified using the Softmax activation function to obtain the final output of the neural network model.

[0056] Furthermore, the convolutional layer is used to extract and abstract high-dimensional information of acoustic features, and the forward gated recurrent unit is used to read the time series information before the current node, and the backward gated recurrent unit is used to read the time series information after the current node.

[0057] Furthermore, in step S2, the binary cross entropy loss function is used to calculate the loss value, the gradient parameters are updated through the back propagation algorithm, and then the trained model parameters are saved. During the model training process, the model with the best performance on the validation set is selected for testing on the test set, and finally the model parameters with the best performance on the test set are saved. The training method of the neural network model is different from the traditional mini-batch gradient descent (mini batch). By changing the batch size and learning rate, the Bi-GRU can better combine time series data for prediction.

[0058] Furthermore, the acoustic features in step S3 include logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate, and the three acoustic features are normalized separately to facilitate subsequent neural network calculations, and the extracted logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate have the same time series length. The trained neural network model can identify the speech and non-speech parts in the audio.

[0059] Furthermore, the output sequence of the neural network model and the input feature sequence have the same length, and the number of neural networks in the convolutional layer of the model and the number of input features remain the same. The output result of the neural network model is a binary 1-dimensional time series. In an embodiment of the present application, the audio in the original audio and video is downsampled to the same frequency as the output of the model, and then the original audio and video are edited according to the output of the model.

[0060] A training method based on a neural network model uses a larger batch of training set data and a larger learning rate for the first round of training. When the loss function is close to convergence, the first round of training is stopped; a smaller batch of training set data and a smaller learning rate are used for the second round of training. When the loss function converges, the second round of training is stopped; this training method is used to continuously adjust the hyperparameters, and finally the model parameters with the best performance on the test set are saved.

[0061] Specifically, according to the output of the model, the clips that need to be retained in the original audio and video are selected, and then they are recombined to obtain the edited audio and video.

[0062] Also included is an automatic audio and video editing device based on a neural network model, the device comprising:

[0063] Preprocessing module: preprocesses the original audio and video to obtain preprocessed audio;

[0064] Feature extraction module: used to extract multiple acoustic features from the preprocessed audio;

[0065] Speech detection module: used to input the multiple acoustic features into the trained neural network model to perform speech detection and output the results;

[0066] Editing module: used to automatically edit the original audio and video according to the output results of the trained neural network model.

[0067] Example 1

[0068] See also Figure 1 , Figure 1 This is a schematic diagram of the network structure of the neural network model used in the real-time example of this application. The neural network model used in the real-time example of this application mainly consists of three parts. The first part is a convolutional layer composed of multiple CNNs, the second part is a Bi-GRU composed of a forward GRU layer and a backward GRU layer, and the third part is a FNN layer.

[0069] For details on the convolutional layer structure in the neural network model in the application embodiment, please refer to Figure 2. First, for the logarithmic Mel spectrum features, a two-layer two-dimensional CNN is used for convolution operations. The first layer uses 8 1*40 filters for convolution operations. The stride of the convolution is set to 1, and the first layer of CNN is used to abstract the features at each time node. The second layer uses 3 5*8 filters for convolution operations. The stride of the convolution is still set to 1, and the padding is set to 2. The padding is set to keep the sequence length of the CNN output consistent with the input sequence length, which is convenient for the final editing. The second layer of CNN can combine the feature information of some surrounding data. In addition, considering that the audio features are highly nonlinear, the LeakyReLU function is used to activate the output of the convolution layer. This activation function has a parameter α to be learned in the negative interval, which effectively avoids the phenomenon of the gradient being 0. The formula of LeakyReLU is as follows:

[0070]

[0071] For the short-term energy features, a one-dimensional CNN layer is used for convolution, with a filter size of 5, a stride of 1, a padding of 2, and the activation function LeakyReLU. For the short-term zero-crossing rate, a one-dimensional CNN layer is used for convolution, with a filter size of 5, a stride of 1, a padding of 2, and the activation function LeakyReLU. Finally, the convolutional layer fuses the outputs of each CNN layer to obtain the final output, which is then input into the Bi-GRU.

[0072] The bidirectional gated recurrent neural network structure in the neural network model in the embodiment of this application can be found in Figure 3 . Bi-GRU combines a layer of forward GRU, a layer of backward GRU and a layer of FNN, so that the output of each time node of the model can read the forward and backward hidden layer state information. In order to prevent the model from over-learning and reduce unnecessary parameters, the width of the GRU hidden layer is set to 2, and the random deactivation rate (dropout) of the hidden layer unit is set to 0.2. The input of FNN is the hidden layer of the forward GRU and backward GRU, the number of input nodes is 4, and the output is 1-dimensional. The Softmax activation function is used to activate the output result of FNN to obtain the final output result of the model. The Softmax activation function can convert numerical values into probability distribution. The formula of the Softmax activation function is as follows:

[0073]

[0074] where z i is the output value of the i-th node, and C is the number of output nodes, that is, the number of classification categories.

[0075] The convolutional layer is combined with Bi-GRU and FNN to obtain a neural network model for the automatic editing task.

[0076] Example 2

[0077] See also Figure 4 , Figure 4 The figure is a flow chart of the neural network model training method in the embodiment of the present application. The training set is recorded manually, with a duration of about 20 minutes, of which speech accounts for about 60%, and the language is Chinese. During the recording, there is a pause of 3 seconds between sentences and 30 seconds between paragraphs. At the same time, some noise is artificially emitted, and urban street noise is added to it to simulate the outdoor recording environment. Compared with stable white noise, irregular noise is more disruptive. The processed speech signal-to-noise ratio is about 0db, which is used to increase the difficulty of model training and enhance the robustness of the model. The validation set and test set use the CHiME-5 dataset, which is a speech recognition challenge dataset in a noisy environment. The dataset contains real, simulated and clean recordings, of which the real recordings are recorded by 6 four-channel microphone arrays. The content is a family dinner, the language is English, and the duration of each recording is about 120 minutes. The audio recorded by the microphone farthest from the speaker's position is selected, and 60 minutes of each are intercepted as the validation set and test set, of which the speech part accounts for about 75%. The audio in the dataset has a low speech volume and is subject to interference from a large amount of irregular noise and far-field reverberation. Approximately 15% of the speech reaches a mean opinion score of 2, meaning that it requires considerable concentration to hear clearly. This data is used to examine the model's performance in extreme environments.

[0078] The labeling is done manually. Please refer to Figure 5 In this embodiment, we invited editors from Shanghai Radio and Television Station to annotate the data set. The audio is displayed in the form of a waveform. After manual listening, the audio to be retained is selected. The retained part is a positive example with a value of 1, otherwise it is a negative example with a value of 0. Figure 6 As shown. The following are the annotation habits of editors: When the pause between sentences exceeds 2 seconds, the editor will split it, and keep 0.5 to 1 second at both ends of the sentence as a buffer. When there is a long pause between speech paragraphs, they will keep about 1 to 2 seconds of space at the beginning and end of the paragraph respectively. When the sentences are continuous, for example, the pause between sentences is less than 2 seconds, the editor will not split them. In addition, it will delete the noise caused by humans, but will not delete the environmental noise that appears in the speech segment. The audio after manual editing has a smooth transition without obvious pauses, and the listener can distinguish different paragraphs. The original labels are saved in the form of binary time series with the same number as the audio sampling points, and then they are downsampled to the same length as the feature sequence.

[0079] To enable the model in this embodiment to better incorporate the connections between audio features for editing, we trained the model using a data format that first analyzes the entire audio and then the local audio, similar to the growth process of a tree, from the trunk to the branches. Furthermore, the training process starts fast and then slows down, with the entire audio fast and the local audio slow. First, the audio in the training set is divided into batches of 95 seconds. Adam is used as the optimizer with a learning rate of 0.01 for the first round of training. The goal of training is to enable the model to learn the overall correlation features, namely the connections between paragraphs. After a certain number of iterations, if the model has not fully converged, training is stopped. If the model is trained using large batches of data until full convergence, although the optimizer may have found the global minimum of the gradient, the model will be overfitted and perform poorly on the validation set. In the second round of training, the audio is divided into batches of 21 seconds. Adam is also used as the optimizer, but the initial learning rate is reduced to 0.001. In addition, an exponential learning rate decay is set with a decay coefficient of 0.95. The goal of this second round of training is to enable the model to focus on details, namely the connections between sentences. Since small batches of data vary significantly, a smaller learning rate can keep model parameters relatively stable. Considering practical engineering applications, the validation and test data sets are not split to ensure the integrity of the test data.

[0080] Compared to traditional fixed-batch mini-batch gradient descent, the neural network model training method in this embodiment increases data diversity by varying the batch size, enabling the model to effectively balance macro and micro information, thereby optimizing the model to its optimal state. During training, hyperparameters are continuously adjusted, and the model that performs best on the validation set is then tested on the test set, and the model parameters are saved.

[0081] Table 1 Comparative experimental results of different training methods

[0082]

[0083] The formula for the accuracy (Acc) evaluation index in Table 1 is as follows:

[0084]

[0085] Among them, TP represents the number of correctly predicted positive samples, TN is the number of correctly predicted negative samples, and R is the total number of original audio samples.

[0086] Experimental results show that the training method used in this example outperforms traditional mini-batch gradient descent, improving the model's accuracy by approximately 3%. The model trained in this example can accurately distinguish between speech and non-speech parts during speech detection tasks, and retains a small amount of audio at both ends of the speech part to improve the fluency of the subsequent editing results.

[0087] Example 3

[0088] like Figure 6 As shown, Figure 6 This is a flow chart of the method for automatic audio and video editing based on a neural network in an embodiment of the present application:

[0089] S11: Sampling the original audio and video to obtain pre-processed audio;

[0090] In a specific implementation of the present invention, the original video or audio is sampled at a sampling frequency of 22050 Hz to obtain preprocessed audio; if the preprocessed audio is multi-channel, it is compressed into a single channel.

[0091] S12: extracting multiple acoustic features from the preprocessed audio;

[0092] In the specific implementation process of the present invention, three types of acoustic features are extracted, including logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate.

[0093] The process of extracting the logarithmic Mel spectrum is as follows: first, the sampled audio is pre-emphasized, and the pre-emphasis coefficient is set to 0.97 to obtain the pre-emphasized audio; the pre-emphasized audio is framed, and the frame length is set to 46ms and the frame shift is set to 23ms to obtain the framed audio; the framed audio is windowed based on the Hamming window to obtain the windowed audio, and the window function adopts the Hamming window function. The formula of the Hamming window is as follows:

[0094]

[0095] The windowed audio is converted from the time domain to the frequency domain using a fast Fourier transform, and then the frequency scale is converted to the Mel scale. The conversion formula is as follows:

[0096]

[0097] Filtering is performed on the Mel scale using 40 equal-area triangular filters to obtain logarithmic Mel spectrum features. After extracting the logarithmic Mel spectrum features, the data of each dimension is normalized. The normalization process is as follows: according to the time dimension, we calculate the mean and standard deviation of the data of each dimension, then subtract the mean from the data of each time unit and divide it by the standard deviation to obtain the normalized data. The formula for the normalization operation is as follows:

[0098]

[0099] where x i is the value at time point i, μ is the mean value, and σ is the standard deviation.

[0100] Short-time energy refers to the energy information contained in a frame of audio. The short-time energy extraction process is as follows: every 512 sampling points are treated as a frame and the frame operation is performed; the window function uses a rectangular window to perform the windowing operation; and the short-time energy value is calculated. The short-time energy calculation process is as follows:

[0101]

[0102] Among them E n is the value of short-time energy, m is the audio frame, and ω(n) is the window function.

[0103] The extracted short-time energy features are then normalized, and the normalization process is the same as step

[00] .

[0104] The short-time zero-crossing rate refers to the number of times the signal passes through zero in each frame. It is higher for unvoiced speech and lower for voiced speech. The short-time zero-crossing rate is calculated as follows: 512 sampling points are considered a frame; the number of times the signal passes through zero in each frame is counted; then, divided by the number of sampling points in each frame, the short-time zero-crossing rate is calculated. The formula for calculating the short-time zero-crossing rate is as follows:

[0105]

[0106] Where m is the audio frame and sgn() is the sign function.

[0107] The extracted short-time zero-crossing rate features are then normalized, and the normalization process is the same as step

[00] .

[0108] By conducting ablation experiments, we can see how different acoustic features perform on the same model.

[0109] Table 2 shows the results of the ablation experiment on the performance of different acoustic features on the same model.

[0110]

[0111] The accuracy in Table 2 is the same as formula (4) in step []

[0112] Experimental results show that the combination of the logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate used in this embodiment enables the performance of the model to reach the best state.

[0113] S13: Inputting the acoustic features into the trained neural network model to perform speech detection and outputting the results;

[0114] In an embodiment of the present application, the extracted logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate are input into the trained neural network model, and a one-dimensional time series output is obtained through calculation by the model. Since the time series output by the model is a probability distribution, the probability distribution is rounded to convert it into the corresponding category. For example, probabilities greater than or equal to 0.5 are regarded as positive examples with a value of 1, and probabilities less than 0.5 are regarded as negative examples with a value of 0, thus obtaining a binary time series. The output sequence has the same length as the original feature sequence to facilitate subsequent editing work.

[0115] S14 automatically edits the original audio and video according to the output result of the model;

[0116] The final output result of the neural network model is a binary time series data of 40 units per second; the original audio and video are downsampled with a sampling frequency of 40 Hz per second to obtain downsampled audio frames, and the downsampled audio frames have the same output frequency as the model; the downsampled results are edited according to the final output result of the neural network model. If the model output is a positive example, the downsampled audio frames are retained accordingly, and then the content of the original audio and video is extracted according to the retained audio frames, and they are recombined to obtain the edited audio and video.

[0117] Example 4

[0118] In the embodiment of the present application, the environment for developing the neural network-based automatic audio and video editing device is as follows:

[0119] The operating environment is Windows 10 pro, the CPU is AMD Ryzen 2700, the GPU is Nvidia GeForce GTX 1080, the memory uses dual-channel 16g DDR4, the development language is Python 3.8, the deep learning framework uses Pytorch 1.9.0 + cuda 11.2, and the development tool is Pycharm.

[0120] See also Figure 7 , Figure 7 Schematic diagram of the structure of the neural network-based automatic audio and video editing device in an embodiment of the present application, the device includes:

[0121] Preprocessing module 201: preprocesses the original audio and video to obtain preprocessed audio;

[0122] Specifically, the wave toolkit is used to downsample the original audio and video with a sampling frequency of 22050 Hz; if the original audio and video are dual-channel, they are compressed into a single channel to obtain the preprocessed audio.

[0123] Feature extraction module 202: used to extract multiple acoustic features from the pre-processed audio;

[0124] Specifically, the preprocessed audio is framed using librosa and soundfile toolkits, with the frame length set to 46ms and the frame shift set to 23ms to obtain the framed audio; the framed audio is windowed based on a Hamming window to obtain the windowed audio, and the window function adopts a Hamming window function; the windowed audio is converted from the time domain to the frequency domain using a fast Fourier transform, and then the frequency scale is converted to the Mel scale; 40 equal-area triangular filters are used for filtering on the Mel scale to obtain logarithmic Mel spectrum features; the wave toolkit is used to frame the audio with every 512 sampling points as a frame; the window function adopts a rectangular window to window the audio; the math toolkit is used to calculate the short-time energy and short-time zero-crossing rate features; the numpy toolkit is used to normalize the values of the logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate to obtain the normalized logarithmic Mel spectrum, short-time energy and short-time zero-crossing rate feature values.

[0125] Speech detection module 203: used to input the multiple acoustic features into the trained neural network model to perform speech detection and output the results;

[0126] Specifically, a neural network model was constructed and trained using the Pytorch framework, and the optimal model parameters were saved. The normalized log-mel spectrum, short-time energy, and short-time zero-crossing rate eigenvalues were input into the saved neural network model for calculation. The model calculations yielded a 1-dimensional time series probability distribution output. The round function in Pytorch was used to round the probability distribution to obtain binary time series data.

[0127] Table 3. The time required for the model to calculate the features of 60 minutes of audio.

[0128]

[0129] CPU represents the time required for CPU calculation, and GPU represents the time required for GPU calculation.

[0130] The results show that the model takes less than 1 minute to calculate 60 minutes of audio, while manual editing takes about 35 minutes.

[0131] Editing module 204: used to automatically edit the original audio and video according to the output results of the trained neural network model;

[0132] Specifically, the audio in the original audio and video is downsampled to the same frequency as the model output using the moviepy.editor toolkit, and the corresponding segments in the original audio and video are extracted according to the binary time series data; the above segments are combined using the tqdm toolkit to obtain the edited audio and video. The editing results of the device in this embodiment can be found in Figure 8 , Figure 8 This is a comparison chart of the model editing and manual editing results in this embodiment. The results show that the device in this embodiment can achieve results that are very similar to manual editing. Through research, it was found that there is a certain fluctuation in the time retained on both sides when manually segmenting the speech, and as the editing work progresses, the fluctuation becomes larger, while the device in this embodiment retains a relatively fixed length of time when segmenting the speech. In a speech segment that was severely interfered with by noise, the editor mistakenly deleted it, but the device in this embodiment correctly retained it. The method and device used in this application have high application value in places such as radio and television stations that require a large amount of editing of voice media. It can also be applied to other fields, such as online classes, conferences and other audio and video.

[0133] The numbers and processing scales described herein are provided to simplify the description of the present invention, and applications, modifications, and variations of the present invention will be readily apparent to those skilled in the art.

[0134] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for automatic speech editing, characterized in that: Including steps: S1. Establishing a neural network model, wherein the neural network model includes a convolutional layer, a recurrent neural network and a feedforward neural network; S2, training the neural network model in step S1; S3. Automatically edit audio and video based on the neural network model, wherein the automatic audio and video editing comprises the following steps: S11, extracting audio from the original audio and video and preprocessing it to obtain preprocessed audio; S12. Extracting a plurality of acoustic features from the audio preprocessed in step S11; the acoustic features including three acoustic features: logarithmic Mel spectrum, short-time energy, and short-time zero-crossing rate; S13, input the acoustic features in step S12 into the trained neural network model for speech detection, and output the results; the convolution layer includes multiple convolutional neural networks, and the convolution layer is composed of multiple convolutional neural networks that perform convolution operations on different acoustic features respectively. The convolution layer is combined with Bi-GRU and a layer of FNN to obtain a neural network model for the automatic speech editing task; the convolution layer activates the output of each convolutional neural network using the LeakyReLU function and then stacks them to obtain the final output result of the convolution layer, and then inputs the output result of the convolution layer into the Bi-GRU to obtain the output result of the Bi-GRU; then, the output result of the Bi-GRU is input into the FNN to obtain the output result of the FNN, and the output result of the FNN is activated and classified using the Softmax activation function to obtain the final output result of the neural network model; S14. Automatically edit the original audio and video according to the result output in step S13.

2. A method for automatic speech editing according to claim 1, characterized in that: The Bi-GRU includes a layer of forward GRU and a layer of backward GRU.

3. A method for automatic speech editing according to claim 2, characterized in that: The convolutional layer is used to extract and abstract the high-dimensional information of the acoustic features, and the forward recurrent neural network is used to read the time series information before the current node, the backward neural network is used to read the time series information after the current node, and the feedforward neural network is used to combine the previous and subsequent time series information.

4. A method for automatic speech editing according to claim 3, characterized in that: In step S2, the binary cross entropy loss function is used to calculate the loss value, and the gradient parameters are updated through the back propagation algorithm. During the model training process, the model with the best performance on the validation set is selected and tested on the test set, and finally the model parameters with the best performance on the test set are saved.

5. A method for automatic speech editing according to claim 4, characterized in that: The three acoustic features are normalized separately to facilitate subsequent neural network calculations, and the extracted logarithmic Mel spectrum, short-time energy, and short-time zero-crossing rate have the same time series length.

6. A method for automatic speech editing according to claim 5, characterized in that: Use a larger batch of training set data and a larger learning rate for the first round of training. When the loss function is close to convergence, stop the first round of training. A second round of training is performed using a smaller batch of training set data and a smaller learning rate. When the loss function converges, the second round of training is stopped. This training method is used to continuously adjust the hyperparameters, and finally the model parameters with the best performance on the test set are saved.

Images