Audio processing method, model training method, device and electronic equipment

Abstract

This application provides an audio processing method, a model training method, an apparatus, and an electronic device. The method includes: acquiring initial audio data, which comprises n frames of audio data; extracting features from the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data; inputting the n sets of feature data into a speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data; determining target audio data based on the n score values ​​and outputting the target audio data, wherein the target audio data is the audio data including audio content in the initial audio data. This application provides an audio processing method, a model training method, an apparatus, and an electronic device that can improve the speech detection effect during speech activity detection.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to an audio processing method, a model training method, an apparatus, and an electronic device. Background Technology

[0002] Voice Activity Detection (VAD) primarily aims to identify and eliminate prolonged silences in the audio signal stream, thereby conserving voice channel resources without compromising service quality. Current VAD techniques mainly rely on Convolutional Neural Networks (CNNs). However, because the last two layers of CNNs are typically fully connected, they cannot meet the requirements of streaming inference, resulting in poor VAD detection performance. Summary of the Invention

[0003] This application provides an audio processing method, a model training method, an apparatus, and an electronic device that can improve the speech detection effect during the speech activity detection process.

[0004] To solve the above-mentioned technical problems, the specific implementation scheme of this application is as follows:

[0005] In a first aspect, embodiments of this application provide an audio processing method, including:

[0006] Acquire initial audio data, which includes n frames of audio data;

[0007] Based on feature extraction of the n frames of audio data, n sets of feature data corresponding to the n frames of audio data are obtained;

[0008] The n sets of feature data are input into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network (GCNN) layer and a pooling layer.

[0009] The target audio data is determined based on the n rating values ​​and then output. The target audio data is the audio data that includes audio content in the initial audio data.

[0010] Secondly, embodiments of this application also provide a model training method, including:

[0011] Acquire training data, which includes m frames of audio data and m expected values, wherein one audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content;

[0012] Feature extraction is performed on the m frames of audio data to obtain m sets of feature data corresponding to the m frames of audio data;

[0013] The m sets of feature data are input into the initial network model, and the initial network model is trained to obtain a speech endpoint detection network model. The initial network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0014] Thirdly, embodiments of this application also provide an audio processing apparatus, including:

[0015] The first acquisition module is used to acquire initial audio data, which includes n frames of audio data.

[0016] The first feature extraction module is used to extract features from the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data.

[0017] The first prediction module is used to input the n sets of feature data into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network (GCNN) layer and a pooling layer.

[0018] The output module is used to determine the target audio data based on the n rating values ​​and output the target audio data, wherein the target audio data is the audio data including audio content in the initial audio data.

[0019] Fourthly, embodiments of this application also provide a model training apparatus, comprising:

[0020] The second step is to acquire training data, which includes m frames of audio data and m expected values. Each audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content.

[0021] The second feature extraction module is used to extract features from the m frames of audio data to obtain m sets of feature data corresponding to the m frames of audio data.

[0022] The training module is used to input the m sets of feature data into the initial network model, train the initial network model to obtain a speech endpoint detection network model, the initial network model includes a first convolutional neural network and at least two second convolutional neural networks, the at least two second convolutional neural networks are connected in series, and the output of the last of the at least two second convolutional neural networks is connected to the input of the first convolutional neural network, each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0023] By incorporating at least two second convolutional neural networks (CNNs) into the initial network model, and ensuring that each CNN includes a pooling layer, the pooling layers in these CNNs can sequentially reduce the dimensionality of the feature data during model training. This ensures that the output of the last CNN is 1-dimensional, thus adapting to the input requirements of the first CNN. Since the pooling layers in these CNNs can reduce the dimensionality of the input data, the trained speech endpoint detection network model can accept data of any dimension. This allows the speech endpoint detection network model to meet the requirements of streaming inference, improving speech detection performance during speech activity detection.

[0024] Fifthly, embodiments of this application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the methods described in the first and second aspects above.

[0025] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described in the first and second aspects above.

[0026] In this embodiment, by setting at least two second convolutional neural networks (CNNs) in the speech endpoint detection network model, and ensuring that each CNN includes a pooling layer, the pooling layers in the at least two CNNs can sequentially reduce the dimensionality of the feature data. This results in the output data of the last CNN being 1-dimensional, thus adapting to the input requirements of the first CNN. Since the pooling layers in the at least two CNNs can reduce the dimensionality of the input data, the speech endpoint detection network model can accept data of any dimension, thereby meeting the requirements of streaming inference and improving the speech detection performance during speech activity detection. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the audio processing method provided in the embodiments of this application;

[0029] Figure 2 This is a schematic diagram of the target network structure in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of the structure of the second convolutional neural network in the target network structure of this application embodiment;

[0031] Figure 4 This is a schematic diagram of the structure of the GCNN layer in the embodiments of this application;

[0032] Figure 5 This is a schematic diagram of the feature extraction process of dilated convolutional neural networks of different lengths in the embodiments of this application;

[0033] Figure 6 This is a flowchart of the model training method provided in the embodiments of this application;

[0034] Figure 7 This is one of the structural diagrams of the audio processing apparatus provided in the embodiments of this application;

[0035] Figure 8 This is one of the structural diagrams of the model training device provided in the embodiments of this application;

[0036] Figure 9 This is a second structural diagram of the audio processing device provided in the embodiments of this application;

[0037] Figure 10 This is the second structural diagram of the model training device provided in the embodiments of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In related technologies, speech activity detection is mainly based on two network structures: Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). LSTM has temporal modeling capabilities, recording historical temporal information and mitigating the vanishing and exploding gradient problems found in recurrent neural networks. CNN has advantages such as small model parameter count, strong local modeling capabilities, and time-shift invariance. However, both networks have their own problems when applied to speech activity detection. Speech activity detection scenarios generally require processing long temporal sequences, and LSTM performance drops significantly when handling such sequences. CNN, when modeling temporal sequences, needs to maximize its receptive field, thus requiring multiple layers of CNNs. Such deep neural network structures are difficult to train, and excessive CNN stacking also slows down inference speed. Furthermore, since the last two layers of CNNs are typically fully connected layers, these layers require non-streaming input and output data, meaning they require fixed-length data, such as 5-second audio clips. In this scenario, CNNs can only process fixed-length data one at a time, and can only receive the next data after processing the previous one, which fails to meet the requirements of streaming inference. Streaming inference, in this context, means that the model can receive data of varying lengths and can output corresponding processing results while continuously reading longer data (e.g., one hour of audio data).

[0040] Based on this, in this embodiment, a first convolutional neural network is used instead of a fully connected layer to achieve the output of the model recognition result. Since there is no limitation on the length of the input and output data of the speech endpoint detection network model when using a first convolutional neural network instead of a fully connected layer, the requirements of streaming inference can be met. Furthermore, the speech endpoint detection network model also includes at least two second convolutional neural networks. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer. Since the graph convolutional neural network layer can extract features from the input feature data, and the pooling layer can perform dimensionality reduction on the features extracted by the graph convolutional neural network layer, the at least two second convolutional neural networks can simultaneously extract features from the input data and reduce the dimensionality of the extracted feature data, so that the output data of the last second convolutional neural network in the at least two second convolutional neural networks is 1-dimensional data, thereby adapting to the input requirements of the first convolutional neural network.

[0041] Furthermore, the graph convolutional neural network layer includes a dilated convolutional neural network (dilated-CNN), a batch normalization (BN) layer, and a pooling layer. Dilated-CNN is a variant of CNN that introduces holes during convolution, resulting in a larger receptive field for the same computational cost. The BN layer is used to accelerate convergence and perform regularization. The pooling layer's main function is to reduce the frequency dimension of the output without changing its temporal dimension, thus facilitating streaming inference with the subsequent first convolutional neural network. Therefore, in this embodiment, the speech activity detection based on the speech endpoint detection network model improves the effectiveness of speech activity detection.

[0042] Please see Figure 1 An audio processing method provided in this application includes:

[0043] Step 101: Obtain initial audio data, which includes n frames of audio data;

[0044] Step 102: Based on feature extraction of the n frames of audio data, obtain n sets of feature data corresponding to the n frames of audio data;

[0045] Step 103: Input the n sets of feature data into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data;

[0046] The scoring value is used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network (CNN). Each second convolutional neural network includes a graph convolutional neural network (GCNN) layer and a pooling layer. The first convolutional neural network can be a 1×1 convolutional neural network to output the scoring value as a fully connected layer.

[0047] Step 104: Determine the target audio data based on the n rating values ​​and output the target audio data, which is the audio data including audio content in the initial audio data.

[0048] Specifically, the audio processing method provided in the embodiments of this application can be used to detect speech activity in the initial audio data and output the target audio data after eliminating the silent segments in the initial audio data.

[0049] The specific process of extracting audio features from the n frames of audio data based on the preset means can be as follows: extract features from each frame of audio data based on existing speech recognition tools. For example, speech features in each frame of audio data can be extracted based on the Kaldi tool. The speech features can be any of the following features: DFT, MFCC, FBank, etc.

[0050] The aforementioned second convolutional neural network is a feedforward neural network. Its artificial neurons can respond to a portion of the surrounding units within their coverage area, exhibiting good performance in large-scale image processing. In this embodiment, the second convolutional neural network can respond to a portion of the surrounding units within its coverage area. Specifically, the convolutional kernel parameter of the second convolutional neural network in the temporal dimension is 2. Thus, during processing, only information from the current frame and the previous frame is processed, without involving information from future frames, reducing the amount of information processed and improving inference efficiency.

[0051] The pooling layer described above can be max pooling, average pooling, etc. In one embodiment of this application, the pooling layer is a max pooling layer, wherein the pooling layer can reduce the frequency dimension of the output without changing the temporal dimension of the output, so as to cooperate with the subsequent first convolutional neural network for streaming inference.

[0052] Please see Figure 2 In one embodiment of this application, the speech endpoint detection network model further includes a feature input layer and an output layer. The at least two second convolutional neural networks are connected in series, with the input of the first second convolutional neural network connected to the output of the feature input layer, and the output of the last second convolutional neural network connected to the input of the first convolutional neural network. The output of the first convolutional neural network is connected to the input of the output layer. Based on... Figure 2 The specific process of the speech endpoint detection network model shown is as follows:

[0053] The extracted n sets of feature data can be sequentially input into the feature input layer of the speech endpoint detection network model. The feature input layer transmits the received feature data to the first of at least two second convolutional neural networks. The GCNN layer in the first second convolutional neural network can extract features from the received feature data to obtain first feature data. The pooling layer in the first second convolutional neural network can perform dimensionality reduction on the first feature data and input the dimensionality-reduced feature data into the next second convolutional neural network. After receiving the dimensionality-reduced feature data output from the upper second convolutional neural network, the next second convolutional neural network extracts features from the first feature data again and performs dimensionality reduction on the extracted features. This process is repeated layer by layer until the last second convolutional neural network outputs the target feature data, where the target feature data is 1-dimensional data. Then, the first convolutional neural network receives the target feature data, scores it based on the target feature data, and outputs the corresponding score value.

[0054] The values ​​of the aforementioned n ratings can range from 0 to 1. In determining the target audio data based on these n ratings, a threshold can be defined between 0 and 1. When the rating of a frame of audio data is greater than or equal to the threshold, that frame of audio data can be determined to include audio content; when the rating of that frame of audio data is less than the threshold, that frame of audio data can be determined to exclude audio content. The data excluding audio content can be considered a silent period, and the data including audio content can be considered a non-silent period. In this way, it can be determined whether each frame of audio data in the n frames includes audio content.

[0055] After determining whether each frame of audio data in the n frames of audio data includes audio content based on the above method, the n frames of audio data can be smoothed to obtain the target audio data. The specific process of smoothing the n frames of audio data can be as follows:

[0056] The n frames of audio data are sorted according to their playback order, and then grouped according to this sorting to obtain k audio data groups. Each audio data group may include m consecutive frames of audio data, where n is an integer greater than 1, k is an integer greater than 1, and m is less than n. Then, m rating values ​​corresponding to the m consecutive frames of audio data in each audio data group are obtained, and the average of the m rating values ​​is calculated. The calculated average value is then compared with the aforementioned threshold. When the average value is greater than or equal to the threshold, the audio data group is determined to include audio content; conversely, when the average value is less than the threshold, the audio data group is determined to not include audio content. In this way, it is possible to determine whether the above k audio data groups include audio content, and then the data groups that include audio content can be output as the target audio data. Alternatively, the audio data groups that include audio content and have consecutive frame numbers can be merged, and the merged audio data group can be output as the target audio data.

[0057] In this embodiment, by setting at least two second convolutional neural networks (CNNs) in the speech endpoint detection network model, and ensuring that each CNN includes a pooling layer, the pooling layers in the at least two CNNs can sequentially reduce the dimensionality of the feature data. This results in the output data of the last CNN being 1-dimensional, thus adapting to the input requirements of the first CNN. Since the pooling layers in the at least two CNNs can reduce the dimensionality of the input data, the speech endpoint detection network model can accept data of any dimension, making it compatible with streaming inference requirements and improving speech detection performance during speech activity detection.

[0058] Optionally, the second convolutional neural network can be a convolutional block formed by multiple network structures connected in series. For example, the second convolutional neural network includes at least two sub-networks, which are connected in series. The sub-networks include the GCNN layer and the pooling layer, and in each sub-network, the output of the GCNN layer is connected to the input of the pooling layer.

[0059] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result.

[0060] Specifically, see Figure 3This is a schematic diagram of the structure of a second convolutional neural network in one embodiment of this application, wherein the second convolutional neural network includes four sub-networks connected in series. Since each sub-network includes a GCNN layer and a pooling layer, each sub-network can further extract features from the features output by the previous sub-network. After feature extraction, each sub-network can perform dimensionality reduction on the extracted features based on the pooling layer, and then output the dimensionality-reduced data to the next sub-network. Thus, since each second convolutional neural network can perform four feature extractions on the features output by the previous second convolutional neural network, the extraction effect of audio features can be improved. Simultaneously, since each second convolutional neural network can perform four dimensionality reductions on the extracted features, the speech endpoint detection network model can receive higher-dimensional feature data, and the number of features carried by higher-dimensional feature data will increase accordingly, thereby further improving the audio detection effect.

[0061] Optionally, the GCNN layer includes a dilated convolutional neural network (dilated-CNN) and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network;

[0062] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0063] The dilated convolutional neural network mentioned above is a variant of the convolutional neural network, and its kernel differs from the standard convolutional kernel. Please refer to [link to relevant documentation]. Figure 5 The dilated convolutional neural network adds holes to the dilated convolution kernel, thus increasing the receptive field under the same computational load.

[0064] Please see Figure 4 The aforementioned threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer. The threshold activation network preserves the nonlinearity of the speech endpoint detection network model while providing a non-scaling sigmoid activation layer. This sigmoid activation layer, acting as a multiplicative residual connection, facilitates gradient backpropagation, enabling the construction of deeper CNN network structures. Since the threshold activation network allows control over the information in each layer, it is advantageous for predicting the VAD performance of future frames using features from the current frame. This structure aligns with the task modeling requirements of VAD classification models and improves VAD accuracy.

[0065] In this implementation, feature extraction is performed using a dilated convolutional neural network (CNN). Since the CNN has a wider receptive field, the number of CNN layers can be reduced, thus simplifying the training process of the speech endpoint detection network model. Furthermore, setting a threshold activation network helps improve the accuracy of the speech activity detection process.

[0066] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0067] Specifically, since the longer the stride of a dilated convolutional neural network, the more deep features it can extract, by setting the stride of the dilated convolutional neural network in any sub-network to be greater than the stride of the dilated convolutional neural network in the sub-network above it, each sub-network in each second convolutional neural network can gradually extract deeper feature information from the audio data, thereby improving the feature extraction effect.

[0068] For example, see Figure 3 In one embodiment of this application, the second convolutional neural network includes four sub-networks. The stride sizes of the dilated convolutional neural networks in the four sub-networks, from the first sub-network to the last sub-network, are 1, 2, 4, and 8, respectively. See also... Figure 5 This is a schematic diagram of the feature extraction process of the dilated convolutional neural network in each sub-network.

[0069] In this implementation, by setting the stride of the dilated convolutional neural network in any sub-network to be greater than the stride of the dilated convolutional neural network in the sub-network above it, the effect of feature extraction can be improved.

[0070] Optionally, the sub-network further includes a batch normalization (BN) layer, and the output of the GCNN layer is connected to the input of the pooling layer through the batch normalization layer.

[0071] In this implementation, by setting a batch normalization layer in the sub-network, the convergence speed during model training can be accelerated, making the model training process more stable, avoiding gradient explosion or gradient vanishing problems, and also playing a certain role in regularization.

[0072] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0073] In this embodiment, a binary classification score is performed based on the features of each extracted audio frame by the second sigmoid activation layer in the first convolutional neural network, thereby outputting the score value corresponding to each audio frame.

[0074] In one embodiment of this application, the initial audio data mentioned above may be dialogue content obtained from a telephone channel conversation scenario, and the audio features may be 40-dimensional FBank features. (See [link to relevant documentation]). Figure 2 The speech endpoint detection network model may include six second convolutional neural networks connected in series. The following explanation uses the processing of one frame of audio data from the initial audio data as an example to further illustrate the audio processing method in this embodiment:

[0075] A 40-dimensional FBank feature is extracted from the target frame audio data. This 40-dimensional FBank feature is then input into a speech endpoint detection network model. Six sequentially cascaded second convolutional neural networks (CNNs) in the CNN model extract features from the 40-dimensional FBank feature. The last layer of the CNN outputs a 1-dimensional target feature. Then, the second sigmoid activation layer in the first CNN scores the target frame audio data based on this 1-dimensional target feature, outputting a score value for the target frame audio data. This process is repeated for the n frames of audio data to obtain the n score values. Finally, the n score values ​​are smoothed to obtain the target audio data.

[0076] The above audio processing method can also be applied to data stream processing, and its specific implementation process is as follows:

[0077] The temporal speech is acquired, and FBank speech features are extracted from the temporal speech with a window length of 25ms, a window shift of 10ms, and 512 FFT points to obtain a feature sequence. The feature sequence is then input into the speech endpoint detection network model for prediction to obtain the score value corresponding to each frame of audio data in the temporal speech. The obtained score values ​​are then smoothed to obtain the target audio data.

[0078] Please see Figure 6 A model training method provided in this application includes:

[0079] Step 601: Obtain training data, which includes m frames of audio data and m expected values, wherein one audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content;

[0080] Step 602: Extract features from the m frames of audio data to obtain m sets of feature data that correspond one-to-one with the m frames of audio data;

[0081] Step 603: Input the m sets of feature data into the initial network model, train the initial network model to obtain a speech endpoint detection network model. The initial network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0082] In this embodiment, a speech endpoint detection network model is trained. Since the speech endpoint detection network model includes at least two second convolutional neural networks (CNNs), and each CNN includes a pooling layer, the pooling layers in these at least two CNNs can sequentially reduce the dimensionality of the feature data. This ensures that the output of the last CNN is 1-dimensional, thus adapting to the input requirements of the first CNN. Because the pooling layers in the at least two CNNs can reduce the dimensionality of the input data, the speech endpoint detection network model can accept data of any dimension, thus meeting the requirements of streaming inference and improving the speech detection performance during speech activity detection.

[0083] Optionally, the step of inputting the m sets of feature data into the initial network model and training the initial network model to obtain the speech endpoint detection network model includes:

[0084] The m sets of feature data are input into the initial network model to obtain m score values ​​output by the initial network model that correspond one-to-one with the m sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content.

[0085] A loss function is generated based on the m expected values ​​and the m score values;

[0086] The initial network model is trained based on the loss function to obtain the speech endpoint detection network model.

[0087] In this implementation, a loss function is constructed based on m expected values ​​and m score values. Then, the initial network model is trained based on the loss function. During the training process, the parameters of the initial network model can be continuously optimized to obtain the trained speech endpoint detection network model.

[0088] Optionally, before acquiring training data, which includes m frames of audio data and m expected values, the method further includes: constructing an initial network model.

[0089] Optionally, the second convolutional neural network includes at least two sub-networks, which are connected in series. Each sub-network includes the GCNN layer and the pooling layer, and in each sub-network, the output of the GCNN layer is connected to the input of the pooling layer.

[0090] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result. Optionally, the GCNN layer includes a dilated convolutional neural network and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network.

[0091] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0092] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0093] Optionally, the threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

[0094] Optionally, the sub-network further includes a batch normalization layer, and the output of the GCNN layer is connected to the input of the pooling layer through the batch normalization layer.

[0095] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0096] The model training method provided in this application is a method for training the speech endpoint detection network model described in the above embodiments. The trained speech endpoint detection network model has all the beneficial effects of the speech endpoint detection network model in the above embodiments, and will not be described again here to avoid repetition.

[0097] Please see Figure 7 An audio processing apparatus 700 provided in this application embodiment includes:

[0098] The first acquisition module 701 is used to acquire initial audio data, which includes n frames of audio data.

[0099] The first feature extraction module 702 is used to extract features from the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data.

[0100] The first prediction module 703 is used to input the n sets of feature data into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network (GCNN) layer and a pooling layer.

[0101] The output module 704 is used to determine the target audio data based on the n rating values ​​and output the target audio data, wherein the target audio data is the audio data including audio content in the initial audio data.

[0102] Optionally, the second convolutional neural network includes at least two sub-networks, which are connected in series. Each sub-network includes the graph convolutional neural network layer and the pooling layer. In each sub-network, the output of the graph convolutional neural network layer is connected to the input of the pooling layer.

[0103] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result.

[0104] Optionally, the graph convolutional neural network layer includes a dilated convolutional neural network and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network;

[0105] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0106] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0107] Optionally, the threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

[0108] Optionally, the sub-network further includes a batch normalization layer, and the output of the graph convolutional neural network layer is connected to the input of the pooling layer through the batch normalization layer.

[0109] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0110] The audio processing apparatus 700 provided in this application embodiment can implement the various processes in the above method embodiments, and will not be described again here to avoid repetition.

[0111] Please see Figure 8 A model training device 800 provided in this application embodiment includes:

[0112] The second method is to acquire training data 801, which is used to acquire training data. The training data includes m frames of audio data and m expected values, wherein one audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content.

[0113] The second feature extraction module 802 is used to extract features from the m frames of audio data to obtain m sets of feature data corresponding to the m frames of audio data.

[0114] The training module 803 is used to input the m sets of feature data into the initial network model, train the initial network model to obtain a speech endpoint detection network model, the initial network model includes a first convolutional neural network and at least two second convolutional neural networks, the at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network, each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0115] Optionally, the training module 803 includes:

[0116] The input submodule is used to input the m sets of feature data into the initial network model to obtain m score values ​​output by the initial network model that correspond one-to-one with the m sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content.

[0117] A generation submodule is used to generate a loss function based on the m expected values ​​and the m score values;

[0118] The training submodule is used to train the initial network model based on the loss function to obtain the speech endpoint detection network model.

[0119] Optionally, the second convolutional neural network includes at least two sub-networks, which are connected in series. Each sub-network includes the GCNN layer and the pooling layer, and in each sub-network, the output of the GCNN layer is connected to the input of the pooling layer.

[0120] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result. Optionally, the GCNN layer includes a dilated convolutional neural network and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network.

[0121] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0122] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0123] Optionally, the threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

[0124] Optionally, the sub-network further includes a batch normalization layer, and the output of the GCNN layer is connected to the input of the pooling layer through the batch normalization layer.

[0125] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0126] The model training device 800 provided in this application embodiment can implement the various processes in the above method embodiments, and will not be described again here to avoid repetition.

[0127] See Figure 9 , Figure 9 This is a structural diagram of an audio processing apparatus 900 provided in another embodiment of this application, as shown below. Figure 9As shown, the audio processing device 900 includes a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor. The various components of the data transmission device 900 are coupled together via a bus interface 903. When the computer program is executed by the processor 901, it performs the following steps:

[0128] Acquire initial audio data, which includes n frames of audio data;

[0129] Feature extraction is performed on the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data;

[0130] The n sets of feature data are input into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0131] The target audio data is determined based on the n rating values ​​and then output. The target audio data is the audio data that includes audio content in the initial audio data.

[0132] Optionally, the second convolutional neural network includes at least two sub-networks, which are connected in series. Each sub-network includes the graph convolutional neural network layer and the pooling layer. In each sub-network, the output of the graph convolutional neural network layer is connected to the input of the pooling layer.

[0133] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result.

[0134] Optionally, the graph convolutional neural network layer includes a dilated convolutional neural network and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network;

[0135] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0136] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0137] Optionally, the threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

[0138] Optionally, the sub-network further includes a batch normalization layer, and the output of the graph convolutional neural network layer is connected to the input of the pooling layer through the batch normalization layer.

[0139] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0140] See Figure 10 , Figure 10 This is a structural diagram of a model training device 1000 provided in another embodiment of this application, as shown below. Figure 10 As shown, the model training device 1000 includes a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. The various components in the data transmission device 1000 are coupled together via a bus interface 1003. When the computer program is executed by the processor 1001, it performs the following steps:

[0141] Acquire training data, which includes m frames of audio data and m expected values, wherein one audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content;

[0142] Feature extraction is performed on the m frames of audio data to obtain m sets of feature data corresponding to the m frames of audio data;

[0143] The m sets of feature data are input into the initial network model, and the initial network model is trained to obtain a speech endpoint detection network model. The initial network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer.

[0144] Optionally, the step of inputting the m sets of feature data into the initial network model and training the initial network model to obtain the speech endpoint detection network model includes:

[0145] The m sets of feature data are input into the initial network model to obtain m score values ​​output by the initial network model that correspond one-to-one with the m sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content.

[0146] A loss function is generated based on the m expected values ​​and the m score values;

[0147] The initial network model is trained based on the loss function to obtain the speech endpoint detection network model.

[0148] Optionally, before acquiring training data, which includes m frames of audio data and m expected values, the method further includes: constructing an initial network model.

[0149] Optionally, the second convolutional neural network includes at least two sub-networks, which are connected in series. Each sub-network includes the GCNN layer and the pooling layer, and in each sub-network, the output of the GCNN layer is connected to the input of the pooling layer.

[0150] The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result. Optionally, the GCNN layer includes a dilated convolutional neural network and a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network.

[0151] The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

[0152] Optionally, in the at least two sub-networks, the stride of the dilated convolutional neural network in any sub-network is greater than the stride of the dilated convolutional neural network in the previous sub-network.

[0153] Optionally, the threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

[0154] Optionally, the sub-network further includes a batch normalization layer, and the output of the GCNN layer is connected to the input of the pooling layer through the batch normalization layer.

[0155] Optionally, the output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

[0156] This application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here.

[0157] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0158] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause an electronic device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0160] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, include: Acquire initial audio data, which includes n frames of audio data; Feature extraction is performed on the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data; The n sets of feature data are input into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer. The pooling layer is used to perform dimensionality reduction processing on the features extracted by the graph convolutional neural network layer so that the output data of the last second convolutional neural network is 1-dimensional data. The target audio data is determined based on the n rating values ​​and the target audio data is output. The target audio data is the audio data that includes audio content in the initial audio data. The second convolutional neural network includes at least two sub-networks, each sub-network including the graph convolutional neural network layer, the graph convolutional neural network layer including the dilated convolutional neural network, wherein the stride of the dilated convolutional neural network in any of the at least two sub-networks is greater than the stride of the dilated convolutional neural network in the previous sub-network.

2. The method according to claim 1, characterized in that, The at least two sub-networks are connected in series, and each sub-network further includes the pooling layer. In each sub-network, the output of the graph convolutional neural network layer is connected to the input of the pooling layer. The graph convolutional neural network layer is used to extract features from the input feature data; the pooling layer is used to reduce the dimensionality of the features extracted by the graph convolutional neural network layer to obtain the feature extraction result.

3. The method according to claim 2, characterized in that, The graph convolutional neural network layer further includes a threshold activation network, and the output of the dilated convolutional neural network is connected to the input of the threshold activation network. The dilated convolutional neural network is used to extract features from the input feature data; the threshold activation network is used to transmit the features extracted by the dilated convolutional neural network to the pooling layer.

4. The method according to claim 3, characterized in that, The threshold activation network includes a parallel tanh activation layer and a first sigmoid activation layer.

5. The method according to claim 3, characterized in that, The sub-network further includes a batch normalization layer, and the output of the graph convolutional neural network layer is connected to the input of the pooling layer through the batch normalization layer.

6. The method according to claim 1, characterized in that, The output of the first convolutional neural network includes a second sigmoid activation layer, which is used to perform binary classification scoring on the initial audio data and output the score value.

7. A model training method, characterized in that, include: Acquire training data, which includes m frames of audio data and m expected values, wherein one audio data corresponds to one expected value, and the expected value is used to characterize whether the corresponding frame of audio data actually includes audio content; Feature extraction is performed on the m frames of audio data to obtain m sets of feature data corresponding to the m frames of audio data; The m sets of feature data are input into an initial network model, and the initial network model is trained to obtain a speech endpoint detection network model. The initial network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network layer and a pooling layer. The pooling layer is used to perform dimensionality reduction processing on the features extracted by the graph convolutional neural network layer so that the output data of the last second convolutional neural network is 1-dimensional data. The second convolutional neural network includes at least two sub-networks, each sub-network including the graph convolutional neural network layer, the graph convolutional neural network layer including the dilated convolutional neural network, wherein the stride of the dilated convolutional neural network in any of the at least two sub-networks is greater than the stride of the dilated convolutional neural network in the previous sub-network.

8. An audio processing apparatus, characterized in that, include: The first acquisition module is used to acquire initial audio data, which includes n frames of audio data. The first feature extraction module is used to extract features from the n frames of audio data to obtain n sets of feature data corresponding to the n frames of audio data. The first prediction module is used to input the n sets of feature data into the speech endpoint detection network model to obtain n score values ​​output by the speech endpoint detection network model corresponding to the n sets of feature data. The score values ​​are used to characterize whether the audio data of the corresponding frame contains audio content. The speech endpoint detection network model includes a first convolutional neural network and at least two second convolutional neural networks. The at least two second convolutional neural networks are connected in series, and the output of the last second convolutional neural network is connected to the input of the first convolutional neural network. Each second convolutional neural network includes a graph convolutional neural network (GCNN) layer and a pooling layer. The pooling layer is used to perform dimensionality reduction processing on the features extracted by the graph convolutional neural network layer so that the output data of the last second convolutional neural network is 1-dimensional data. An output module is used to determine target audio data based on the n rating values ​​and output the target audio data, wherein the target audio data is the audio data including audio content in the initial audio data; The second convolutional neural network includes at least two sub-networks, each sub-network including the graph convolutional neural network layer, the graph convolutional neural network layer including the dilated convolutional neural network, wherein the stride of the dilated convolutional neural network in any of the at least two sub-networks is greater than the stride of the dilated convolutional neural network in the previous sub-network.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 7.

Images