Audio-oriented key semantic information detection method

Abstract

The invention discloses an audio-oriented key semantic information detection method, and belongs to the technical field of speech recognition. A key semantic information detection model based on the method comprises an identification network and a plurality of classifiers in one-to-one correspondence with predefined keywords. The recognition network is used for outputting the audio features corresponding to each frame in the audio to be detected, and the audio features are input into all the classifiers at the same time. And the classifier is used for judging the probability that the input audio feature contains the keyword corresponding to the classifier. And for any frame, combining the probabilities output by all classifiers into a probability set corresponding to the frame. When the key semantic information detection model is trained, a refined pooling loss strategy is used as a training target. According to the method, by means of the refined pooling loss and keyword time sequence detection module and the like, the calculation efficiency is improved, and the influence of the keyword position on the detection effect is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of speech recognition, and in particular relates to a key semantic information detection method for audio. Background Art

[0002] Key semantic information detection plays a vital role in the field of audio information security, and has wide application value in communication security, privacy protection, content monitoring, and many other aspects. Traditional detection methods mainly rely on matching specific keywords, using simple pattern matching techniques such as regular expressions or dictionary queries. However, these methods have obvious deficiencies in accuracy and adaptability in complex environments.

[0003] In recent years, with the rapid development of deep learning technology, deep learning-based audio key semantic information detection methods have achieved remarkable results. These methods automatically learn and extract audio features through neural network models to effectively identify key semantic information in audio content.

[0004] Automatic speech recognition (ASR) is a method of converting speech into text for recognition. When using ASR for key semantic information detection, it is usually necessary to transcribe the entire audio sequence and then search for keywords in the transcribed text. This means that no matter where the keyword appears in the audio, the system must process the complete audio data. Since the entire audio stream needs to be processed, the amount of calculation is large and the processing time is long. Especially when the audio length is long or real-time processing is required, the efficiency problem is more prominent. In addition, the uncertainty of the keyword position makes the ASR system more susceptible to interference from background noise and non-keyword speech, increasing the probability of misrecognition.

[0005] The fixed window method analyzes or processes data by sliding a fixed-length window over the signal. When using a fixed time window for feature extraction and detection, if the keyword length or occurrence position does not match the preset window, the keyword may be truncated or omitted. Therefore, the fixed window method is highly sensitive to the position of the keyword and cannot adapt to the situation where the keyword appears at any position in the audio.

[0006] It can be seen that the existing key semantic information detection methods based on deep learning have two main problems. First, the computational complexity is large and the efficiency is low. It cannot run on resource-constrained devices and maintain good detection performance. Second, the detection effect is greatly affected by the position of the keywords. Summary of the invention

[0007] The present invention proposes a method for detecting key semantic information in audio, the purposes of which are: 1. to solve the problem of large computational complexity and low efficiency of existing methods; 2. to solve the problem that the detection effect is greatly affected by the position of keywords.

[0008] The technical solution of the present invention is as follows: A method for detecting key semantic information in audio, wherein the audio to be detected is input into a key semantic information detection model to obtain the probability that the audio to be detected contains key semantic information, wherein the key semantic information detection model includes a recognition network and a plurality of classifiers corresponding to predefined keywords one by one;

[0009] The recognition network is used to output the audio features corresponding to each frame in the audio to be detected, and each audio feature is simultaneously input into all classifiers; the classifier is used to determine the probability that the input audio feature contains the keyword corresponding to the classifier; for any frame, the probabilities output by all classifiers are combined into a probability set corresponding to the frame; the maximum value in the probability set corresponding to all frames of the audio to be detected is taken as the detection result of the audio to be detected; When training the key semantic information detection model, a refined pooling loss strategy is used as the training target, that is, for the first training samples, and its loss function is: ; in: , It is The detection results of the training samples obtained by the key semantic information detection model are: It is The number of frames of training samples, It is The first training sample The probability set corresponding to the frame, It is the preset minimum number of continuous frames; It is The true labels of the training samples.

[0010] As a further improvement of the audio-oriented key semantic information detection method: the recognition network includes a plurality of keyword timing detection modules connected in series, and the keyword timing detection module includes a feature extraction layer, an output normalization layer, a ReLU activation layer and a Dropout layer connected in series, and the output of the Dropout layer is also added to the input of the feature extraction layer through residual enhancement.

[0011] As a further improvement of the audio-oriented key semantic information detection method: the feature extraction layer obtains the dilated time domain features corresponding to each frame of the input audio feature by dilating the audio key feature extraction operator; The expansion factors of the expanded audio key feature extraction operators in each keyword timing detection module increase successively.

[0012] As a further improvement of the audio-oriented key semantic information detection method, the calculation process of the expanded audio key feature extraction operator is: ; in, Refers to the first audio feature input to the feature extraction layer frames, is the feature extraction operator, is the size of the feature extraction operator, is the expansion factor; is the first audio feature input to the feature extraction layer The dilated time domain features of the frames.

[0013] As a further improvement of the audio-oriented key semantic information detection method: A keyword sequence detection module, whose expansion factor .

[0014] As a further improvement of the audio-oriented key semantic information detection method, the output normalization layer normalizes the dilated time domain features of each frame output by the feature extraction layer in the following manner: ; in, It is The dilated time domain features of the frames, It is the standardized The dilated time domain features of the frames, and are the mean and variance of the dilated time domain features of all frames output by the feature extraction layer, is a constant.

[0015] As a further improvement of the audio-oriented key semantic information detection method: the key semantic information detection model also includes a standardization module and a dimension optimization module for preprocessing; the acoustic features of the audio to be detected are first input into the standardization module to obtain standardized acoustic features, and then the standardized acoustic features are input into the dimension optimization module to obtain preprocessed audio features that match the dimensions of the recognition network, and then the preprocessed audio features are input into the recognition network.

[0016] As a further improvement of the audio-oriented key semantic information detection method, the processing process of the standardization module is: Step A-1: ​​Calculate the mean of the acoustic features of all the audio to be detected in this input and variance : ; ; in, is the number of input samples, For the The feature vector of samples; Step A-2: Calculate the acoustic features input to the standardization module The corresponding standardized acoustic characteristics : .

[0017] As a further improvement of the audio-oriented key semantic information detection method, the processing process of the dimension optimization module is as follows: Step B-1: Normalize the acoustic features input to the dimensionality optimization module Perform dimension transformation: ; Among them, The dimension is , The dimension is , then the weight matrix The dimension is , the bias vector The dimension is ; Step B-2: Obtain preprocessed audio features through nonlinear activation : .

[0018] As a further improvement of the audio-oriented key semantic information detection method, the classifier includes a fully connected layer and a Sigmoid activation function, and the processing process is: Step D-1: Input audio features to the classifier The dimension is transformed through the fully connected layer to obtain a scalar value : ; in, is the weight vector, is the offset; Step D-2: Use the Sigmoid activation function to convert the scalar value Mapping to probability : .

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The refined pooling loss adopted in the present invention adopts the method of selecting the maximum activation value (i.e., the maximum posterior probability) in the feature map in the time dimension. Its essence is to reduce the dimension of the feature. The reduction of the feature size brings about a reduction in the number of model parameters, thereby improving the efficiency of the model.

[0020] 2. Selecting the maximum activation value in the feature map in the refined pooling loss means that no matter where the keyword appears, as long as there is a strong response, refined pooling can effectively extract it, which makes the model robust to changes in the keyword position and is not affected by the specific position.

[0021] 3. The keyword timing detection module increases the receptive field layer by layer by using the expanded audio key feature extraction operator, which can capture long-range data dependencies and improve the effect of keyword detection.

[0022] 4. The expanded audio key feature extraction operator is used in the keyword timing detection module, allowing the feature extraction operator to cover a wider input area, so that the network can cover longer input sequences without significantly increasing the number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 This is the structural diagram of the keyword timing detection module. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0025] A key semantic information detection method for audio, which inputs the audio to be detected into a key semantic information detection model to obtain the probability that the audio to be detected contains key semantic information.

[0026] like Figure 1 The key semantic information detection model includes a standardization module, a dimension optimization module and a recognition network connected in sequence, and also includes multiple classifiers corresponding to predefined keywords one by one.

[0027] The standardization module and the dimension optimization module are used for preprocessing. The acoustic features of the audio to be detected are first input into the standardization module to obtain standardized acoustic features, and then the standardized acoustic features are input into the dimension optimization module to obtain preprocessed audio features that match the dimensions of the recognition network, and then the preprocessed audio features are input into the recognition network. The recognition network is used to output the audio features corresponding to each frame in the audio to be detected, and each audio feature is input into all classifiers at the same time. The classifier is used to determine the probability that the input audio feature contains the keyword corresponding to the classifier. For any frame, the probabilities output by all classifiers are combined into a probability set corresponding to the frame. The maximum value in the probability set corresponding to all frames of the audio to be detected is taken as the detection result of the audio to be detected.

[0028] Specifically: A. Standardized modules.

[0029] This module is used to normalize the input acoustic features so that the eigenvalue distribution is close to the normal distribution, thereby facilitating the improvement of the uniformity and efficiency of subsequent model processing.

[0030] The processing process of the standardization module is: Step A-1: ​​Calculate the mean of the acoustic features of all the audio to be detected in this input and variance , the mean and variance of the samples in this batch are calculated during training and testing, and the actual prediction is calculated based on the audio that needs to be detected. The calculation method is: ; ; in, is the number of input samples, For the The feature vector of samples; Step A-2: Calculate the acoustic features input to the standardization module The corresponding standardized acoustic characteristics : .

[0031] B. Dimension optimization module.

[0032] This module maps the dimensions of the normalized acoustic features to the dimensions required by the model, adapting to the input requirements of the subsequent network and ensuring that the data flow is effectively transmitted in the network.

[0033] The processing process of the dimension optimization module is: Step B-1: Normalize the acoustic features input to the dimensionality optimization module Perform dimension transformation: ; Among them, The dimension is , The dimension is , then the weight matrix The dimension is , the bias vector The dimension is ; Step B-2: Obtain preprocessed audio features through nonlinear activation : .

[0034] Introducing the nonlinear ReLU function can help the model learn complex data patterns.

[0035] C. Identify the network.

[0036] like Figure 2 The recognition network includes four keyword sequence detection modules connected in series. The keyword sequence detection module includes a feature extraction layer, an output normalization layer, a ReLU activation layer and a Dropout layer connected in series.

[0037] The keyword time series detection module increases the receptive field layer by layer by using the dilation audio key feature extraction operator, so as to capture long-range data dependencies, which is crucial for keyword detection. The dilation rate of each layer increases exponentially with the number of layers, which can improve the network's ability to capture long-term dependencies without excessively increasing the computational burden. This architecture is particularly suitable for processing audio processing tasks with strong time series dependencies. Through sophisticated hierarchical design and efficient data processing strategies, the accuracy and efficiency of keyword detection are greatly improved.

[0038] The specific structure is as follows: C-1. Feature extraction layer.

[0039] The feature extraction layer obtains the dilated time domain features corresponding to each frame of the input audio feature by dilating the audio key feature extraction operator. In addition, the dilation factors of the dilated audio key feature extraction operators in each keyword timing detection module are increased in sequence.

[0040] Specifically, the calculation process of the expansion audio key feature extraction operator is: ; in, Refers to the first audio feature input to the feature extraction layer frames, is the feature extraction operator, is the size of the feature extraction operator, is the expansion factor, for the A keyword sequence detection module, whose expansion factor . is the first audio feature input to the feature extraction layer The dilated time domain features of the frames.

[0041] In this module, the expansion audio key feature extraction operator is specifically used to allow the feature extraction operator to cover a wider input area, so that the network can cover longer input sequences without significantly increasing the number of parameters.

[0042] C-2. Output normalization layer.

[0043] The output normalization layer is used to adjust the output after key feature extraction and standardize the mean and variance of these outputs, thereby improving the convergence speed and stability of the model.

[0044] The output normalization layer normalizes the dilated time domain features of each frame output by the feature extraction layer in the following manner: ; in, It is The dilated time domain features of the frames, It is the standardized The dilated time domain features of the frames, and are the mean and variance of the dilated time domain features of all frames output by the feature extraction layer, respectively; is a small constant that prevents the denominator from being zero.

[0045] C-3, ReLU activation layer.

[0046] The ReLU activation function is used to increase the nonlinearity of the network. It sets all negative values ​​to zero and retains positive values ​​at each time point.

[0047] C-4, Dropout layer.

[0048] The Dropout layer randomly drops (sets to zero) some activation values. It is a regularization method that can prevent the model from overfitting.

[0049] The output of the Dropout layer is also added to the input of the feature extraction layer through residual enhancement to form a short-circuit path. This design allows the gradient to flow directly through these short-circuit connections, alleviating the problem of gradient disappearance or explosion in deep networks, and maintaining the integrity of information in deep network transmission.

[0050] D. Classifier.

[0051] Each classifier is responsible for predicting the posterior probability of a keyword, and each classifier contains a single output node. This structure allows the system to independently process the detection of each keyword in a multi-keyword environment, improving the flexibility and application scope of the model.

[0052] The classifier includes a fully connected layer and a Sigmoid activation function, and the processing process is: Step D-1: Input audio features to the classifier The dimension is transformed through the fully connected layer to obtain a scalar value : ; in, is the weight vector, is the offset; Step D-2: Use the Sigmoid activation function to convert the scalar value Mapping to probability : .

[0053] This step limits the output between 0 and 1, representing the posterior probability of the keyword appearing.

[0054] When training the key semantic information detection model, the training samples include positive samples and negative samples. The positive samples are audio clips containing only one keyword, and the negative samples are audio clips containing a non-keyword that needs to be ignored. The training samples are input into the model to obtain the corresponding probability set results, and then the parameters of the model are updated according to the loss function.

[0055] This method uses a refined pooling loss strategy as the training target, that is, for the training samples, and its loss function is: ; in: , It is The detection results of the training samples obtained by the key semantic information detection model are: It is The number of frames of training samples, It is The first training sample The probability set corresponding to the frame, It is the preset minimum number of continuous frames; It is The true labels of the training samples.

[0056] By using the refined pooling loss, the model can automatically learn the end timestamps of keywords, so it no longer depends on alignment or the end timestamps of keywords. Specifically, for positive samples, the refined pooling loss only optimizes the frames with the highest posterior probability and ignores other frames. For negative samples, the refined pooling loss will minimize the frames with the highest posterior probability, so the posterior probabilities of all frames in the negative samples will be minimized.

[0057] This end-to-end key semantic information detection method can provide efficient and accurate key semantic information recognition services in different application scenarios, especially when dealing with complex audio environments and multi-keyword recognition tasks, showing good performance and adaptability. The implementation of this method simplifies the traditional process of key semantic information detection and significantly improves the operational efficiency and practicality of the system by directly learning the features of keywords from audio signals.

[0058] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A method for detecting key semantic information in audio, inputting the audio to be detected into a key semantic information detection model, and obtaining the probability that the audio to be detected contains key semantic information, characterized in that: The key semantic information detection model includes a recognition network and a plurality of classifiers corresponding to predefined keywords one by one; The recognition network is used to output the audio features corresponding to each frame in the audio to be detected, and each audio feature is simultaneously input into all classifiers; the classifier is used to determine the probability that the input audio feature contains the keyword corresponding to the classifier; for any frame, the probabilities output by all classifiers are combined into a probability set corresponding to the frame; the maximum value in the probability set corresponding to all frames of the audio to be detected is taken as the detection result of the audio to be detected; When training the key semantic information detection model, a refined pooling loss strategy is used as the training target, that is, for the first training samples, and its loss function is: ； in: , It is The detection results of the training samples obtained by the key semantic information detection model are: It is The number of frames of training samples, It is The first training sample The probability set corresponding to the frame, It is the preset minimum number of continuous frames; It is The true labels of the training samples.

2. The method for detecting key semantic information for audio according to claim 1, characterized in that: The recognition network includes a plurality of keyword timing detection modules connected in series in sequence, and the keyword timing detection module includes a feature extraction layer, an output normalization layer, a ReLU activation layer and a Dropout layer connected in sequence, and the output of the Dropout layer is also added to the input of the feature extraction layer through residual enhancement.

3. The method for detecting key semantic information for audio according to claim 2, wherein: The feature extraction layer obtains the dilated time domain features corresponding to each frame of the input audio feature by dilating the audio key feature extraction operator; The expansion factors of the expanded audio key feature extraction operators in each keyword timing detection module increase successively.

4. The method for detecting key semantic information for audio according to claim 3, characterized in that: The calculation process of the expansion audio key feature extraction operator is: ； in, Refers to the first audio feature input to the feature extraction layer frames, is the feature extraction operator, is the size of the feature extraction operator, is the expansion factor; is the first audio feature input to the feature extraction layer The dilated time domain features of the frames.

5. The method for detecting key semantic information for audio according to claim 4, characterized in that: For A keyword sequence detection module, whose expansion factor .

6. The method for detecting key semantic information for audio according to claim 3, characterized in that: The output normalization layer normalizes the dilated time domain features of each frame output by the feature extraction layer in the following manner: ； in, It is The dilated time domain features of the frames, It is the standardized The dilated time domain features of the frames, and are the mean and variance of the dilated time domain features of all frames output by the feature extraction layer, is a constant.

7. The method for detecting key semantic information for audio according to any one of claims 1 to 6, characterized in that: The key semantic information detection model also includes a standardization module and a dimension optimization module for preprocessing; the acoustic features of the audio to be detected are first input into the standardization module to obtain standardized acoustic features, and then the standardized acoustic features are input into the dimension optimization module to obtain preprocessed audio features that match the dimensions of the recognition network, and then the preprocessed audio features are input into the recognition network.

8. The method for detecting key semantic information for audio according to claim 7, characterized in that: The processing process of the standardization module is: Step A-1: ​​Calculate the mean of the acoustic features of all the audio to be detected in this input and variance : ； ； in, is the number of input samples, For the The feature vector of samples; Step A-2: Calculate the acoustic features input to the standardization module The corresponding standardized acoustic characteristics : 。 9. The method for detecting key semantic information for audio according to claim 7, characterized in that: The processing process of the dimension optimization module is: Step B-1: Normalize the acoustic features input to the dimensionality optimization module Perform dimension transformation: ； Among them, The dimension is , The dimension is , then the weight matrix The dimension is , the bias vector The dimension is ; Step B-2: Obtain preprocessed audio features through nonlinear activation : 。 10. The method for detecting key semantic information for audio according to claim 7, characterized in that: The classifier includes a fully connected layer and a Sigmoid activation function, and the processing process is: Step D-1: Input audio features to the classifier The dimension is transformed through the fully connected layer to obtain a scalar value : ； in, is the weight vector, is the offset; Step D-2: Use the Sigmoid activation function to convert the scalar value Mapping to Probability : 。

Images