Audio signal content analysis method, device, equipment and storage medium

By acquiring audio signal characteristics and combining embedded features and audio features, determining feature probability distribution and selecting potential features, the problem of generating single description information in audio signal content analysis is solved, diversified description information generation is realized, and user understanding is enhanced Diversity.

CN117594060BActive Publication Date: 2024-12-03BEIJING UNIV OF POSTS & TELECOMM
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202311435217.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-31
Publication Date
2024-12-03
Estimated Expiration
2043-10-31

AI Technical Summary

Technical Problem

The existing audio signal content analysis methods generate text statements that are relatively single and cannot fully reflect the diverse understanding of audio signals, making it difficult for users to understand different description information of the same audio signal.

Method used

By obtaining the characteristics of the audio signal, combining embedded features and audio features, the feature probability distribution is determined, and potential features are selected through resampling operations to generate diverse description information, ensuring that different descriptive words are generated at the same time step during each analysis process. This provides a variety of ways of understanding while ensuring accuracy.

Benefits of technology

It realizes that different target description information are generated based on the same audio signal while ensuring the accuracy of generating target description information, and enhances the diversified understanding and expression of audio signal content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117594060B_ABST
    Figure CN117594060B_ABST
Patent Text Reader

Abstract

The present application proposes an audio signal content analysis method, device and storage medium, the method comprising: obtaining target audio features of a target audio signal to be processed; determining the feature probability distribution of the current time step based on the target embedding features and the target audio features, the target embedding features being the original word embedding features of the description words of the previous time step or the preset word embedding features of the target audio signal; selecting the target potential features of the current time step from multiple potential features of the feature probability distribution based on the feature probability distribution; generating target description information corresponding to the target audio signal based on the target word potential features and the target audio features of multiple time steps. The embodiments of the present application realize that different description information can be generated based on the audio signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an audio signal content analysis method, device, equipment and storage medium. Background Art

[0002] At present, audio signals can provide us with rich information in daily life. Audio signal content analysis is a task that enables machines to accurately perceive the content of audio signals like humans. It is also a key step towards general artificial intelligence, enabling machines to have human-like auditory perception capabilities. In daily life, it has great application prospects, for example, it can help people with disabilities integrate into society more naturally, and serve as a life assistant to remind users.

[0003] Currently, it is possible to encapsulate multimodal signals through an encoder based on a deep learning model and use a decoder to generate sentences about the signal.

[0004] Due to the ambiguity of audio signals and the complexity of language representation, people's descriptions of audio content are more diverse. Although the current audio signal content analysis methods can generate relatively accurate sentences, the generated sentences are relatively simple. Summary of the invention

[0005] The present application proposes an audio signal content analysis method, device, equipment and storage medium, which can solve the current technical problem that the text sentences generated based on audio signals are relatively single.

[0006] The first embodiment of the present application provides an audio signal content analysis method, including:

[0007] Acquire target audio features of a target audio signal to be processed;

[0008] Determine a feature probability distribution of a current time step based on a target embedding feature and the target audio feature, wherein the target embedding feature is an original word embedding feature of a description word at a previous time step or a preset word embedding feature of the target audio signal;

[0009] Selecting the target potential feature of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution;

[0010] Based on the target potential features of multiple time steps and the target audio features, target description information corresponding to the target audio signal is generated.

[0011] The embodiment of the second aspect of the present application provides an audio signal content analysis device, including:

[0012] An acquisition module, used for acquiring target audio features of a target audio signal to be processed;

[0013] A determination module, configured to determine a feature probability distribution of a current time step based on a target embedding feature and the target audio feature, wherein the target embedding feature is an original word embedding feature of a description word at a previous time step or a preset word embedding feature of the target audio signal;

[0014] A selection module, configured to select the target potential feature of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution;

[0015] A generation module is used to generate target description information corresponding to the target audio signal based on target potential features of multiple time steps and the target audio features.

[0016] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0017] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0018] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0019] In an embodiment of the present application, the target audio features of the target audio signal to be processed are first obtained, and the feature probability distribution of the current time step is further obtained through the target embedding features, the target audio features and the target audio features. The target embedding features are the original word embedding features of the previous time step or the preset word embedding features of the sample audio signal. Since the target embedding features are added in the process of determining the feature probability distribution, the accuracy of the feature probability distribution can be effectively improved. Moreover, since each text generation process will resample the target potential features of the current time step from the feature probability distribution based on the feature probability distribution, it can be achieved that the corresponding description words will not be generated based on the same target potential features at the same time step in each audio signal content analysis process. Therefore, different target description information can be generated based on the same audio signal while ensuring the accuracy of the generated target description information.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0022] In the attached picture:

[0023] Figure 1 A flow chart of an audio signal content analysis method provided by an embodiment of the present application is shown;

[0024] Figure 2 A flowchart of generating a posterior probability distribution provided by an embodiment of the present application is shown;

[0025] Figure 3 A flow chart showing the generation of a priori probability distribution provided by an embodiment of the present application is shown;

[0026] Figure 4 A flowchart showing a description word generation process provided by an embodiment of the present application is shown;

[0027] Figure 5 A flowchart of another audio signal content analysis method provided by an embodiment of the present application is shown;

[0028] Figure 6 A schematic diagram of the structure of an audio signal content analysis device provided by an embodiment of the present application is shown;

[0029] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0030] Figure 8 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0032] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0033] The audio signal content analysis method of the present application can be executed by a computing device, and the computing device can be a server, such as a server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, an advertising machine, an all-in-one machine, etc. The present application does not limit the device type and device quantity of the computing device.

[0034] At present, deep learning-based audio subtitle generation methods have made great progress, encapsulating the multimodal signal of audio through an encoder and using a decoder to generate target description information about the signal.

[0035] In one embodiment, generating target description information corresponding to an audio signal can be viewed as a sequence-to-sequence (Seq2Seq) problem, where the input sequence is an audio signal and the output sequence is the corresponding subtitle text. To solve this problem, an encoder-decoder model can be used to generate subtitle text.

[0036] The encoder-decoder model was originally used for machine translation tasks, but it is also applicable to text generation tasks. The model consists of two core components: an encoder and a decoder. The encoder processes the input sequence and converts it into a fixed-length representation; the decoder generates an output sequence from the representation.

[0037] In the audio-to-caption scenario, the encoder can be a speech recognition model that takes the audio signal as input and generates a set of high-level features, such as phonemes, words, or sentences. These features are passed to the decoder, which generates the corresponding caption text based on these features.

[0038] Specifically, the encoder can be a recurrent neural network (RNN), which inputs the audio signal frame by frame and outputs a fixed-length vector as a representation. The decoder can also be an RNN, which takes the representation and previously generated text as input and gradually generates the output sequence through autoregression.

[0039] The entire encoder-decoder model can be trained in an end-to-end manner to maximize the probability of the generated subtitle text. In the inference phase, given a new audio input, the encoder generates the corresponding representation, and then the decoder generates a text sequence based on the representation.

[0040] Although the above method can ensure the accuracy of the generated target description information, the target description information output for the same audio signal is unique. Due to the ambiguity of audio signals and the complexity of language representation, people's descriptions of audio content are more diverse. If the audio signal can correspond to multiple understanding methods, only outputting target description information for one understanding method may cause people who see the target description information to be unable to understand. Therefore, it is currently necessary to enrich the target description information for the same audio signal.

[0041] In order to solve the above problems, the embodiments of the present application provide an audio signal content analysis method, apparatus, device and storage medium. In the embodiments of the present application, the target audio features of the target audio signal to be processed are first obtained, and the feature probability distribution of the current time step is further obtained through the target embedding features, the target audio features and the target audio features. The target embedding features are the original word embedding features of the previous time step or the preset word embedding features of the sample audio signal. Since the target embedding features are added in the process of determining the feature probability distribution, the accuracy of the feature probability distribution can be effectively improved. Since each text generation process will resample the target potential features of the current time step from the feature probability distribution based on the feature probability distribution, it can be achieved that the corresponding description words will not be generated based on the same target potential features at the same time step in each audio signal content analysis process. Therefore, different target description information can be generated based on the same audio signal while ensuring the accuracy of the generated target description information.

[0042] Assuming that an audio signal can correspond to multiple understanding methods, target description information of one understanding method is output for the first time, but the user does not understand it. The user can receive an operation of regenerating the sentence input by the user, and output target description information of another understanding method until the output target description information can be understood by the user.

[0043] In each embodiment of the present application, an audio signal content analysis method of a target audio signal to be processed is used as an example for description. In addition, as for the execution subject, in each embodiment of the present application, a computing device is used as an example for description.

[0044] The following describes an audio signal content analysis method, apparatus, device, and storage medium according to an embodiment of the present application in conjunction with the accompanying drawings.

[0045] See also Figure 1 , the method specifically comprises the following steps:

[0046] S101. Obtain target audio features of a target audio signal to be processed.

[0047] The target audio signal may be voice, song, background sound or other audio.

[0048] Methods for extracting target audio features from the target audio signal may include: short-time Fourier transform, Mel-frequency cepstral coefficients, linear prediction coding coefficients, cepstral coefficients, frequency domain features, time domain features, perception-related features, chrominance frequency cepstral coefficients, short-time amplitude envelope, or extracting target audio features from the target audio signal through open source large-scale pretrained audio neural networks (Pretrained AudioNeural Networks, referred to as PANN).

[0049] S102: Determine the feature probability distribution of the current time step based on the target embedding feature and the target audio feature.

[0050] The target embedding feature is the original word embedding feature of the description word at the previous time step or the preset word embedding feature of the target audio signal.

[0051] It should be noted that in order to generate more diverse target description information, in the process of generating the target description information, the feature probability distribution of each time step can be generated in sequence.

[0052] The time step is used to indicate the description position of the descriptive word in the target description information. For example, if the target description information is "I am Chinese", the first time step is the description position of "I". The second time step is the description position of "I am", and so on.

[0053] Based on the target embedding features and the target audio features, determining the feature probability distribution of the current time step can be achieved based on deep learning.

[0054] For example, Hidden Markov Model, Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN) or Transformer Model, etc.

[0055] The above-mentioned various models can all be implemented as a priori model, so that the target embedding features and target audio features can be input into the priori model to obtain the feature probability distribution of the current time step.

[0056] Based on the target embedding features and the target audio features, determining the feature probability distribution of the current time step can be achieved based on a non-deep learning method.

[0057] For example, Gaussian Mixture Model (GMM), Support Vector Machine (SVM), Random Forest, Hidden Markov Model (HMM), etc.

[0058] Among them, the feature probability distribution can be the probability of each word embedding feature in the current time step. For example, in the above target description information "I am Chinese", the probability of "yes" in the second time step is 40%, the probability of "love" is 40%, and the probability of "call" is 20%.

[0059] The characteristic probability distribution can be uniform distribution, normal distribution, exponential distribution, Beta distribution or Dirichlet distribution, etc. The type of characteristic probability distribution can be flexibly set according to specific circumstances.

[0060] S103. Selecting target potential features of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution.

[0061] Since the feature probability distribution is the probability of each word in the time step, the word embedding features of each word can be determined from the feature probability distribution. For example, the multiple word embedding features of the second time step in the above target description information "I am Chinese" are the word embedding features corresponding to "yes", "love" and "call".

[0062] Among them, the resampling operation can be after determining multiple word embedding features, and the target potential feature of the current time step can be selected from the multiple word embedding features according to the probability corresponding to each word embedding feature represented by the feature probability distribution.

[0063] Based on the above assumptions, the probability that the target latent feature is the word embedding feature of "is" is 40%, the probability that it is the word embedding feature of "love" is 40%, and the probability that it is the word embedding feature of "call" is 20%.

[0064] If the sentence generation process is performed multiple times, the first target latent feature may be the word embedding feature of "yes", the second target latent feature may be the word embedding feature of "love", and the third target latent feature may be the word embedding feature of "call". This enables different word embedding features to be generated at the same time step in each audio signal content analysis process, and each target description information is generated by the word embedding features at different time steps, so that different target description information can be generated in different audio signal content analysis processes.

[0065] S104: Generate target description information corresponding to the target audio signal based on the target potential features and target audio features at multiple time steps.

[0066] The target description information may be video, audio, text or picture, etc.

[0067] In some embodiments, after determining the target potential features of the current time step, a descriptive word corresponding to the target potential features can be generated through a recurrent neural network or a converter model, and further, the target potential features of multiple time steps are obtained, and the descriptive words corresponding to the multiple time steps are gradually generated, so that the descriptive words corresponding to the multiple time steps are spliced ​​to generate target description information corresponding to the target audio signal.

[0068] An embodiment of the present application provides an audio signal content analysis method. In the embodiment of the present application, the target audio features of the target audio signal to be processed are first obtained, and the feature probability distribution of the current time step is further obtained through the target embedding features, the target audio features and the target audio features. The target embedding features are the original word embedding features of the previous time step or the preset word embedding features of the sample audio signal. Since, for example, the target embedding features are added in the process of determining the feature probability distribution, the accuracy of the feature probability distribution can be effectively improved. Moreover, since each text generation process will resample the target potential features of the current time step from the feature probability distribution based on the feature probability distribution, it can be achieved that the corresponding description words will not be generated based on the same word target potential features at the same time step in each audio signal content analysis process. Therefore, different target description information can be generated based on the same audio signal while ensuring the accuracy of the generated text.

[0069] In some embodiments, since there is no label target description information corresponding to the target audio in the data to be processed, the feature probability distribution of the current time step can be determined based on a priori model. The above S102, determining the feature probability distribution of the current time step based on the target embedding feature and the target audio feature, includes:

[0070] The target embedding features and the target audio features are input into the prior model of the audio signal content analysis model to determine the prior probability distribution of the current time step.

[0071] In some embodiments, the parameters of the prior model may be initialized, and the target audio features and the target embedding features may be obtained.

[0072] Since the prior model generally adopts an autoregressive approach to generate feature probability distributions for multiple time steps, before generating the feature probability distribution for the current time step, the description word generated based on the feature probability distribution for the previous time step can be obtained, and therefore, the word embedding feature of the description word can be obtained.

[0073] If the current time step is the first time step of the target description information corresponding to the target audio signal, then the preset word embedding feature of the target audio signal can be obtained. Generally, the preset word embedding feature is a word embedding feature representing the start character.

[0074] Furthermore, the word embedding features of the description words at the previous time step can be concatenated with the target audio features, or the preset word embedding features can be concatenated to obtain a concatenated word embedding feature, and the concatenated word embedding feature can be input into the prior model to obtain the feature probability distribution of the current time step.

[0075] The target embedding features and target audio features are input into the prior model of the audio signal content analysis model to obtain the prior probability distribution of the current time step, including:

[0076] If there is an original word embedding feature of the description word at the previous time step, the original word embedding feature and the target audio feature are input into the prior model to obtain the prior probability distribution of the current time step;

[0077] If the original word embedding feature does not exist, the preset word embedding feature and the target audio feature are input into the prior model to obtain the prior probability distribution of the current time step.

[0078] The target embedding features and target audio features are input into the prior model of the audio signal content analysis model to obtain the prior probability distribution of the current time step, including:

[0079] Input the target embedding feature and the target audio feature into the prior model to obtain the distribution parameter of the current time step, where the distribution parameter includes at least one of the following: mean and variance;

[0080] Draw a Gaussian distribution of the sample audio signal at the current time step based on the distribution parameters.

[0081] In some embodiments, the audio signal content analysis model also includes: a language decoding model and a text arrangement model, which generates target description information corresponding to the target audio signal based on target latent features and target audio features of multiple time steps, including: inputting word prior embedding features and target audio features of any time step into the language decoding model to determine the description word of the time step; inputting the description words of multiple time steps into the text arrangement model to determine the target description information corresponding to the target audio signal.

[0082] After selecting the target embedding feature of any time step in the feature probability distribution, the target audio feature and the target embedding feature can be concatenated, and the concatenated feature can be input into the language decoding model to obtain the description word of the time step.

[0083] Further, description words of multiple time steps are obtained by autoregression, and the description words of multiple time steps are input into the text arrangement model to obtain target description information corresponding to the target audio signal.

[0084] In some embodiments, the audio signal content analysis model is obtained in advance through training, and the training process of the audio signal content analysis model includes:

[0085] Preprocessing the sample data to determine sample audio features of the sample audio signal and multiple label word embedding features of the label sentence, the sample data including the sample audio signal and the label sentence;

[0086] Inputting the sample audio features and the multiple label word embedding features into the posterior model of the initial audio signal content analysis model and inputting the sample embedding features and the sample audio features into the initial prior model of the initial audio signal content analysis model, and calculating a first loss function value;

[0087] A second loss function value is calculated based on a first output result obtained by inputting the sample audio features and the multiple label words into the posterior model and an initial language decoding model of the initial audio signal content analysis model;

[0088] Based on inputting the first result into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model, a third loss function value is calculated;

[0089] Based on the first loss function value, the second loss function value and the third loss function value, the model parameters of the audio signal content analysis model are adjusted to continue training until the preset fourth training completion condition is met to obtain a trained audio signal content analysis model.

[0090] Among them, the audio signal content analysis model can be implemented as a variational autoencoder, which can obtain the first loss function value of the initial prior model, the second loss function value of the initial language decoding model, the third loss function value of the initial text arrangement model, the first weight coefficient of the first loss function, the second weight coefficient of the second loss function and the third weight coefficient of the third loss function.

[0091] Among them, the first weight coefficient, the second weight coefficient and the third weight coefficient can be flexibly set based on actual conditions.

[0092] In one embodiment, the total loss function value of the audio signal content analysis model can be determined based on the first loss function value of the initial prior model, the second loss function value of the initial language decoding model, the third loss function value of the initial text arrangement model, the first weight coefficient of the first loss function, the second weight coefficient of the second loss function and the third weight coefficient of the third loss function, and the model parameters of the audio signal content analysis model can be adjusted based on the total loss function value to continue training until the preset fourth training completion condition is met to obtain a trained audio signal content analysis model.

[0093] In one embodiment, the total loss function value may be determined based on the following formula:

[0094] L total =αL kl+βL ce +γL global (1)

[0095] Among them, L total is the total loss function, α is the first weight coefficient, L kl is the first loss function, β is the second weight coefficient, L ce is the second loss function, γ is the third weight coefficient, L global For the third loss function, after determining the first target loss function value, the second target loss function value and the third target loss function value, at least one weight coefficient among the first weight coefficient, the second weight coefficient and the third weight coefficient can be adjusted.

[0096] Among them, the fourth training completion condition may be that the fourth loss function value is less than the corresponding threshold, or the number of training iterations reaches a preset number.

[0097] Among them, the preprocessing operation can be to extract sample audio features from the sample audio signal in the manner of S101 above, and perform a word segmentation operation on the label sentence to obtain multiple words or multiple characters of the label sentence. If the label sentence is in Chinese, then the multiple characters of the label sentence can be obtained through the word segmentation operation. If the label sentence is in English, the multiple words of the label sentence can be obtained through the word segmentation operation, and the multiple words or multiple characters are further encoded to obtain word embedding features of the multiple words or multiple characters.

[0098] In some embodiments, the sample audio features and the multiple label word embedding features are input into the posterior model of the initial audio signal content analysis model, and the sample embedding features and the sample audio features are input into the initial prior model of the initial audio signal content analysis model, and the first loss function value is calculated, including:

[0099] Input the sample audio features and multiple label word embedding features into the posterior model to determine the sample posterior probability distribution at the current time step;

[0100] Input the sample embedding features and the sample audio features into the initial prior model to determine the sample prior probability distribution at the current time step, where the sample embedding features are the label word embedding features of the description words at the previous time step or the preset word embedding features;

[0101] Based on the sample posterior probability distribution and the sample prior probability distribution, the first loss function value is calculated.

[0102] The sample audio features and multiple sample word embedding features are input into the posterior model to obtain the sample posterior probability distributions corresponding to the multiple time steps, and the sample posterior probability distributions of the current time step are obtained from the sample posterior probability distributions corresponding to the multiple time steps.

[0103] Since the posterior probability distribution is based on the content analysis of the labeled audio signal, and the label sentence is the target description information corresponding to the target audio signal, the posterior probability distribution is accurate, and thus the posterior probability distribution can be used as a label training prior model.

[0104] Given the observed data, the posterior probability distribution of the current time step can be determined based on the observed data, where the posterior probability distribution can be calculated by the following formula:

[0105]

[0106] q(Z post |X,A) is the posterior probability distribution for each time step t, Z post is a latent variable, X, A represent observed data and auxiliary variables respectively, where X is the embedding feature of multiple sample words, X = (x1,…,x T ), A is the sample audio feature, and the formula is the product of the probability density functions of the posterior distribution of time steps t from 1 to T. This means that we consider the posterior distribution of each time step together to obtain the posterior probability distribution of the entire time series.

[0107] Furthermore, the posterior probability distribution of the current time step can be obtained from the posterior probability distribution of the entire time series.

[0108] The prior probability distribution can be calculated by the following formula:

[0109]

[0110] is the prior probability distribution for each time step t, is a tuple of sample embedding features of a generated word, A is the sample audio feature, represents the estimated value of the observed data up to time step t. Specifically, the formula is the product of the probability density function of the prior distribution of time steps t from 1 to T. This means that we consider the estimated value of the observed data before each time step together with the auxiliary variables to obtain the prior probability distribution of the entire time series.

[0111] The first loss function value can be calculated by the following formula:

[0112]

[0113] Where L kl is the first loss function, L kl It means that by calculating the posterior distribution q(Z|X,A) and the prior distribution The KL divergence between them is used to measure the difference between them. q(Z|X,A) represents the posterior distribution. represents the prior distribution. This KL divergence can be understood as the information difference between the posterior distribution and the prior distribution given the observed data and auxiliary variables. If L kl The smaller it is, the closer the posterior distribution is to the prior distribution; conversely, L kl The larger the value, the greater the difference between the two distributions.

[0114] To explain the generation process of the posterior probability distribution in detail, Figure 2 A flowchart of generating a posterior probability distribution provided by an embodiment of the present application is shown, assuming that the posterior probability distribution is a Gaussian distribution, such as Figure 2 As shown in the figure, the sample audio features and multiple sample word embedding features are input into the posterior model, and the posterior distribution means of multiple time steps and the posterior distribution variances of multiple time steps are obtained through two linear layers in the posterior model. Further, a Gaussian distribution graph of each time step can be constructed based on the posterior distribution mean and posterior distribution variance of each time step.

[0115] To explain the generation process of prior probability distribution in detail, Figure 3 A flowchart of generating a priori probability distribution provided by an embodiment of the present application is shown, assuming that the priori probability distribution is a Gaussian distribution, such as Figure 3 As shown, the sample embedding features and the sample audio features are input into the initial prior model, and the prior distribution mean and the prior distribution variance of the current time step are obtained through the two current layers in the initial prior model. Further, the Gaussian distribution map of the current time step can be constructed based on the prior distribution mean and the prior distribution variance of the current time step.

[0116] In some embodiments, the first output result is a sample posterior probability distribution at the current time step, and the second loss function value is calculated based on the first output result obtained by inputting the sample audio features and the multiple label words into the initial prior model and the initial language decoding model of the initial audio signal content analysis model, including:

[0117] Based on the sample posterior probability distribution, the target potential features of the current time step are selected from the sample posterior probability distribution through a resampling operation;

[0118] Obtain the sample description words of the current time step determined by the target potential feature through the initial language decoding model;

[0119] Generate label description words corresponding to sample description words;

[0120] Based on the label description words and the sample description words, the second loss function value is calculated.

[0121] Among them, the sample posterior probability distribution of the current time step output by the delay model can be obtained;

[0122] A target sample word embedding feature is selected from multiple sample word embedding features of the sample posterior probability distribution based on the sample posterior probability distribution.

[0123] When the sample posterior probability distribution of the current time step is determined, a target latent feature can be selected from multiple latent features of the sample posterior probability distribution, and the target latent feature can be concatenated with the target audio feature.

[0124] It should be noted that in all the embodiments of the present application, the splicing method can be flexibly set according to the actual situation, for example, it can be linear splicing, simple splicing or multi-layer splicing, etc.

[0125] The sample description word at the current time step can be determined by the following formula:

[0126]

[0127] in, is the sample description word, Z and A are the target sample word embedding features and sample audio features respectively, is the sample description word at time step t, z t is the target sample word embedding feature at time step t.

[0128] Given the target sample word embedding features and sample audio features at each time step, the sample description words at each time step can be obtained according to formula (4).

[0129] After obtaining the descriptors at any time step, the label descriptors corresponding to the sample descriptors may be generated by a preset method, wherein the preset method may be a maximum likelihood estimation method or a negative log-likelihood optimization method, and the like.

[0130] The second loss function value can be determined based on the following formula:

[0131]

[0132] Among them, L ce is the second loss function, is the expectation, Z is the sample description word, X is the label description word, and A is the sample audio feature.

[0133] It represents the variational inference of the sample description word Z given the label description word X and the sample audio feature A, that is, the posterior distribution q(Z|X,A) is obtained by approximation method.

[0134] logp(X|Z,A) represents the logarithm of the probability that the model generates the label description word X given the sample description word Z and the audio feature A.

[0135] By minimizing Lce , we can maximize the probability of the model generating the output label description word X given the label description word X and the sample audio feature quantity A, thereby improving the performance of the model.

[0136] To explain the generation process of description words at each time step in detail, Figure 4 A flowchart of a description word generation process provided by an embodiment of the present application is shown.

[0137] like Figure 4 As shown:

[0138] First, determine whether the current process is a training process. If it is a training process, obtain the sample posterior probability distribution of the current time step output by the posterior model, and select the target sample word embedding feature from multiple sample word embedding features of the sample posterior probability distribution based on the sample posterior probability distribution; obtain the sample description word of the current time step determined by the initial language decoding model using the target sample word embedding feature.

[0139] If it is not a training process, obtain the prior probability distribution of the current time step output by the trained prior model, select the target potential feature from multiple potential features of the prior probability distribution based on the prior probability distribution; obtain the sample description word of the current time step determined by the initial language decoding model based on the target sample word embedding feature.

[0140] The second output result is the sample description word of the current time step. Based on the first result input into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model, a third loss function value is calculated, including:

[0141] Obtaining sample audio signals through an initial text arrangement model to determine sample description information corresponding to the sample audio signals;

[0142] Based on the label statement and sample description information, the third loss function value is calculated.

[0143] The label sentence can be obtained from the training sample data. After obtaining the sample description words corresponding to multiple time steps, the multiple sample description words are input into the initial text arrangement model to determine the sample description information, wherein the sample description words are obtained based on an untrained language decoding model.

[0144] Based on the sample description information and the label statement, calculating the third loss function value may be obtaining first global information of the sample description information and second global information of the label statement, and determining the third loss function value by the difference value between the first global information and the second global information.

[0145] The third loss function value can be determined based on the following formula:

[0146] Lglobal =cos(e pre ,e gt ) (6)

[0147] Among them, L global is the third loss function, cos(·) is the cosine loss function, e pre is the first global information, e gt It is the second global information.

[0148] To explain the training process of the audio signal content analysis model in detail, Figure 5 A flow chart of a training process provided by an embodiment of the present application is shown.

[0149] like Figure 5 As shown:

[0150] First, the sample audio signals and label sentences in the sample data are preprocessed. For example, the sample audio features of the sample audio signals are extracted based on the PANN audio pre-training model, and the label sentences are segmented and the word embedding feature encoding operations are performed to obtain multiple sample word embedding features of the label sentences.

[0151] Input the audio signal features and the sample embedding features of the previous time step into the initial prior model to obtain the sample prior probability distribution of the current time step output by the initial prior model; input the sample audio features and multiple sample word embedding features into the posterior model at the same time to obtain the sample posterior probability distribution of the current time step output by the posterior model;

[0152] Based on the sample posterior probability distribution and sample prior probability distribution, calculate the first loss function value

[0153] Get the sample posterior probability distribution of the current time step output by the posterior model;

[0154] A target latent feature is selected from multiple sample latent features of the sample posterior probability distribution based on the sample posterior probability distribution.

[0155] Obtain the sample description word of the current time step determined by the initial language decoding model based on the target sample word embedding feature; generate a sample description word corresponding to the sample description word; and calculate the second loss function value based on the label description word and the sample description word.

[0156] Obtain a label sentence corresponding to the sample audio signal; obtain the sample audio signal through the posterior model and the initial language decoding model, and determine the sample description words corresponding to multiple time steps; input the multiple sample description words into the initial text arrangement model to determine the label sentence; based on the label sentence and the label sentence, calculate the value of the third loss function.

[0157] Based on the first loss function value of the initial prior model, the second loss function value of the initial language decoding model, the third loss function value of the initial text arrangement model, the first weight coefficient of the first loss function, the second weight coefficient of the second loss function and the third weight coefficient of the third loss function, the total loss function value of the audio signal content analysis model is determined, and the model parameters of the audio signal content analysis model are adjusted based on the total loss function value to continue training until the preset fourth training completion condition is met to obtain a trained audio signal content analysis model.

[0158] The present application also provides an audio signal content analysis device, which is used to perform the audio signal content analysis method provided in any of the above embodiments. Figure 6 As shown, the device comprises:

[0159] An acquisition module 601 is used to acquire target audio features of a target audio signal to be processed;

[0160] A determination module 602 is used to determine a feature probability distribution of a current time step based on a target embedding feature and a target audio feature, where the target embedding feature is an original word embedding feature of a description word at a previous time step or a preset word embedding feature of a target audio signal;

[0161] A selection module 603 is used to select a target potential feature of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution;

[0162] The generating module 604 is used to generate target description information corresponding to the target audio signal based on the target potential features and the target audio features at multiple time steps.

[0163] An embodiment of the present application provides an audio signal content analysis device. In the embodiment of the present application, the target audio features of the target audio signal to be processed are first obtained, and the feature probability distribution of the current time step is further obtained through the target embedding features, the target audio features and the target audio features. The target embedding features are the original word embedding features of the previous time step or the preset word embedding features of the sample audio signal. Since, for example, the target embedding features are added in the process of determining the feature probability distribution, the accuracy of the feature probability distribution can be effectively improved. Moreover, since each text generation process will resample the target potential features of the current time step from the feature probability distribution based on the feature probability distribution, it can be achieved that the corresponding description words will not be generated based on the same word target potential features at the same time step in each audio signal content analysis process. Therefore, different target description information can be generated based on the same audio signal while ensuring the accuracy of the generated target description information.

[0164] In some embodiments, the determination module 602 is specifically configured to:

[0165] The target embedding features and the target audio features are input into the prior model of the audio signal content analysis model to determine the prior probability distribution of the current time step.

[0166] In some embodiments, the audio signal content analysis model further includes: a language decoding model and a text arrangement model, and the generation module 604 is specifically used to:

[0167] Input the word prior embedding features and target audio features at any time step into the language decoding model to determine the description word of the time step;

[0168] The description words of multiple time steps are input into the text arrangement model to determine the target description information corresponding to the target audio signal.

[0169] In some embodiments, the audio signal content analysis model is obtained in advance through training, and the training process of the audio signal content analysis model includes:

[0170] Preprocessing the sample data to determine sample audio features of the sample audio signal and multiple label word embedding features of the label sentence, the sample data including the sample audio signal and the label sentence;

[0171] Inputting the sample audio features and the multiple label word embedding features into the posterior model of the initial audio signal content analysis model and inputting the sample embedding features and the sample audio features into the initial prior model of the initial audio signal content analysis model, and calculating a first loss function value;

[0172] A second loss function value is calculated based on a first output result obtained by inputting the sample audio features and the multiple label words into the posterior model and an initial language decoding model of the initial audio signal content analysis model;

[0173] Based on inputting the first result into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model, a third loss function value is calculated;

[0174] Based on the first loss function value, the second loss function value and the third loss function value, the model parameters of the audio signal content analysis model are adjusted to continue training until the preset fourth training completion condition is met to obtain a trained audio signal content analysis model.

[0175] In some embodiments, the computing module is used to:

[0176] Input the sample audio features and multiple label word embedding features into the posterior model to determine the sample posterior probability distribution at the current time step;

[0177] Input the sample embedding features and the sample audio features into the initial prior model to determine the sample prior probability distribution at the current time step, where the sample embedding features are the label word embedding features of the description words at the previous time step or the preset word embedding features;

[0178] Based on the sample posterior probability distribution and the sample prior probability distribution, the first loss function value is calculated.

[0179] In some embodiments, the first output result is a sample posterior probability distribution at the current time step, and the computing module is further used to:

[0180] Based on the sample posterior probability distribution, the target potential features of the current time step are selected from the sample posterior probability distribution through a resampling operation;

[0181] Obtain the sample description words of the current time step determined by the target potential feature through the initial language decoding model;

[0182] Generate label description words corresponding to sample description words;

[0183] Based on the label description words and the sample description words, the second loss function value is calculated.

[0184] In some embodiments, the second output result is a sample description word at the current time step, and the computing module is further used to:

[0185] Obtaining sample audio signals through an initial text arrangement model to determine sample description information corresponding to the sample audio signals;

[0186] Based on the label statement and sample description information, the third loss function value is calculated.

[0187] The audio signal content analysis device provided in the embodiment of the present application and the audio signal content analysis method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0188] The present application also provides an electronic device to perform the above-mentioned audio signal content analysis method. Figure 7 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 7 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the audio signal content analysis method provided in any of the aforementioned embodiments of the present application.

[0189] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0190] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 701 is used to store programs, and the processor 700 executes the program after receiving an execution instruction. The audio signal content analysis method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 700, or implemented by the processor 700.

[0191] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 700. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0192] The electronic device provided in the embodiment of the present application and the audio signal content analysis method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0193] The present application also provides a computer-readable storage medium corresponding to the audio signal content analysis method provided in the above embodiment. Figure 8 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the audio signal content analysis method provided by any of the aforementioned embodiments will be executed.

[0194] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0195] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the audio signal content analysis method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0196] It should be noted that:

[0197] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0198] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0199] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0200] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for analyzing audio signal content, characterized in that: include: Acquire target audio features of a target audio signal to be processed; Determine a feature probability distribution of a current time step based on a target embedding feature and the target audio feature, wherein the target embedding feature is an original word embedding feature of a description word at a previous time step or a preset word embedding feature of the target audio signal; The original word embedding feature is a word embedding feature of a description word generated by a feature probability distribution of a previous time step, and the preset word embedding feature is a word embedding feature representing a start character; Selecting the target potential feature of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution; Generate target description information corresponding to the target audio signal based on target potential features and the target audio features at multiple time steps; The determining, based on the target embedding feature and the target audio feature, the feature probability distribution of the current time step comprises: Inputting the target embedding feature and the target audio feature into a prior model of the audio signal content analysis model to determine a prior probability distribution of the current time step; The audio signal content analysis model is obtained in advance through training, and the training process of the audio signal content analysis model includes: Preprocessing the sample data to determine a sample audio feature of a sample audio signal and a plurality of label word embedding features of a label sentence, wherein the sample data includes the sample audio signal and the label sentence; Input the sample audio features and the multiple label word embedding features into the posterior model of the initial audio signal content analysis model, and input the sample embedding features and the sample audio features into the initial prior model of the initial audio signal content analysis model, and calculate a first loss function value; Calculate a second loss function value based on a first output result obtained by inputting the sample audio feature and the multiple label word input features into the posterior model and an initial language decoding model of the initial audio signal content analysis model; Based on inputting the first output result into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model, a third loss function value is calculated; Based on the first loss function value, the second loss function value and the third loss function value, the model parameters of the audio signal content analysis model are adjusted, and training is continued until a preset fourth training completion condition is met to obtain a trained audio signal content analysis model.

2. The method according to claim 1, characterized in that The audio signal content analysis model further includes: a language decoding model and a text arrangement model. The target description information corresponding to the target audio signal is generated based on the target word potential features and the target audio features at multiple time steps, including: Inputting the target latent feature and the target audio feature at any time step into the language decoding model to determine a description word for the time step; The description words of multiple time steps are input into the text arrangement model to determine the target description information corresponding to the target audio signal.

3. The method according to claim 1, characterized in that The step of inputting the sample audio features and the multiple label word embedding features into the posterior model of the initial audio signal content analysis model and inputting the sample embedding features and the sample audio features into the initial prior model of the initial audio signal content analysis model to calculate a first loss function value includes: Inputting the sample audio features and the multiple label word embedding features into the posterior model to determine the sample posterior probability distribution at the current time step; Inputting the sample embedding feature and the sample audio feature into the initial prior model to determine the sample prior probability distribution at the current time step, wherein the sample embedding feature is the label word embedding feature of the description word at the previous time step or the preset word embedding feature; Based on the sample posterior probability distribution and the sample prior probability distribution, a first loss function value is calculated.

4. The method according to claim 3, characterized in that The first output result is a sample posterior probability distribution at the current time step, and the second loss function value is calculated based on the first output result obtained by inputting the sample audio feature and the multiple label word features into the initial prior model and the initial language decoding model of the initial audio signal content analysis model, including: Based on the sample posterior probability distribution, the target potential feature of the current time step is selected from the sample posterior probability distribution through a resampling operation; Obtaining a sample description word of the target potential feature at the current time step determined by an initial language decoding model; Generate a label description word corresponding to the sample description word; Based on the label description word and the sample description word, a second loss function value is calculated.

5. The method according to claim 4, characterized in that The second output result is a sample description word at the current time step, and the third loss function value is calculated based on inputting the first output result into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model, including: Obtaining sample description information corresponding to the sample audio signal determined by an initial text arrangement model for the sample audio signal; Based on the label statement and the sample description information, a third loss function value is calculated.

6. An audio signal content analysis device, characterized in that: include: An acquisition module, used for acquiring target audio features of a target audio signal to be processed; A determination module, configured to determine a feature probability distribution of a current time step based on a target embedding feature and the target audio feature, wherein the target embedding feature is an original word embedding feature of a description word at a previous time step or a preset word embedding feature of the target audio signal; The original word embedding feature is a word embedding feature of a description word generated by a feature probability distribution of a previous time step, and the preset word embedding feature is a word embedding feature representing a start character; The determining, based on the target embedding feature and the target audio feature, the feature probability distribution of the current time step comprises: Inputting the target embedding feature and the target audio feature into a prior model of the audio signal content analysis model to determine the prior probability distribution of the current time step; The audio signal content analysis model is obtained in advance through training, and the training process of the audio signal content analysis model includes: Preprocessing the sample data to determine a sample audio feature of a sample audio signal and a plurality of label word embedding features of a label sentence, wherein the sample data includes the sample audio signal and the label sentence; Input the sample audio features and the multiple label word embedding features into the posterior model of the initial audio signal content analysis model, and input the sample embedding features and the sample audio features into the initial prior model of the initial audio signal content analysis model, and calculate a first loss function value; Calculate a second loss function value based on a first output result obtained by inputting the sample audio feature and the multiple label word input features into the posterior model and an initial language decoding model of the initial audio signal content analysis model; Calculating a third loss function value based on inputting the first output result into the second output result of the initial language decoding model and the text arrangement model of the initial audio signal content analysis model; Based on the first loss function value, the second loss function value, and the third loss function value, adjust the model parameters of the audio signal content analysis model, and continue training until a preset fourth training completion condition is met to obtain a trained audio signal content analysis model; A selection module, configured to select the target potential feature of the current time step from the feature probability distribution through a resampling operation based on the feature probability distribution; A generation module is used to generate target description information corresponding to the target audio signal based on target potential features of multiple time steps and the target audio features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor runs the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Keyword detection method and related device

    CN109671433A

  • Training data generation method and device, and computer readable storage medium

    CN112037769A

  • Speech recognition method, training method of related model, related equipment and device thereof

    CN112259100A

  • Multi-modal signal content analysis method and device, electronic equipment and storage medium

    CN115952255A

  • Voice keyword recognition method and device and related equipment

    CN116564279A