Environment audio open set identification method, model and electronic equipment

Abstract

The invention discloses an environment audio open set identification method, a model and electronic equipment. The method comprises the steps of performing preliminary classification, obtaining a first output vector # imgabs0 #, performing dimension reduction to obtain a second output vector # imgabs1 #, performing training by utilizing a loss function # imgabs2 #, a density loss # imgabs3 # and an offset loss # imgabs6 # between # imgabs4 # and # imgabs5 # of a generative adversarial network, dividing # imgabs7 # into C mutually different subsets # imgabs8 # and # imgabs9 # according to a classification label, distributing a mutual inversion point # imgabs11 # for # imgabs10 # and maximizing the difference between # imgabs12 # and # imgabs13 #, distributing an attractor # imgabs15 # for # imgabs14 # to control and open set data is controlled in a low-magnitude region by using # imgabs16 #, and attractor-reciprocal point loss # imgabs17 # is calculated. According to the environment audio open set identification method, unknown categories can be identified.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental audio recognition, and in particular to an environmental audio open set recognition method, an environmental audio open set recognition model and an electronic device. Background Art

[0002] Acoustic object recognition has achieved remarkable results using various traditional machine learning algorithms and deep learning methods under the closed set assumption. However, the closed set assumption has limitations in real-world scenarios. For example, in the field of autonomous driving, it is unrealistic to collect samples of all possible categories of acoustic events for model training. Even in relatively controlled indoor environments, encountering objects outside the training set is inevitable. When models trained based on the closed set paradigm are directly applied to open environments, they tend to confidently classify unknown categories as one of the known categories in the training set. This is because classification problems often generate probabilities through the Softmax layer of the output layer, and the dimension of this layer is fixed. This idealized closed set assumption, especially in practical applications such as audio surveillance and autonomous driving, can lead to false alarms or misidentifications, causing serious safety issues. Although open set recognition algorithms have been studied in non-audio application fields, they have not fully utilized the generative model, and the prototype point learning is relatively single, without considering the risks of open space, and cannot guarantee good intra-class aggregation. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an open set recognition method for ambient audio, which can recognize unknown categories and maintain good intra-class aggregation while considering the risk of open space.

[0004] The invention also proposes an environmental audio open set recognition model.

[0005] The invention also provides an electronic device.

[0006] According to the first aspect of the present invention, the method for identifying an open set of ambient audio includes: Use the closed set classification algorithm to preliminarily classify the ambient audio and obtain the first output vector and the second output vector after dimensionality reduction ; The As a generated training set, and using the kernel density estimation to generate adversarial networks to generate the first edge samples , after dimensionality reduction, we get the second edge sample ; Using the loss function of the generative adversarial network , the density loss of the kernel density estimate and the With the The offset loss between To train: In the formula, , is a constant coefficient; Representing the training of the generative adversarial network; G , D Represent the generator and discriminator respectively; According to C classification labels, Divide into C distinct subsets , , for the Assign reciprocal points and maximize the With the The difference, for each of the Distribution Attractor To control the The sample distribution of The open set data is controlled in the low-level region, and the attractor-reciprocal point loss is calculated according to the following formula : In the formula, Expressing the Loss of restraint; Expressing the Loss of restraint; is the distance cross entropy loss; It is the loss of constraining the data distribution range; , It is a hyperparameter used to control the weight of the loss function.

[0007] The environmental audio open set recognition method according to the embodiment of the present invention can recognize unknown categories and maintain good intra-class aggregation while considering the risk of open space.

[0008] In addition, the open set recognition method for ambient audio according to the embodiment of the present invention also has the following additional technical features: According to some embodiments of the present invention, the environmental audio open set recognition method further includes: Get each of the The set of centers of ; Calculate the Middle Data points With center point The distance between , ; Using the The distribution function and kernel function fitting probability density function , calculated according to the following formula Belongs to the interval [ , ] probability: According to the following formula, the : In the formula, is the number of the second edge samples; Encode the first label; For the The data points in The distance between .

[0009] In some embodiments of the present invention, the : In the formula, For the The sample mean of the class.

[0010] In some specific embodiments of the present invention, the environmental audio open set recognition method further includes: Get the Each data point in The eigenvector of ; Calculate the With each pair of and stated The distance between them is calculated according to the following formula: Classified as kind: In the formula, Indicates the category corresponding to the maximum value of the C results in brackets.

[0011] In some embodiments of the present invention, the : In the formula, is the number of batches currently being trained; is the second label encoding; is the belong Probability of class; is a hyperparameter used to control the distance-to-probability conversion; is a hyperparameter used to control the denominator from being zero.

[0012] In some specific embodiments of the present invention, the , : In the formula, is a hyperparameter used to control the weight of Euclidean distance and dot product.

[0013] In some embodiments of the present invention, the : In the formula, is the boundary value.

[0014] In some specific embodiments of the present invention, the : In the formula, Represents the calculation of the bi-norm.

[0015] According to the second aspect of the present invention, the open set recognition model of the environmental audio includes: a first classifier, the first classifier includes a feature extraction layer, a convolution layer, a pre-logit layer, a first logit layer and a first kernel density estimation layer, the first classifier uses a closed set classification algorithm to preliminarily classify the environmental audio, and the pre-logit layer is used to output a first output vector The first logit layer is used to Reduce dimension and output the second output vector , the first kernel density estimation layer is used to generate a probability density curve for each category; a generative adversarial device, the generative adversarial device includes a generator, a discriminator, a second logit layer and a second kernel density estimation layer, the generator uses the kernel density estimation effect of the second kernel density estimation layer to generate a first edge sample The second logit layer is used to Reduce dimension and output the second edge sample To the second kernel density estimation layer, the discriminator is used to determine the Does it belong to the , the generative adversarial machine uses the loss function of the generative adversarial network , the density loss of the kernel density estimate and the With the The offset loss between training; a second classifier, wherein the second classifier comprises a plurality of convolutional layers, and the second classifier is used to classify the , Perform attractor-reciprocal point training, according to The classification labels will be described Divided into distinct subsets , , for the Assign reciprocal points and maximize the With the The difference, for each of the Distribution Attractor To control the The sample distribution of The open set data is controlled in the low-level area, according to the Constraint loss , Constraint loss , distance cross entropy loss And the loss of constraining the data distribution range Calculate attractor-reciprocal point loss .

[0016] The environmental audio open set recognition model according to the embodiment of the present invention can recognize unknown categories and maintain good intra-class aggregation while considering open space risks.

[0017] According to an embodiment of the third aspect of the present invention, an electronic device includes a processor and a memory, wherein the processor is connected to the memory, and the memory is used to store a computer program. When the computer program is executed by the processor, the method for identifying an open set of ambient audio as described in the embodiment of the first aspect of the present invention is implemented.

[0018] The electronic device according to the embodiment of the present invention can identify unknown categories and maintain good intra-category aggregation while considering open space risks.

[0019] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of the structure of an open-set recognition model for ambient audio according to an embodiment of the present invention; Figure 2 is a schematic structural diagram of a first classifier according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a generative adversarial device according to an embodiment of the present invention; Figure 4 is a schematic structural diagram of a second classifier according to an embodiment of the present invention; Figure 5 : is a schematic diagram of sample distribution according to an embodiment of the present invention; wherein (a) is the original sample distribution; (b) is the original GAN ​​generated sample distribution; (c) is the generated sample distribution without Offset Loss constraint; (d) is the Offset Loss + Density Loss generated sample distribution; Figure 6 is a schematic diagram of training results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] The following describes an environmental audio open-set recognition method and an environmental audio open-set recognition model according to an embodiment of the present invention with reference to the accompanying drawings.

[0023] like Figure 1-Figure 6 As shown, the environmental audio open set recognition model according to an embodiment of the present invention includes: a first classifier, a generative adversarial classifier and a second classifier.

[0024] Specifically, the first classifier includes a feature extraction layer, a two-dimensional convolution layer, a pre-logit layer, a first logit layer, and a first kernel density estimation layer. The first classifier uses a closed set classification algorithm to preliminarily classify the ambient audio, which is a closed set data set. The feature extraction layer uses MFCC (Mel-scale Frequency Cepstral Coefficients) for feature extraction. After the two-dimensional convolution layer, the pre-logit layer outputs the first output vector , is used as a training set for sample generation. Since the samples at the edge of the pre-logit layer are not necessarily aligned with the edge samples of the first logit layer, constraints need to be imposed in the first logit layer. Reduce dimension and output the second output vector The first kernel density estimation layer is used to generate a probability density curve for each category.

[0025] Since open set samples are unknown, without open set samples as reference, even if the closed set classification algorithm works well, it is difficult to apply it to open set recognition problems. Interference samples generally surround the sample cluster, which brings great difficulties to the judgment of the model. Therefore, the present invention generates edge samples for data enhancement based on kernel density estimation constrained generative adversarial networks (GAN).

[0026] Get GAN can then be trained. The generative adversarial model includes a generator, a discriminator, a second logit layer, and a second kernel density estimation layer. Through noise sampling, the generator receives a random noise distribution , and generate as much as possible Close forged samples, i.e. first edge samples , in order to deceive the discriminator. The discriminator determines that the input sample is a real sample Or forge samples, and achieve the training goal through adversarial training between the two. The second logit layer is used to Reduce dimension and output the second edge sample To the second kernel density estimation layer. The loss function of the generated adversarial network is calculated according to the following formula : (1) In the formula, Representing the training of the generative adversarial network; G , D Represent the generator and discriminator respectively; Indicates that the sample comes from probability; Indicates that the sample comes from The optimization objective function can make the distribution of forged samples as close as possible to the distribution of original samples, such as Figure 5 (a) and Figure 5 (b) as shown.

[0027] because It is a high-dimensional vector, and the size of the environmental audio dataset is usually much smaller than that of the image and text datasets, so it is difficult to fit its distribution. In addition, we need to generate edge data for each type of sample. If the original data is understood as a number of hypersphere distributions, then the edge samples are wrapped around the periphery of these hyperspheres, which is also difficult to visualize. In order to simplify the modeling, Divide into Subset , after calculating the center point of each type of sample, we get the center point set , and calculate the corresponding distance to get a new one-dimensional data set : (2) In the formula, For the known class The distance set of the class, is the distance function. Middle Data points With center point The distance between , . Converting to a distance set operation can reduce the dimension of high-dimensional data points to one-dimensional data points, which is more convenient for modeling. When it is plotted as a histogram, the edge samples that need to be generated are equivalent to the samples at the tail of the histogram. Similarly, Express The data is converted into corresponding distance sets.

[0028] Next, the kernel density estimation of the second kernel density estimation layer is used to estimate Modeling. Assume The cumulative distribution function of , the probability density function is ,but: (3) Introducing the empirical distribution function : (4) yes An unbiased estimate of In observation The number of occurrences and The ratio of , and substitute it into And after introducing the kernel function: (5) In the formula, is the kernel function, is the bandwidth of the kernel function.

[0029] Therefore, using The distribution function and kernel function fitting probability density function , calculated according to the following formula Belongs to the interval [ , ] probability: (6) Since the data points generated are around each type of sample, that is, the data points farther from the corresponding center point, the following formula is used to calculate : (7) In the formula, is the number of second edge samples; Encode the first label; for The data points in The distance between .

[0030] In addition, in order to make the generated samples distributed as evenly as possible around the corresponding category hypersphere, and avoid the network output shifting in a certain direction of the hypersphere (such as Figure 5 (c) shows that the Offset Loss constraint needs to be applied to make the network converge in the expected direction. In fact, the fake samples to be generated are understood as the spherical shell of a hypersphere. This spherical shell should share the same center with the corresponding category hypersphere. The distance between the center of the spherical shell and the center of the hypersphere can be constrained to ensure that the generated fake samples are evenly distributed around the hypersphere. Therefore, the offset loss is calculated according to the following formula: : (8) In the formula, For the The sample mean of the class.

[0031] Therefore, the generative adversarial machine uses the loss function of the generative adversarial network , density loss for kernel density estimation as well as and The offset loss between To train: (9) In the formula, , is a constant coefficient; Represents the training of a generative adversarial network; G , D They represent the generator and the discriminator respectively.

[0032] The second classifier includes multiple convolutional layers. , Perform attractor-reciprocal point training, according to The classification labels will Divided into distinct subsets , .for Assign reciprocal points And maximize and The role of the reciprocal point is to push the samples as far as possible. Distribution Attractor To control The role of the attractor is to ensure that the distribution of one type of samples is as small as possible while pushing the reciprocal points away, and to use The open set data is controlled in the low-level region. The attractor-reciprocal point loss is calculated according to the following formula : (10) In the formula, Express Loss of restraint; Express Loss of restraint; is the distance cross entropy loss; It is the loss of constraining the data distribution range; , is a hyperparameter used to control the weight of the loss function. and Conduct training, The data in is pushed away from the low-density region by the dual point, while maintaining strong intra-class consistency under the influence of the attractor; at the same time, The data in is constrained to remain in low-density regions.

[0033] Specifically, obtain Each data point in The eigenvector of ; calculate With each pair and The distance between them is calculated according to the following formula Classified as kind: (11) In the formula, Indicates the category corresponding to the maximum value of the C results in brackets.

[0034] The attractor should be located at the center of a type of sample points as much as possible, and the difference between the reciprocal point and the sample points of this type should be as large as possible. Therefore, the position and angle direction of the reciprocal point and the attractor in the feature space should be relative, such as Figure 6 As shown. Calculated according to the following formula : (12) In the formula, is the number of batches currently being trained; is the second label encoding; yes belong Probability of class; is a hyperparameter used to control the distance-to-probability conversion; is a hyperparameter used to control the denominator from being zero.

[0035] In some embodiments of the present invention, the following formula is used to calculate : (13) Calculate according to the following formula , : (14) In the formula, is a hyperparameter used to control the weight of Euclidean distance and dot product.

[0036] Calculate according to the following formula : (15) In the formula, is the boundary value.

[0037] In order to keep unknown samples in the low-level area and push known samples away, we use The second norm of the sample in is constrained to be close to 0, and it is calculated according to the following formula : (16) In the formula, Represents the calculation of the bi-norm.

[0038] After dimensionality reduction visualization, most of the open set samples are distributed at the tail of the closed set samples. The generated samples are constrained by kernel density estimation so that the forged samples are located at the tail of the closed set samples as much as possible, so as to use these forged samples to approximate the open set samples and complete data enhancement. In addition, the present invention proposes to use attractor-reciprocal point learning to maintain intra-class aggregation while taking into account the risk of open space. The experiment selected UrbanSound8K, AudioEventDataset and TUT AcousticScenes 2017 as closed set datasets, and selected categories that do not overlap with these datasets from ESC-50 as open set samples. Finally, the results were compared with open set recognition algorithms in other fields, indicating that the method proposed in the present invention performs well in the open set recognition task of environmental audio. Specifically, for the UrbanSound8K dataset, AUROC and OSCR are 0.9251 and 0.8743 respectively; for the AudioEventDataset, AUROC and OSCR are 0.7921 and 0.7135 respectively; and for the TUT Acoustic Scenes 2017 dataset, AUROC and OSCR are 0.8209 and 0.6262 respectively. The openness of the above three datasets is over 90%, which shows that the proposed algorithm still shows strong performance in highly open scenes.

[0039] The environmental audio open set recognition model according to the embodiment of the present invention can recognize unknown categories and maintain good intra-class aggregation while considering open space risks.

[0040] The environmental audio open set recognition method according to the embodiment of the present invention can recognize unknown categories and maintain good intra-class aggregation while considering the risk of open space.

[0041] According to an embodiment of the present invention, the electronic device includes a processor and a memory, the processor and the memory are connected, the memory is used to store a computer program, and when the computer program is executed by the processor, the environmental audio open set recognition method as in the embodiment of the first aspect of the present invention is implemented.

[0042] The electronic device according to the embodiment of the present invention can identify unknown categories and maintain good intra-category aggregation while considering open space risks.

[0043] Other structures and operations of the electronic device according to the embodiment of the present invention are known to those skilled in the art and will not be described in detail here.

[0044] In the description of the present invention, it is to be understood that the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0045] In the description of the present invention, "a first feature" or "a second feature" may include one or more of the features. A first feature "above" or "below" a second feature may include direct contact between the first and second features, or may include contact between the first and second features through another feature between them instead of direct contact. A first feature "above", "above" or "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature.

[0046] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "specific embodiments", "examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0047] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for identifying an open set of ambient audio, characterized in that: include: Use the closed set classification algorithm to preliminarily classify the ambient audio and obtain the first output vector and the second output vector after dimensionality reduction ; The As a generated training set, and using the kernel density estimation to generate adversarial networks to generate the first edge samples , after dimensionality reduction, we get the second edge sample ; Using the loss function of the generative adversarial network , the density loss of the kernel density estimate and the With the The offset loss between To train: In the formula, , is a constant coefficient; Representing the training of the generative adversarial network; G , D Represent the generator and discriminator respectively; According to C classification labels, Divide into C distinct subsets , , for the Assign reciprocal points and maximize the With the The difference, for each of the Distribution Attractor To control the The sample distribution of The open set data is controlled in the low-level region, and the attractor-reciprocal point loss is calculated according to the following formula : In the formula, Expressing the Loss of restraint; Expressing the Loss of restraint; is the distance cross entropy loss; It is the loss of constraining the data distribution range; , It is a hyperparameter used to control the weight of the loss function.

2. The method for identifying an open set of ambient audio according to claim 1, characterized in that: Also includes: Get each of the The set of centers of ; Calculate the Middle Data points With center point The distance between , ; Using the The distribution function and kernel function fitting probability density function , calculated according to the following formula Belongs to the interval [ , ] probability: According to the following formula, the : In the formula, is the number of the second edge samples; Encode the first label; For the The data points in The distance between .

3. The method for identifying an open set of ambient audio according to claim 2, characterized in that: According to the following formula, the : In the formula, For the The sample mean of the class.

4. The method for identifying an open set of ambient audio according to claim 2, characterized in that: Also includes: Get the Each data point in The eigenvector of ; Calculate the With each pair of and stated The distance between them is calculated according to the following formula: Classified as kind: In the formula, Indicates the category corresponding to the maximum value of the C results in brackets.

5. The method for identifying an open set of ambient audio according to claim 4, characterized in that: According to the following formula, the : In the formula, is the number of batches currently being trained; is the second label encoding; is the belong Probability of class; is a hyperparameter used to control the distance-to-probability conversion; is a hyperparameter used to control the denominator from being zero.

6. The method for identifying an open set of ambient audio according to claim 5, characterized in that: According to the following formula, the , : In the formula, is a hyperparameter used to control the weight of Euclidean distance and dot product.

7. The method for identifying an open set of ambient audio according to claim 4, characterized in that: According to the following formula, the : In the formula, is the boundary value.

8. The method for identifying an open set of ambient audio according to claim 4, characterized in that: According to the following formula, the : In the formula, Represents the calculation of the bi-norm.

9. An open set recognition model for ambient audio, characterized in that: include: The first classifier includes a feature extraction layer, a convolution layer, a pre-logit layer, a first logit layer and a first kernel density estimation layer. The first classifier uses a closed set classification algorithm to preliminarily classify the ambient audio. The pre-logit layer is used to output a first output vector The first logit layer is used to Reduce dimension and output the second output vector , the first kernel density estimation layer is used to generate a probability density curve for each category; Generate an adversarial device, the generative adversarial device includes a generator, a discriminator, a second logit layer and a second kernel density estimation layer, the generator uses the kernel density estimation effect of the second kernel density estimation layer to generate a first edge sample The second logit layer is used to Reduce dimension and output the second edge sample To the second kernel density estimation layer, the discriminator is used to determine the Does it belong to the , the generative adversarial machine uses the loss function of the generative adversarial network , the density loss of the kernel density estimate and the With the The offset loss between Conduct training; The second classifier includes a plurality of convolutional layers, and the second classifier is used to classify the , Perform attractor-reciprocal point training, according to The classification labels will be described Divided into distinct subsets , , for the Assign reciprocal points and maximize the With the The difference, for each of the Distribution Attractor To control the The sample distribution of The open set data is controlled in the low-level area, according to the Constraint loss , Constraint loss , distance cross entropy loss And the loss of constraining the data distribution range Calculate attractor-reciprocal point loss .

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is connected to the memory, and the memory is used to store a computer program. When the computer program is executed by the processor, the method for identifying an open set of ambient audio according to any one of claims 1 to 8 is implemented.

Images