Improved prototype network training method, sound classification method, device and equipment

Abstract

The present application discloses a training method, a sound classification method, a device and equipment for an improved prototype network, wherein the improved prototype network is composed of a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, and the network model can be trained using a small amount of audio data. When performing sound classification, the above-mentioned trained improved prototype network is used to extract the prototype feature representation of the audio to be tested, and the distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category is calculated. The matching score of the audio to be tested belonging to the known sound category is calculated based on the above distance value, and finally, the audio to be tested is judged to be a known category or an unknown category based on the matching score. If it is a known category, its specific category is predicted based on the matching score. The present invention realizes the classification of open environment sound under small sample data, which can well solve the problems caused by the small number of collected sound samples and incomplete sound categories in real life.

Description

Technical Field

[0001] The invention belongs to the field of fault diagnosis, and in particular relates to an improved prototype network training method, a sound classification method, a device and equipment. Background Art

[0002] With the rapid development of artificial intelligence technology, computer hearing technology has been more widely used, such as intelligent robot applications, audio monitoring, automated navigation, environmental safety monitoring, smart home, autonomous driving, etc. The current environmental sound classification technology is often targeted at closed set sound classification. The classification model can only recognize the sound categories learned in the training phase. When encountering a sound category that has not been learned in the training phase, the model will judge it as the category closest to the sound category in the training phase. However, the real living environment is an open environment. The model will inevitably encounter sound categories that have not been learned in the training phase. There are many types of sounds, and it is difficult to collect training data for all sound categories. This results in the model judging the unlearned sound categories as known sound categories when predicting the sound category, making the sound classification model based on the closed data set unsuitable for practical applications.

[0003] In addition, training deep neural network models relies on a large amount of labeled data. When the training samples are insufficient, the model training will be overfitted, reducing the classification accuracy of the model. However, in real life, collecting a large amount of training data is time-consuming and laborious, and insufficient training samples have become a major obstacle to deep learning. In an open environment, there is still no effective solution for the key technology of using a small number of training samples to classify environmental sounds.

[0004] With the rapid development of the economy, the number of cars is increasing, and car safety has also received great attention. Regular inspection of cars can to a certain extent find the location and cause of the fault, but the power, safety and other performance of the car will change during driving, so it is extremely important to perform fault detection on the car during operation. At present, automobile fault diagnosis generally focuses on some key components for fixed-point detection, and the detection range is not comprehensive enough. Summary of the invention

[0005] Based on this, the present invention aims to provide a training method, a sound classification method, a device and an apparatus for improving a prototype network, and utilizes the improved prototype network to extract prototype features of the audio data to be tested, so as to solve the problem of sound classification in an open environment with small samples.

[0006] In a first aspect, the present invention provides a method for training an improved prototype network, wherein the network trained by the method is used to extract prototype feature representations of audio, and the method comprises:

[0007] Obtain the acoustic spectrogram of the audio sample set and initialize the true prototype feature representation;

[0008] Constructing an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, wherein the time-domain attention convolution module and the frequency-domain attention convolution module respectively extract the time-domain attention features and the frequency-domain attention features of the acoustic spectrum, the recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time-domain attention features and the frequency-domain attention features, and the fully connected layer calculates the training prototype feature representation according to the time-dependent features;

[0009] Input the acoustic spectrum into the improved prototype network to obtain the training prototype feature representation;

[0010] The prototype network loss is calculated based on the trained prototype feature representation and the true prototype feature representation. The prototype network loss is used to update the network parameters of the improved prototype network. When the prototype network loss is not greater than the preset loss threshold and the number of training iterations reaches the preset number, the training is completed and the improved prototype network for extracting the prototype feature representation of the audio is output.

[0011] Furthermore, the process of calculating the prototype network loss based on the training prototype feature representation and the real prototype feature representation includes:

[0012] According to the training prototype feature representation f(x;θ) and the real prototype feature representation m y Calculate prototype loss loss prototype =||f(x;θ)-m y ||;

[0013] Calculate the distance-based cross entropy discriminant loss loss based on the training prototype feature representation f(x; θ) and the true category y of the audio discriminative =-logp(y|x), p(y|x) represents the probability that the audio sample (x, y) belongs to category y,

[0014] Calculate the prototype network loss loss = loss discriminative +λloss prototype , λ represents the weight parameter of prototype loss.

[0015] Furthermore, the training method further includes updating the prototype feature representation of the known audio category, specifically including:

[0016] Get the prototype feature representation f(x) of all audio samples belonging to the same category i i ;θ);

[0017] Compute the prototype feature representation of category i S i represents the set of all audio samples belonging to category i.

[0018] Furthermore, the above training method also includes data enhancement of the acoustic spectrum, and the data enhancement process includes:

[0019] A data enhancement operation of one or more combinations of time domain warping, time domain masking, or frequency domain masking is performed on the acoustic spectrum.

[0020] Furthermore, the recurrent neural network layer is a Bi-GRU network.

[0021] In a second aspect, the present invention provides a sound classification method, comprising:

[0022] Collecting the acoustic spectrum of the audio to be tested;

[0023] Input the acoustic spectrum of the audio to be tested into the improved prototype network trained in the first aspect to obtain the prototype feature representation;

[0024] Calculate the Euclidean distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category;

[0025] Calculate the matching score between the audio to be tested and the known audio category according to the Euclidean distance;

[0026] Predict the audio category of the audio to be tested based on the matching score.

[0027] Furthermore, the calculation of the above matching score includes:

[0028] Based on the distance calculated above, the softmax function is used to calculate the probability value logist that the audio to be tested belongs to audio category i i (x), the probability value is recorded as the matching score that the audio to be tested x belongs to the known audio category i.

[0029] Furthermore, the audio category of the audio to be tested is predicted according to the matching score, including:

[0030] A maximum value is determined among the matching scores of the audio to be tested and the known audio categories. When the maximum value is greater than a preset score threshold, the audio to be tested belongs to the audio category corresponding to the maximum value; otherwise, the audio to be tested belongs to an unknown audio category.

[0031] In a third aspect, the present invention provides a network training device, comprising:

[0032] A sample acquisition unit, used to acquire the acoustic spectrogram of the audio sample set and initialize the true prototype feature representation;

[0033] A network construction unit, used to construct an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, wherein the time-domain attention convolution module and the frequency-domain attention convolution module respectively extract the time-domain attention features and the frequency-domain attention features of the acoustic spectrum, the recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time-domain attention features and the frequency-domain attention features, and the fully connected layer calculates the training prototype feature representation according to the time-dependent features;

[0034] A loss calculation unit, used to calculate the prototype network loss based on the training prototype feature representation and the real prototype feature representation;

[0035] The network updating unit is used to update the network parameters of the improved prototype network according to the prototype network loss. When the prototype network loss is not greater than a preset loss threshold and the number of training iterations reaches a preset number, the training is completed and an improved prototype network for extracting prototype feature representations of audio is output.

[0036] Furthermore, the process of calculating the prototype network loss by the loss calculation unit includes:

[0037] According to the training prototype feature representation f(x;θ) and the real prototype feature representation m y Calculate prototype loss loss prototype =||f(x;θ)-m y ||;

[0038] Calculate the training prototype feature representation f(x;θ) and the real prototype feature representation m y The Euclidean distance d(f(x;θ),m y );

[0039] Calculate the distance-based cross entropy discriminant loss loss based on the training prototype feature representation f(x; θ) and the true category y of the audio discriminative =-logp(y|x), p(y|x) represents the probability that the audio sample (x, y) belongs to category y,

[0040] Calculate the prototype network loss loss = loss discriminative +λloss prototype , λ represents the weight parameter of feature loss.

[0041] Furthermore, the network updating unit also performs a process of updating the feature representation of the known audio category prototype, specifically performing the following process:

[0042] Get the prototype feature representation f(x) of all audio samples belonging to the same category i i ;θ);

[0043] Compute the prototype feature representation of category i S i represents the set of all audio samples belonging to category i.

[0044] Furthermore, the above-mentioned network training device also includes a data enhancement unit for performing data enhancement operations of one or more combinations of time domain distortion, time domain masking or frequency domain masking on the acoustic spectrum.

[0045] In a fourth aspect, the present invention provides a sound classification device, comprising:

[0046] A data acquisition unit, used for acquiring an acoustic spectrum of the audio to be tested;

[0047] A prototype feature extraction unit, embedded with an improved prototype network trained by the network training device of the third aspect, for extracting prototype feature representation of the audio to be tested;

[0048] A feature distance calculation unit, used to calculate the Euclidean distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category;

[0049] The matching unit is used to calculate the matching score between the audio to be tested and the known audio category according to the Euclidean distance, and determine the audio category of the audio to be tested according to the matching score.

[0050] In a fifth aspect, the present invention provides an electronic device, comprising: a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is used to call the program stored in the memory to execute the improved prototype network training method provided by the above-mentioned first aspect embodiment and / or any possible implementation method combined with the first aspect embodiment, or to execute the sound classification method provided by the above-mentioned second aspect embodiment.

[0053] In a sixth aspect, the present invention further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the improved prototype network training method provided in the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment, or executes the sound classification method provided in the above-mentioned second aspect embodiment.

[0054] It can be seen from the above technical solutions that compared with the prior art, the beneficial effects of the present invention are:

[0055] The present invention proposes a training method of an improved prototype network and a method for sound classification using the improved prototype network. The improved prototype network consists of a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, wherein the time-domain attention convolution module and the frequency-domain attention convolution module focus on the local time-frequency features of the audio, and the recurrent neural network layer can better identify the temporal dependency of the audio. The network model can be trained using a small amount of audio data. When performing sound classification, the prototype feature representation of the audio to be tested is extracted using the trained network, the Euclidean distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category is calculated, and the matching score of the audio to be tested belonging to the known sound category is calculated according to the Euclidean distance. Finally, the audio to be tested is judged to be a known category or an unknown category based on the matching score. If it is a known category, its specific category is predicted according to the matching score. By judging the matching score between the prototype feature representation of the sound to be tested and the prototype of the known sound category, the sound category to which the audio to be tested belongs can be effectively judged. The present invention realizes the open environment sound classification under small sample data, which can well solve the problems caused by the small number of collected sound samples and incomplete sound categories in real life. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 An improved prototype network structure diagram provided for an embodiment of the present invention;

[0058] Figure 2 A flowchart of a training method for an improved prototype network provided by an embodiment of the present invention;

[0059] Figure 3 A flow chart of a sound classification method provided by an embodiment of the present invention;

[0060] Figure 4 is a structural diagram of a network training device provided by an embodiment of the present invention;

[0061] Figure 5 is a structural diagram of a sound classification device provided by an embodiment of the present invention;

[0062] Figure 6 It is a diagram of the electronic device architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The embodiments provided below of the present invention will illustrate a training process of an improved prototype network, using the trained network to extract the prototype feature representation of audio, which is used to determine the category to which audio of unknown category belongs, and solve the problem of audio classification of unknown category in an open environment in real life.

[0065] Figure 1 The structure diagram of the improved prototype network provided by an embodiment of the present invention is shown, which includes a time domain attention convolution module, a frequency domain attention convolution module, a recurrent neural network layer and a fully connected layer. When constructing these modules, batch normalization, dropout, activation function (such as sigmoid, softmax), etc. can be selected in a targeted manner.

[0066] In a specific embodiment, the temporal attention convolution module is composed of 2 layers of CNN and 2 layers of maximum pooling layers, wherein the input passes through a CNN with a convolution kernel size of 3×3 and a step size of 1×1, and then passes through a maximum pooling layer with a kernel size of F×1 to reduce the frequency dimension to 1, and then passes through a scaling function to obtain a temporal attention feature based on the frame level. The scaling function here can select a sigmoid function. This process can be expressed as:

[0067] A T =σ(AvgPool F×1 (Conv 3×3 (·)))

[0068] Among them, Conv 3×3 Represents the convolution operation of the CNN layer, AvgPool F×1 Represents the pooling operation of the maximum pooling layer.

[0069] The frequency domain attention convolution module has a similar structure to the time domain attention convolution module, except that the convolution kernel of the maximum pooling layer is 1×T. The process of input passing through this module can be expressed as:

[0070] A F =σ(AvgPool 1×T (Conv 3×3 (·)))

[0071] Specifically, the recurrent neural network layer can select commonly used RNN, LSTM, GRU and their evolved versions. Figure 1 In the example case, Bi-GRU is selected as the recurrent neural network layer of the improved prototype network, with 256 units and dropout set to 0.5.

[0072] Figure 2 An implementation process of improving the prototype network training process is shown, which may include the following steps:

[0073] Step S10: Obtain the acoustic spectrogram and true prototype feature representation of the audio sample set.

[0074] Specifically, the audio samples used for training carry true category labels, which are used to compare with the output results to obtain feedback values ​​for training the network.

[0075] Since the basic application of the prototype network is small sample learning, an insufficient number of training samples will affect the performance of the network model. In a further embodiment, in order to increase the number of training samples, the acoustic spectrum of the audio sample set can be enhanced, and then the original acoustic spectrum and the enhanced data are input into the network model as the audio sample set.

[0076] In terms of examples, data enhancement may be time domain distortion, time domain masking, frequency domain masking, etc., or a combination of these operations.

[0077] Step S11. Construct an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer.

[0078] Specifically, the time domain attention convolution module and the frequency domain attention convolution module respectively extract the time domain attention features and the frequency domain attention features of the acoustic spectrum, the recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time domain attention features and the frequency domain attention features, and the fully connected layer calculates the training prototype feature representation according to the time-dependent features.

[0079] The audio samples are transformed into dimension M∈R F×T×C The acoustic spectrum of the image is then fed into the time domain attention convolution module and the frequency domain attention convolution module at the same time. The obtained time domain attention feature can be expressed as A T ∈R (1×T×1) , Similarly, the frequency domain attention feature can be expressed as A F ∈R (F×1×1) .

[0080] Before entering the recurrent neural network layer, the acoustic spectrogram, time domain attention features, and frequency domain attention features can be fused. The purpose of feature fusion is to merge the features extracted from the input into a feature that is more discriminative than the input feature. In many works, fusing features of different scales is an important means to improve segmentation performance. Low-level features have higher resolution and contain more position and detail information, but because they have passed fewer convolutions, they are less semantic and have more noise; high-level features have stronger semantic information, but their resolution is very low and their perception of details is poor.

[0081] In terms of examples, the embodiment of the present invention adopts a feature fusion method with class weight allocation, and the fused time-frequency attention feature can be expressed as M′=α·A T +β·M+γ·A F , α+β+γ=1.

[0082] Audio signals are highly dependent on the environment, especially time. The recurrent neural network layer can be used to better extract the time-dependent features in audio samples. Finally, the time-dependent features are passed through the fully connected layer FC to obtain the prototype feature representation.

[0083] Step S12: Input the acoustic spectrum into the improved prototype network to obtain the training prototype feature representation.

[0084] Step S13. Calculate the prototype network loss based on the trained prototype feature representation and the real prototype feature representation. The prototype network loss is used to update the network parameters of the improved prototype network. When the prototype network loss is not greater than a preset loss threshold and the number of training iterations reaches a preset number, the training is completed and an improved prototype network for extracting the prototype feature representation of the audio is output.

[0085] Specifically, the process of calculating the prototype network loss includes:

[0086] Step S131. Based on the training prototype feature representation f(x;θ) and the real prototype feature representation m y Calculate prototype loss loss prototype =||f(x;θ)-m y ||.

[0087] Step S132: Calculate the distance-based cross entropy discriminant loss loss based on the training prototype feature representation f(x; θ) and the true category y of the audio discriminative =-logp(y|x), p(y|x) represents the probability that the audio sample (x, y) belongs to category y,

[0088] Step S133. According to the prototype loss loss prototype And cross entropy discriminant loss loss discriminativeCalculate the prototype network loss loss = loss discriminative +λloss prototype , λ represents the weight parameter of feature loss.

[0089] When the prototype network loss is not greater than the preset loss threshold, it means that the improved prototype network does not need to be updated, and when the number of training iterations reaches the preset number, the trained improved prototype network and the prototype feature representation of the known audio category can be output.

[0090] When the prototype network loss is greater than the preset loss threshold, it indicates that the network parameters need to be updated, and the prototype feature representation of the known audio category needs to be updated, and the audio samples currently used for training are added to complete the prototype feature representation of the known audio category. According to the idea of ​​the prototype network, the prototype feature of a certain category is obtained by averaging all vectors of the same category. Therefore, in an embodiment of the present invention, updating the prototype feature representation of the known audio category may include the following steps:

[0091] Step S134: Obtain the prototype feature representation f(x i ;θ).

[0092] Step S135. Calculate the prototype feature representation of category i S i represents the set of all audio samples belonging to category i.

[0093] The audio samples used for training have real category labels. All audio samples belonging to a certain category can be determined based on the labels, and their prototype feature representation is obtained through the above step S12. Together with other existing audio samples belonging to the same category, a set is formed, and the prototype feature representation of the category is obtained through the calculation of the above steps S134 to S135. For example, in the tth training, there are a audio samples of the category, and an audio sample of category i is added in the iterative update in the t+1th training. When the prototype feature representation of category i is updated, this sample needs to be added to recalculate the mean of the prototype feature representation of a+1 samples.

[0094] The above network model training method can be used to extract the prototype features of the input audio in an open environment. Figure 3 A method for sound classification using the improved prototype network according to an embodiment of the present invention is described.

[0095] The sound classification method provided in this embodiment includes the following steps:

[0096] Step S20: Collect the acoustic spectrum of the audio to be tested.

[0097] Specifically, the audio to be tested can be collected through a common recording device, and the audio is pre-processed to calculate its acoustic spectrum. In terms of examples, the acoustic spectrum can be a spectrogram, a Mel spectrogram, a gammatone spectrogram, etc.

[0098] Since the collected audio data often contains background noise and silent segments, it is first preprocessed to obtain clean audio segments with sound. The preprocessing may include: filtering out the ambient background noise of the audio to be tested and detecting and cutting the audio segments.

[0099] Step S21. Input the acoustic spectrum of the audio to be tested into the improved prototype network trained in the first aspect to obtain the prototype feature representation.

[0100] Step S22: Calculate the distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category.

[0101] Specifically, calculating the distance between two feature representations actually reflects the similarity measurement between the audio to be tested and the known audio category, and the measurement methods such as cosine distance, Euclidean distance, Manhattan distance, Chebyshev distance, Pearson coefficient, etc. can be used.

[0102] In terms of examples, the embodiment of the present invention provides an example using Euclidean distance, and the distance calculation between the two can be expressed as f(x; θ) represents the prototype feature representation of the audio to be tested, and represents the prototype feature representation of the known audio category i.

[0103] Step S23: Calculate the matching score between the audio to be tested and the known audio category according to the distance calculated above.

[0104] Specifically, by calculating the probability that the audio to be tested belongs to a certain audio category i as its matching score with the audio category, the probability can be predicted using a logistic regression algorithm. In the example, the softmax function is used to calculate the probability logist i (x).

[0105] Step S24: Predict the audio category of the audio to be tested according to the matching score.

[0106] Specifically, step S24 includes the following steps:

[0107] Step S241. Determine the maximum value maxlogist according to the above calculated probability i (x).

[0108] Step S242. Determine the maximum probability maxlogist i (x) Whether it is greater than a preset score threshold T, if so, proceed to S243, otherwise proceed to S244.

[0109] Step S243. maxlogisti (x)<T considers that the audio to be tested belongs to an unknown sound category, and directly outputs the judgment result, for example, by displaying words such as "unknown audio" through a visual interface, or broadcasting the judgment result, or issuing an alarm signal. The specific method is not further limited by the present invention.

[0110] Step S244. maxlogist i (x) ≥ T is considered that the audio to be tested belongs to a known sound category, and the audio to be tested belongs to the maximum probability maxlogist i (x) corresponds to the audio category, for example, the maximum value corresponds to the bird's cry, that is, the probability calculation believes that the probability that the audio to be tested belongs to the bird's cry is the greatest, and the judgment result that the audio to be tested belongs to the bird's cry is output.

[0111] The following is a detailed introduction to the above sound classification method using a practical application scenario for automobile status detection.

[0112] In-car noise, in-car sound quality, car safety, etc. are all related to various components, but data collection is very difficult and it is difficult to collect all noise sources. If a closed set model is used, if the abnormal sound emitted by a certain component of the car does not belong to the category of the closed set, the model will still judge it as the most similar category in the closed set, which will greatly affect the safety monitoring of the car. The embodiment of the present invention uses the open environment sound method for car status detection, which can effectively solve the problem of sound classification in an open environment, reduce the probability of classifying unknown sounds into known sound categories, and to a certain extent solve the problem of incorrect judgment of abnormal sounds in cars.

[0113] First, the corresponding digital twin model is installed on the car. After the car is started, the audio to be tested is collected through the sensor equipment, and the collected audio is preprocessed to calculate the acoustic spectrum features of the audio to be tested, and input into the preset improved prototype network to calculate the prototype feature representation of the audio to be tested; then the distance between the prototype feature representation and the prototype feature of the known sound category is calculated, and the distance is used to determine whether the audio to be tested is a known sound category or an unknown sound category. If it is a known sound category, its specific sound category is determined, and the position and status of the component are determined according to the sound category. If it is an unknown sound category, it is marked as an unknown sound category, and the status of the component is updated and displayed in real time in the digital twin model. In actual applications or tests, the audio of the unknown sound category can be manually judged in the later stage to determine which part of the car it belongs to and whether it is an abnormal sound, or it can be used as sample data of the known sound category to train the improved prototype network during network training. When training the improved prototype network, the training samples can be collected by sensors installed at different positions of the car, and labeled by engineers and other professionals, such as normal sound of component A, abnormal sound of component A, etc.

[0114] A network training method is described in detail in the above disclosed embodiment. The above disclosed training method can be implemented by using various forms of equipment. Therefore, the present invention also discloses a training device corresponding to the above method. A specific embodiment is given below for detailed description.

[0115] A network training device 100 provided by an embodiment of the present invention is as follows Figure 4 As shown, including:

[0116] The sample acquisition unit 110 is used to acquire the acoustic spectrogram of the audio sample set and initialize the true prototype feature representation.

[0117] The network construction unit 120 is used to construct an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer. The time-domain attention convolution module and the frequency-domain attention convolution module respectively extract the time-domain attention features and the frequency-domain attention features of the acoustic spectrum. The recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time-domain attention features and the frequency-domain attention features. The fully connected layer calculates the training prototype feature representation according to the time-dependent features.

[0118] The loss calculation unit 130 is used to calculate the prototype network loss according to the training prototype feature representation and the real prototype feature representation.

[0119] The network updating unit 140 is used to update the network parameters of the improved prototype network according to the prototype network loss. When the prototype network loss is not greater than a preset loss threshold and the number of training iterations reaches a preset number, the training is completed and an improved prototype network for extracting prototype feature representations of audio is output.

[0120] In an optional embodiment, the process of calculating the prototype network loss by the loss calculation unit 130 and the process of updating the network by the network updating unit 140 according to the prototype network loss can refer to the contents recorded in the aforementioned embodiment.

[0121] The above disclosed embodiment also describes in detail a sound classification method. The above disclosed sound classification method can be implemented using various forms of equipment. Therefore, the present invention also discloses a sound classification device corresponding to the above method. A specific embodiment is given below for detailed description.

[0122] An embodiment of the present invention provides a sound classification device 200 as follows Figure 5 As shown, including:

[0123] The data collection unit 210 is used to collect the acoustic spectrum of the audio to be tested.

[0124] The prototype feature extraction unit 220 is embedded with an improved prototype network trained by the network training device of the third aspect, and is used to extract the prototype feature representation of the audio to be tested.

[0125] The feature distance calculation unit 230 is used to calculate the distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category.

[0126] The matching unit 240 is used to calculate a matching score between the audio to be tested and the known audio category according to the calculated distance, and predict the audio category of the audio to be tested according to the matching score.

[0127] The sound classification device 200 provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0128] See also Figure 6 , which shows a hardware structure block diagram of an electronic device, including: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0129] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0130] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0131] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0132] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to: implement each processing flow of the aforementioned improved prototype network training method, or implement each processing flow of the aforementioned sound classification method.

[0133] An embodiment of the present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the training method of the improved prototype network provided in the above embodiment and / or any possible implementation method in combination with the embodiment, or executes the sound classification method provided in the above embodiment.

[0134] It should be understood that although Figure 2 as well as Figure 3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 as well as Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an improved prototype network, characterized in that: The method comprises: Obtain acoustic spectrograms and true prototype feature representations of audio sample sets; Constructing an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, wherein the time-domain attention convolution module and the frequency-domain attention convolution module respectively extract the time-domain attention features and the frequency-domain attention features of the acoustic spectrum, the recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time-domain attention features and the frequency-domain attention features, and the fully connected layer calculates the training prototype feature representation according to the time-dependent features; Inputting the acoustic spectrum into the improved prototype network to obtain a training prototype feature representation; The prototype network loss is calculated based on the training prototype feature representation and the real prototype feature representation, and the prototype network loss is used to update the network parameters of the improved prototype network. When the prototype network loss is not greater than a preset loss threshold and the number of training iterations reaches a preset number, the training is completed and an improved prototype network for extracting the prototype feature representation of the audio is output.

2. The training method according to claim 1, characterized in that: The calculation process of the prototype network loss includes: According to the training prototype feature representation f(x;θ) and the real prototype feature representation m y Calculate prototype loss loss prototype =||f(x;θ)-m y ||; Calculate the training prototype feature representation f(x;θ) and the real prototype feature representation m y The Euclidean distance d(f(x;θ),m y ); Calculate the distance-based cross entropy discriminant loss loss based on the training prototype feature representation f(x; θ) and the true category y of the audio discriminative =-log p(y|x), p(y|x) represents the probability that the audio sample (x, y) belongs to category y, Calculate the prototype network loss loss = loss discriminative +λloss prototype , λ represents the weight parameter of the prototype loss.

3. The training method according to claim 1, characterized in that: The method further includes updating the prototype feature representation of the known audio category, specifically comprising: Get the prototype feature representation f(x) of all audio samples belonging to the same category i i ;θ); Compute the prototype feature representation of category i S i represents the set of all audio samples belonging to category i.

4. A sound classification method for an open environment, characterized in that: include: Collecting the acoustic spectrum of the audio to be tested; Inputting the acoustic spectrum of the audio to be tested into the improved prototype network trained by the training method according to any one of claims 1 to 3 to obtain the prototype feature representation of the audio to be tested; Calculating the Euclidean distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category; Calculating a matching score between the audio to be tested and the known audio category according to the Euclidean distance; The audio category of the audio to be tested is determined according to the matching score.

5. The sound classification method according to claim 4, characterized in that: The calculation of the matching score includes: Based on the Euclidean distance, the probability value logist of the audio to be tested belonging to the audio category i is calculated using the softmax function. i (x), the probability value is recorded as the matching score that the audio to be tested x belongs to the known audio category i.

6. The sound classification method according to claim 4, characterized in that: Determining the audio category of the audio to be tested according to the matching score includes: A maximum value is determined among the matching scores of the audio to be tested and the known audio categories. When the maximum value is greater than a preset score threshold, it is determined that the audio to be tested belongs to the audio category corresponding to the maximum value; otherwise, the audio to be tested belongs to an unknown audio category.

7. A vehicle status detection method based on sound classification, characterized in that: include: Collect the working audio during the operation of the car and calculate the sound spectrum of the working audio; Inputting the acoustic spectrum into the improved prototype network trained by the training method according to any one of claims 1 to 3 to obtain a prototype feature representation of the working audio; Calculating the Euclidean distance between the prototype feature representation and the prototype feature representation of the known audio category; Calculating a matching score between the working audio and the known audio category according to the Euclidean distance; The audio category of the working audio is determined according to the matching score, and the component and its status are determined according to the audio category.

8. A training device for an improved prototype network, characterized in that: include: A sample acquisition unit, used to obtain an acoustic spectrum and a true prototype feature representation of an audio sample set; A network construction unit, used to construct an improved prototype network including a time-domain attention convolution module, a frequency-domain attention convolution module, a recurrent neural network layer and a fully connected layer, wherein the time-domain attention convolution module and the frequency-domain attention convolution module respectively extract the time-domain attention features and the frequency-domain attention features of the acoustic spectrum, the recurrent neural network layer calculates the time-dependent features according to the acoustic spectrum, the time-domain attention features and the frequency-domain attention features, and the fully connected layer calculates the training prototype feature representation according to the time-dependent features; A loss calculation unit, used to calculate the prototype network loss according to the training prototype feature representation and the real prototype feature representation; A network updating unit is used to update the network parameters of the improved prototype network according to the prototype network loss. When the prototype network loss is not greater than a preset loss threshold and the number of training iterations reaches a preset number, the training is completed and the improved prototype network for extracting the prototype feature representation of the audio is output.

9. A sound classification device, characterized in that: include: A data acquisition unit, used for acquiring an acoustic spectrum of the audio to be tested; A prototype feature extraction unit, embedded with an improved prototype network trained by the training device according to claim 8, for extracting the prototype feature representation of the audio to be tested; A feature distance calculation unit, used to calculate the Euclidean distance between the prototype feature representation of the audio to be tested and the prototype feature representation of the known audio category; A matching unit is used to calculate a matching score between the audio to be tested and a known audio category according to the distance, and determine the audio category of the audio to be tested according to the matching score.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to call the program stored in the memory to execute the training method according to any one of claims 1 to 3, or to execute the sound classification method according to any one of claims 4 to 6.

Images