A training method, recognition method, device and medium for a sound recognition model

Abstract

The present invention discloses a training method, a recognition method, a device and a medium for a sound recognition model. The present invention calculates a first energy value of a sound data set, performs a first screening process according to the first energy value calculation result, and obtains first data; performs gain processing, filter frequency domain enhancement processing and random order merging processing on the first data, and combines at least one of the gain processing result, the filter frequency domain enhancement processing result and the random order merging processing result, which is beneficial to increasing the robustness of the sound recognition model training process to data that may carry various noises; performs Mel spectrum calculation on the second data to obtain a training sample, inputs the training sample into a teacher network training to obtain a teacher network model, inputs the training sample into a student network and performs distillation learning on the student network according to the teacher network model to obtain a sound recognition model, which is lightweight and has improved recognition ability. The present invention can be widely used in the field of data processing technology.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a training method, a recognition method, a device and a medium for a sound recognition model. Background Art

[0002] There are various kinds of sounds in nature, such as biological sounds (biological sounds produced by animals such as birds and insects), natural sounds (such as wind and rain), and human sounds (generally referring to the sounds produced by human activities). In recent years, the use of sound information to study the ecological environment has received increasing attention. For example, the temporal and spatial variation characteristics of forest sound information are used to reveal the laws of bird activity, and animal sounds are used to monitor species and population density. For different applications, it is often necessary to identify the sounds of interest from a large number of recording files.

[0003] At present, there are still some challenges in the task of environmental sound recognition. Unlike speech and music signals, environmental sound is more complex in spectrum and time domain structure, which makes it more difficult to design the corresponding recognition system. For speech and music signals, since both contain a lot of structured information, such as phonemes, statistical models can usually be used for modeling, such as hidden Markov models. However, for environmental sound signals, the complexity of their sources causes their spectrum structure to vary greatly. At the same time, due to the differences in acquisition equipment and environment, the existing sound data sets contain a lot of noise and silence segments, which will also affect the training of the model. In addition, since the workload of manually annotating sound data sets is large, and the scale of existing sound data sets is generally small, how to implement a recognition model with strong generalization ability on a small data set is also a challenge. Finally, most of the current work based on environmental sound research is still in the theoretical stage without actual application, which also limits the promotion of research related to environmental sound recognition. Summary of the invention

[0004] In view of this, in order to solve the above technical problems, the purpose of the present invention is to provide a training method, recognition method, device and medium for a sound recognition model.

[0005] The technical solution adopted in the embodiment of the present invention is:

[0006] A method for training a sound recognition model, comprising:

[0007] Get the sound dataset;

[0008] Performing a first energy value calculation on the sound data set, and performing a first screening process according to a result of the first energy value calculation to obtain first data;

[0009] Performing gain processing, filtering frequency domain enhancement processing and random order merging processing on the first data, and combining at least one of the gain processing results, the filtering frequency domain enhancement processing results and the random order merging processing results to obtain second data; the filtering frequency domain enhancement processing includes directivity attenuation enhancement processing and data frequency domain random enhancement processing

[0010] Performing Mel-spectrogram calculation on the second data to obtain training samples;

[0011] The training samples are input into the teacher network for training to obtain a teacher network model, the training samples are input into the student network and the student network is distilled and learned according to the teacher network model to obtain a sound recognition model.

[0012] Further, the calculating a first energy value of the sound data set includes:

[0013] Performing frame division, frame stacking and windowing operations on the sound data set to obtain a plurality of frames of first signals;

[0014] Performing a first discrete Fourier transform on the first signal to determine a first amplitude spectrum of the first signal for each frame;

[0015] The sum of the energy values ​​of each of the first amplitude spectra is calculated according to the preset number of frequency points to obtain a first energy value calculation result.

[0016] Further, performing a first screening process according to the first energy value calculation result to obtain first data includes:

[0017] Taking a preset number of frames of continuous first signals as an energy block, and calculating a block energy value corresponding to each of the energy blocks according to the first energy value calculation result;

[0018] Determine a target frame number from the energy block with the largest block energy value;

[0019] Performing time domain conversion on the target frame sequence number to obtain a time domain index, and determining a sampling point sequence number according to the time domain index and a preset sampling range;

[0020] The energy block with the largest block energy value is cut and screened according to the sampling point sequence number and the preset data length to obtain the first data.

[0021] Further, performing gain processing and filtering frequency domain enhancement processing on the first data includes:

[0022] Slicing the first data to obtain a plurality of slice data, selecting a first value from a first preset gain range, determining a second value according to the first value and a preset offset, performing a first multiplication on the second value and the slice data respectively, and resynthesizing the first multiplication results to obtain a gain processing result;

[0023] The directivity attenuation enhancement process includes: determining a cutoff frequency from a preset frequency range, filtering the first data by the cutoff frequency, and obtaining first enhanced data;

[0024] The data frequency domain random enhancement processing is: performing a first frame windowing processing on the first data, and performing a second discrete Fourier transform on the first frame windowing processing result to obtain a second amplitude spectrum and a phase spectrum, performing frequency band segmentation on the second amplitude spectrum to obtain a plurality of frequency bands, determining a gain value according to a second preset gain range, performing a second multiplication on the gain value and the frequency band respectively, splicing the second multiplication result, performing an inverse Fourier transform conversion according to the phase spectrum and the spliced ​​second multiplication result to obtain second enhanced data; the filtered frequency domain enhancement processing result includes the first enhanced data and the second enhanced data.

[0025] Further, performing random merging processing on the first data, and combining at least one of a gain processing result, a filtering frequency domain enhancement processing result, and a random merging processing result to obtain second data, includes:

[0026] Calculating the number of sound events according to the preset number of sampling points and the total number of sampling points of the first data;

[0027] Performing a second energy value calculation on the first data, and performing a second screening process a number of times according to the second energy value calculation result to obtain the number of sound events, that is, sound events;

[0028] Randomly sorting and splicing the sound events to obtain a random merging processing result;

[0029] At least one of the gain processing result, the filtering frequency domain enhancement processing result and the out-of-order merging processing result is randomly combined to obtain second data.

[0030] Further, performing Mel spectrum calculation on the second data to obtain training samples includes:

[0031] Performing a second frame windowing process on the second data, performing a third discrete Fourier transform on the result of the second frame windowing process, and calculating a power spectrum according to the result of the third discrete Fourier transform;

[0032] Determine a Mel filter function according to a preset frequency upper limit and a preset frequency lower limit;

[0033] A mel spectrum is obtained according to the mel filter function and the power spectrum; and the training sample includes the mel spectrum.

[0034] Furthermore, the training samples have real labels; the step of inputting the training samples into a student network and performing distillation learning on the student network according to the teacher network model to obtain a sound recognition model includes:

[0035] Inputting the training sample into the teacher network model to obtain a predicted label;

[0036] Inputting the training sample into the student network to obtain a first prediction result at a first temperature and a second prediction result at a second temperature;

[0037] Calculating a distillation loss according to the first prediction result and the prediction label, calculating a student loss according to the second prediction result and the true label, calculating a first product of the distillation loss and a first weight and calculating a second product of the student loss and a second weight, and determining a loss value according to the sum of the first product and the second product;

[0038] Return to the step of inputting the training samples into the teacher network model until the number of training times or the loss value meets the preset conditions, thereby obtaining a sound recognition model.

[0039] The embodiment of the present invention also provides an identification method, including:

[0040] Get the data to be identified.

[0041] The data to be recognized is input into a sound recognition model to obtain a recognition result; the sound recognition model is trained by the training method of the sound recognition model.

[0042] An embodiment of the present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method.

[0043] An embodiment of the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method.

[0044] The beneficial effects of the present invention are as follows: by performing a first energy value calculation on a sound data set, performing a first screening process according to the first energy value calculation result, obtaining the first data, it is beneficial to extract effective components from the weakly labeled sound data set as much as possible for subsequent training; performing gain processing, filtering frequency domain enhancement processing and random order merging processing on the first data, and combining at least one of the gain processing results, filtering frequency domain enhancement processing results and random order merging processing results, since the second data is obtained by combining after gain processing, filtering frequency domain enhancement processing and random order merging processing, it is beneficial to increase the robustness of the data that may carry various noises during the training process of the sound recognition model; performing Mel spectrum calculation on the second data to obtain training samples, inputting the training samples into the teacher network training to obtain the teacher network model, inputting the training samples into the student network and performing distillation learning on the student network according to the teacher network model to obtain the sound recognition model, and using the distillation technology to make the sound recognition model lightweight and improve the recognition ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the steps of the training method of the sound recognition model of the present invention;

[0046] Figure 2 A schematic diagram of teacher network training and distillation learning in a specific embodiment of the present invention;

[0047] Figure 3 It is a structural diagram of a student network in a specific embodiment of the present invention;

[0048] Figure 4 It is a structural schematic diagram of a feature extraction block in a specific embodiment of the present invention;

[0049] Figure 5 It is a structural schematic diagram of a convolution block in a specific embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the structure of the spatial attention block in a specific embodiment of the present invention;

[0051] Figure 7 Schematic diagram of the steps of the identification method of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0053] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] like Figure 1 As shown, an embodiment of the present invention provides a method for training a sound recognition model, comprising steps S100-S500:

[0056] S100: Acquire a sound dataset.

[0057] Optionally, the sound dataset may include an audio file containing at least one of the categories of biological sounds, human sounds, and natural sounds. Table 1 shows the audio files used in the embodiments of the present invention, including bird, insect, bird-insect (mixed bird and insect sounds), human, human-insect (mixed human and insect sounds), silence, and bird-human (mixed human and bird sounds). The audio file duration is greater than or equal to 10 minutes. Other embodiments may include audio files of other categories without specific limitation.

[0058] Sampling rate Format Quantization bits Number of channels bird 22050Hz wav 16 1 insect 22050Hz wav 16 1 bird-insect 22050Hz wav 16 1 human 22050Hz wav 16 1 human-insect 22050Hz wav 16 1 silence 22050Hz wav 16 1 bird-human 22050Hz wav 16 1

[0059] S200: Calculate a first energy value for a sound data set, and perform a first screening process according to a result of the first energy value calculation to obtain first data.

[0060] In the embodiment of the present invention, step S200 can effectively extract environmental sound samples containing sound events from the collected audio files, thereby achieving the effect of screening the data required for model training.

[0061] Optionally, performing a first energy value calculation on the sound data set in step S200 includes steps S211-S213:

[0062] S211 , performing frame division, frame stacking and windowing operations on the sound data set to obtain a plurality of frames of first signals.

[0063] Optionally, record the number of data sampling points N of an audio file, set the frame length wlen (including but not limited to 1024), set the frame shift inc (including but not limited to 256), and obtain the number of frames nf of the audio file by formula (1). When the frame overlap is 0 (i.e., when inc = wlen), each frame after uniform framing is x in (n,λ), where n is the sampling point number and λ is the frame number.

[0064]

[0065] Set the current frame x in (n,λ) and the previous frame x in (n,λ-1) performs frame overlap (overlap frame) with a length of overlap=wlen-inc to obtain x on (n,λ), the total length of the frame after stacking is still wlen, n=0,1,…,wlen-1.

[0066]

[0067] x on (n,λ) is windowed, and the window type is a Hamming window w(n,α), where α (including but not limited to 0.46) and the window length is equal to the frame length wlen. Thus, all windowed frames x are obtained. w (n,λ), that is, the first signal of several frames.

[0068]

[0069] x w (n,λ)=x on (n,λ)*w(n,α)0≤n≤wlen-1 (4)

[0070] S212 . Perform a first discrete Fourier transform on the first signal to determine a first amplitude spectrum of the first signal of each frame.

[0071] For each frame, the first signal x w (n,λ) performs a first discrete Fourier transform (first DFT operation) of N points (including but not limited to, for example, 1024), and takes the modulus of the first DFT operation result to obtain the first amplitude spectrum X(λ,k) of the first signal of each frame, where k represents the frequency point and λ is the frame number. Optionally, due to the symmetry of the Fourier transform, only the first N points of the spectrum can be processed. f frequency points for analysis, where N f=N / 2+1, j is an imaginary number:

[0072]

[0073] S213 . Calculate the sum of energy values ​​of each first amplitude spectrum according to the preset number of frequency points to obtain a first energy value calculation result.

[0074] Optionally, the frequency point amplitude values ​​of the target frequency band in each frame on the first amplitude spectrum X(k,λ) are summed to obtain the energy sum En(λ) of each frame signal (the sum of the energy values ​​of each first amplitude spectrum, that is, the first energy value calculation result), where f t is the background noise frequency (including but not limited to 200Hz), N t is the frequency point number corresponding to the background noise frequency (preset frequency point number), f s is the audio sampling rate (including but not limited to 16000Hz), Indicates rounding down. For example, when the result is 3.5, it is rounded down to 3.

[0075]

[0076]

[0077] Optionally, in step S200, performing a first screening process according to the first energy value calculation result to obtain first data includes steps S221-S224:

[0078] S221 . Take a preset number of frames of continuous first signals as an energy block, and calculate a block energy value corresponding to each energy block according to a first energy value calculation result.

[0079] S222. Determine a target frame number from the energy block with the largest block energy value.

[0080] Optionally, a preset number of frames (block_frames) of continuous first signals are taken as an energy block, the energy value of each energy block is calculated, and the target frame number is determined from the energy block with the largest block energy value. For example, the target frame number includes but is not limited to the center frame number λ of the center frame of the largest energy block. t (Calculated by formula (11)). Where frame_nums is the total number of frames of the first signal, blocks is the number of energy blocks after segmentation, En(λ b ) represents the energy of each energy block, λ b is the energy block subscript, λ max_block Indicates the subscript of the energy block with the highest energy.

[0081]

[0082]

[0083]

[0084] λ t =λ max_block *(block_frames-1)+(block_frames / 2) (11)

[0085] S223, performing time domain conversion on the target frame sequence number to obtain a time domain index, and determining a sampling point sequence number according to the time domain index and a preset sampling range.

[0086] Optionally, determine the target frame number λ t After that, the time domain conversion is performed through formula (12) to obtain the time domain index index of the target frame number in the time domain t .

[0087]

[0088] Assume that the preset random offset sampling point number is offset_nums (the point number setting is related to the sampling rate. For example, in the case of a 22050 Hz sampling rate, it can be set to offset_nums = 22050*0.5 = 11025), so the preset sampling range can be:

[0089] [index t -offset_nums,index t + offset_nums], randomly select a sampling point number from the preset sampling range to get the sampling point number index s .

[0090] For example, assuming that the time length of each frame is 2 seconds, and the target frame number is 3 according to formula (11), then the time domain index of the target frame number in the time domain represents the number of the sampling point at the 5th second, because the time range of the 3rd frame is 4 to 6 seconds, and the sampling point at the center is the sampling point position at 5 seconds. It should be noted that when there is no overlap between the frames, that is, overlap = 0, the calculation is performed according to the above example. When there is overlap between the frames when framing, the calculation is performed according to formula (12).

[0091] S224 , cutting and screening the energy block with the largest block energy value according to the sampling point sequence number and the preset data length to obtain first data.

[0092] Optionally, suppose the preset data length to be obtained is valid_len (including but not limited to 22050*4=88200), then the sampling point sequence number index s The point is the center and expands to the left and right respectively. Sampling points are obtained to obtain the starting position and the ending position. Cutting and screening are performed according to the starting position and the ending position to obtain the effective training sample, that is, the first data x va lid.

[0093] S300, performing gain processing, filtering frequency domain enhancement processing and random order merging processing on the first data, and combining at least one of the gain processing result, the filtering frequency domain enhancement processing result and the random order merging processing result to obtain second data. The filtering frequency domain enhancement processing includes directivity attenuation enhancement processing and data frequency domain random enhancement processing.

[0094] The embodiment of the present invention generates the second data through step S300, which can effectively improve the robustness of the sound recognition model in recognizing sound, so that the sound recognition model can still achieve efficient recognition in complex environmental sound scenes.

[0095] Optionally, performing gain processing and filtering frequency domain enhancement processing on the first data in step S300 includes steps S311-S313:

[0096] S311, slice the first data to obtain a plurality of slice data, select a first value from a first preset gain range, determine a second value according to the first value and a preset offset, perform a first multiplication on the second value and the slice data respectively, re-synthesize the first multiplication results, and obtain a gain processing result.

[0097] Optionally, assuming that the first data x valid The number of sampling points is N valid , with N clip (including but not limited to 22050) is the number of sampling points for each slice, for x valid Slice to get multiple slice data x clip (λ c ), where λ c is the slice number. For each slice data x clip (λ c ) performs random amplitude enhancement or attenuation, specifically: randomly select a first value (gain value rn_s) from a first preset gain range [rn_l, rn_h] (including but not limited to [-10, 10]), and then randomly select a second value in [rn_s-rn_o, rn_h+rn_o] as the final gain, and directly perform a first multiplication with the slice data to obtain the gain-processed data (the first multiplication result), and the first multiplication result after random gain processing is resynthesized into complete audio data to obtain the gain processing result. It should be noted that rn_l and rn_h are the lower limit and upper limit of the first-level random gain, respectively, and rn_o (including but not limited to 2) is the preset offset of the second-level gain.

[0098] S312, directivity attenuation enhancement processing is: determining a cutoff frequency from a preset frequency range, filtering the first data by the cutoff frequency, and obtaining first enhanced data.

[0099] Optionally, a cutoff frequency f is obtained by randomly selecting a value from a preset frequency range (including but not limited to [200, 1000]). cut , for the first data x valid Perform filtering processing, such as first-order or second-order Butterworth low-pass filtering, to obtain first enhanced data.

[0100] S313, data frequency domain random enhancement processing is: performing a first frame windowing processing on the first data, and performing a second discrete Fourier transform on the first frame windowing processing result to obtain a second amplitude spectrum and a phase spectrum, performing frequency band segmentation on the second amplitude spectrum to obtain a plurality of frequency bands, determining a gain value according to a second preset gain range, performing a second multiplication of the gain value and the frequency band respectively, splicing the second multiplication results, performing an inverse Fourier transform conversion according to the phase spectrum and the spliced ​​second multiplication results to obtain second enhanced data; the filtered frequency domain enhancement processing result includes the first enhanced data and the second enhanced data.

[0101] Optionally, a first frame windowing process is performed on the first data. The principle is similar to that described in step S200 and will not be repeated here. A second discrete Fourier transform is performed on the result of the first frame windowing process to obtain a second amplitude spectrum X a (k,λ) and phase spectrum θ(k,λ). Assuming the frequency domain segmentation bandwidth is band_width Hz, for the second amplitude spectrum X a (k,λ) is used for frequency band segmentation. Where fre_resolution is the frequency resolution, band_bins is the number of frequency points occupied by each frequency band after segmentation, bands is the number of frequency bands after segmentation, and X s (s,λ) represents each frequency band after the frequency band is divided, and s is the frequency band index.

[0102]

[0103]

[0104]

[0105] X s (s,λ)=X a (k,λ),i*band_bins≤k≤(i+1)*band_bins,0≤i <bands (16)

[0106] In the embodiment of the present invention, the gain value is determined according to the second preset gain range. For example, the method of step S311 is used to determine the bands second-level values ​​in a secondary random manner as the final gain, that is, the gain value, and then the gain value is respectively compared with the frequency band X. s (s,λ) is multiplied for the second time to obtain the frequency band amplitude spectrum after gain, that is, the second multiplication result, and the second multiplication result is re-spliced ​​according to the frequency dimension. The spliced ​​second multiplication result and the phase spectrum θ(k,λ) are converted back to the time domain signal through inverse Fourier transform to obtain the final enhanced signal, that is, the second enhanced data.

[0107] Optionally, in step S300, the first data is subjected to random merging processing, and at least one of the gain processing result, the filtering frequency domain enhancement processing result and the random merging processing result is combined to obtain the second data, including steps S321-S324:

[0108] S321. Calculate the number of sound events according to the preset number of sampling points and the total number of sampling points of the first data.

[0109] Optionally, assuming that the number of sampling points (the total number of sampling points) of the first data is N valid , let the event length (preset number of sampling points) be N event (including but not limited to 22050) sampling points, the first data x valid Divide into sound events, that is, the number of sound events is event__nums.

[0110] S322, performing a second energy value calculation on the first data, and performing a second screening process a number of times according to the second energy value calculation result to obtain a number of sound events, that is, sound events.

[0111] Optionally, the start and end positions of each event segmentation are determined by the same method as step S200, and the second energy value is calculated for the first data. Referring to the method for calculating the first energy value, the second screening process refers to the method of S221-S224. Each time the second screening process is performed, a sound event can be obtained. After several (event__nums) second screening processes, event__nums sound events event1, event2, ..., event can be obtained. event_nums .

[0112] S323, randomly sort and concatenate the sound events to obtain a random merging processing result.

[0113] Optionally, the order of all sound events is randomly shuffled to obtain a rearranged sound event list, which is finally reconnected to obtain enhanced audio data, i.e., the result of the shuffle merging process.

[0114] S324: randomly combine at least one of the gain processing result, the filtering frequency domain enhancement processing result, and the out-of-order merging processing result to obtain second data.

[0115] Optionally, at least one of the gain processing result, the filtering frequency domain enhancement processing result, and the out-of-order merging processing result is randomly combined in a randomized manner to obtain the second data. It is understandable that the second data may include one of the gain processing result, the filtering frequency domain enhancement processing result, and the out-of-order merging processing result, or may include two or three of the gain processing result, the filtering frequency domain enhancement processing result, and the out-of-order merging processing result.

[0116] S400: Perform mel spectrum calculation on the second data to obtain training samples.

[0117] Optionally, step S400 includes steps S411-S413:

[0118] S411, performing a second frame windowing process on the second data, performing a third discrete Fourier transform on the result of the second frame windowing process, and calculating a power spectrum according to the result of the third discrete Fourier transform.

[0119] Optionally, the second frame windowing process refers to the process of the first frame windowing process, which will not be described in detail. In the embodiment of the present invention, the second frame windowing process result is subjected to a third discrete Fourier transform, and the modulus value of the third discrete Fourier transform result is taken and squared to obtain the power spectrum X p 2 (k,λ). It should be noted that due to the symmetry of the Fourier transform, we can use only the first N f frequency points for analysis, where N f =N / 2+1, we can get k∈[1,N f ], the framing operation is performed according to formula (1) to obtain λ∈[1,nf].

[0120] S412: Determine a Mel filter function according to a preset upper frequency limit and a preset lower frequency limit.

[0121] S413: Obtain a Mel spectrum according to the Mel filter function and the power spectrum.

[0122] In the embodiment of the present invention, the training sample is a Mel spectrum. Optionally, assuming the actual frequency of a specific sound event in the target environment: for example, the preset frequency lower limit is f l , the preset frequency upper limit is f h (Unit: Hz), here we set f l =200Hz, f h=12000Hz, according to the conversion relationship between Mel frequency and actual frequency:

[0123]

[0124] Where, f is the actual frequency, f mel is the Mel frequency, and the upper and lower limits of the Mel frequency (in Mel) of the Mel filter bank to be calculated are F mel (f l ) and F mel (f h Specifically, a Mel filter bank is constructed to obtain the target frequency range [F mel (f l ),F mel (f h )], 1≤m≤M, where k represents the frequency point, m is the index of the filter, and M is the total number of filters to be set, where M (for example, 128, the larger the value of M, the better the frequency resolution of the obtained Mel spectrum). The expression of the H(k,m) filter (Mel filter function) is shown in formula (18), and each filter has a triangular filtering characteristic, and its center frequency is f(m), as shown in formula (19).

[0125]

[0126]

[0127] Among them, f l is the lowest frequency in the filter frequency range (in Hz), f h is the highest frequency of the filter frequency range, N1 is the length of FFT, f s is the sampling frequency, F mel In this embodiment of the present invention, the power spectrum X of the data slice is p 2 (k,λ) obtains the Mel spectrum of the slice data through the Mel filter function H(k,m) See formula (20).

[0128]

[0129] S500, input the training samples into the teacher network to obtain the teacher network model, input the training samples into the student network and perform distillation learning on the student network according to the teacher network model to obtain the sound recognition model.

[0130] In the embodiment of the present invention, the training sample has a real label, and the real label can be achieved through manual labeling, such as labeling in the training sample or pre-labeling in the sound data set, without specific limitation.

[0131] like Figure 2 As shown, in an embodiment of the present invention, the training sample (Traindata) is first input into the teacher network training to obtain the teacher network model. Optionally, the teacher network uses the ResNet18 backbone as the feature extractor of the teacher network, and the fully connected layer (FC) is set according to the number of environmental sound categories. Labels (specifically classified labels) can be output, and the number of small batch samples set for training is 32. The Adam optimizer with an initial learning rate of 2x10-4 is used for optimization. After training the model for several epochs including but not limited to 200 epochs, the model parameters with the highest verification accuracy are saved, and the parameters of the teacher network can be obtained. The teacher network model is determined according to the parameters of the teacher network.

[0132] Optionally, step S500 includes steps S511-S514:

[0133] S511. Input the training samples into the teacher network model to obtain the predicted labels.

[0134] Optionally, the student network is distilled through the teacher network model. During the distillation learning process, the training samples are input into the teacher network model, and soft labels (prediction labels) are generated at a high temperature T (for example, T = 10). The prediction label contains the softmax probability q of category i under the temperature coefficient T i , the calculation process is as follows:

[0135]

[0136] Among them, i represents any category, z i Represents the logit output of category i, that is, the output after the fully connected layer.

[0137] S512: Input the training sample into the student network to obtain a first prediction result at a first temperature and a second prediction result at a second temperature.

[0138] At the same time, during the distillation learning process, the training samples are input into the student network, which is a lightweight model Light_EnvSound_Model, so that soft prediction (first prediction result) is obtained at the first temperature T=10, and hard prediction (second prediction result) is obtained at the second temperature T=1.

[0139] like Figure 3As shown, optionally, the student network consists of 4 feature extraction blocks Env_Block and multiple pooling layers and fully connected layers. In the first 3 Env_Blocks, each is connected to a maximum pooling layer (Max_Pooling), and the convolution kernel size of each Max_Pooling is set to 2*2. The fourth Env_Block layer is connected to a global average pooling layer (Global_Avg_Pooling). Among them, the number of output channels in the 4 Env_Block feature extraction blocks is set to 8, 16, 32, and 64 respectively. After the feature block extraction, the obtained features are input to two fully connected layers (Full_Connect). The number of output neural nodes of the first fully connected layer is 128, and the number of output neural nodes of the second fully connected layer is set to the specific number of environmental sound categories (class_nums), that is, the output soft prediction (first prediction result) is output.

[0140] Optionally, the structure of the feature extraction block Env_Block is as follows Figure 4 As shown in the figure, the input feature input flows in two branches, the first part of the left branch is a convolution layer with a convolution kernel size of 1*1 and an output channel number that is 6 times the number of input channels, connected to a batch normalization layer (BN) and an activation function (ReLU); the second part is a Depth_Block convolution block, and the third part is a convolution layer with a convolution kernel size of 1*1 and an output channel number that is the target output channel number, connected to a batch normalization layer (BN) and an activation function (ReLU). The right branch is a spatial attention block SpaceAtt_Block. After the features flow through the left and right branches, two feature maps of the same size are obtained respectively, and the result of multiplying them point by point is used as the feature output Output.

[0141] Among them, the structure of the Depth_Block convolution block is as follows Figure 5 As shown in the figure, it is divided into two parts. The first part is a grouped convolution layer with a convolution kernel size of 3*3 and the number of convolution kernels equal to the number of input channels. The number of groups is the same as the number of input channels, followed by a batch normalization layer (BN) and an activation function ReLU; the second part is a convolution layer with a convolution kernel size of 1*1 and the number of convolution kernels equal to the number of target output channels, followed by a batch normalization layer (BN) and an activation function ReLU, and outputs Output. The structure of the spatial attention block SpaceAtt_Block is shown in the figure. Figure 6As shown in the figure, the input data is processed by the left and right branches respectively and then spliced ​​along the channel dimension. Finally, it is convolved through a convolution layer with a convolution kernel size of 7*7 and a convolution kernel number of 1, and then output under the action of the activation function Sigmoid. The left and right branches have a maximum pooling layer Max_Pooling and an average pooling layer Avg_Pooling of the channel dimension respectively.

[0142] S513. Calculate the distillation loss according to the first prediction result and the prediction label, calculate the student loss according to the second prediction result and the true label, calculate the first product of the distillation loss and the first weight and calculate the second product of the student loss and the second weight, and determine the loss value according to the sum of the first product and the second product.

[0143] Optionally, the cross entropy is calculated based on the first prediction result and the predicted label to obtain the Distillation Loss (distillation loss) denoted as L soft , calculate the cross entropy based on the second prediction result and the true label (Hard Labels), and get the student loss Student Loss recorded as L hard It should be noted that the first weight α and the second weight β can be adjusted according to actual conditions. For example, α=0.9, β=0.1, and the loss value L is calculated as follows:

[0144] L=αL soft +βL hard

[0145] S514, return to the step of inputting the training samples into the teacher network model until the number of training times or the loss value meets the preset conditions, and obtain the sound recognition model.

[0146] Optionally, during the training process, according to whether the number of training times or the loss value meets the preset conditions, such as whether a certain number of iterations (training times) epochs (such as 200 times) is reached or whether the loss value is less than or equal to the preset loss threshold, if the preset conditions are not met, return to step S100, that is, return to the step of inputting the training sample into the teacher network model, until the number of training times or the loss value meets the preset conditions, record the parameters of the student network at this time, and use the parameters of the student network to obtain the student network model, that is, the sound recognition model. At this time, the sound recognition model is a lightweight sound recognition model (Light_EnvSound_Model) after distillation. With a parameter amount of only 0.36MB, it maintains a strong recognition ability for environmental sounds and high recognition accuracy. The recognition rate has reached 95.66% after testing.

[0147] like Figure 7 As shown, the embodiment of the present invention further provides an identification method, including steps S600-S700:

[0148] S600, obtaining data to be identified,

[0149] S700: Input the data to be recognized into the sound recognition model to obtain the recognition result.

[0150] The sound recognition model is trained by the above-mentioned sound recognition model training method.

[0151] The embodiment of the present invention further provides an electronic device, the electronic device includes a processor and a memory, the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the training method or recognition method of the sound recognition model of the above embodiment. The electronic device of the embodiment of the present invention includes but is not limited to mobile phones, tablet computers, computers, and car computers.

[0152] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the training method or recognition method of the sound recognition model of the aforementioned embodiment.

[0154] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method or recognition method of the sound recognition model of the foregoing embodiment.

[0155] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0156] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0159] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. A person skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents, without causing the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a sound recognition model, characterized in that: include: Get the sound dataset; Performing a first energy value calculation on the sound data set, and performing a first screening process according to a result of the first energy value calculation to obtain first data; Performing gain processing, filtering frequency domain enhancement processing, and random order merging processing on the first data, and combining at least one of the gain processing result, the filtering frequency domain enhancement processing result, and the random order merging processing result to obtain second data; The filtering frequency domain enhancement processing includes directivity attenuation enhancement processing and data frequency domain random enhancement processing; Performing Mel-spectrogram calculation on the second data to obtain training samples; Input the training samples into the teacher network for training to obtain a teacher network model, input the training samples into the student network and perform distillation learning on the student network according to the teacher network model to obtain a sound recognition model; The performing gain processing and filtering frequency domain enhancement processing on the first data includes: Slicing the first data to obtain a plurality of slice data, selecting a first value from a first preset gain range, determining a second value according to the first value and a preset offset, performing a first multiplication on the second value and the slice data respectively, and resynthesizing the first multiplication results to obtain a gain processing result; The directivity attenuation enhancement process includes: determining a cutoff frequency from a preset frequency range, filtering the first data by the cutoff frequency, and obtaining first enhanced data; The data frequency domain random enhancement processing is: performing a first frame windowing processing on the first data, and performing a second discrete Fourier transform on the first frame windowing processing result to obtain a second amplitude spectrum and a phase spectrum, performing frequency band segmentation on the second amplitude spectrum to obtain a plurality of frequency bands, determining a gain value according to a second preset gain range, performing a second multiplication on the gain value and the frequency band respectively, splicing the second multiplication result, and performing an inverse Fourier transform conversion according to the phase spectrum and the spliced ​​second multiplication result to obtain second enhanced data; the filtering frequency domain enhancement processing result includes the first enhanced data and the second enhanced data; The training samples have real labels; the training samples are input into the student network and the student network is distilled and learned according to the teacher network model to obtain a sound recognition model, including: Inputting the training sample into the teacher network model to obtain a predicted label; Inputting the training sample into the student network to obtain a first prediction result at a first temperature and a second prediction result at a second temperature; Calculating a distillation loss according to the first prediction result and the prediction label, calculating a student loss according to the second prediction result and the true label, calculating a first product of the distillation loss and a first weight and calculating a second product of the student loss and a second weight, and determining a loss value according to the sum of the first product and the second product; Return to the step of inputting the training samples into the teacher network model until the number of training times or the loss value meets the preset conditions, thereby obtaining a sound recognition model.

2. The method for training a sound recognition model according to claim 1, characterized in that: The calculating a first energy value of the sound data set includes: Performing frame division, frame stacking and windowing operations on the sound data set to obtain a plurality of frames of first signals; Performing a first discrete Fourier transform on the first signal to determine a first amplitude spectrum of the first signal for each frame; The sum of the energy values ​​of each of the first amplitude spectra is calculated according to the preset number of frequency points to obtain a first energy value calculation result.

3. The method for training a sound recognition model according to claim 2, characterized in that: The first screening process is performed according to the first energy value calculation result to obtain the first data, including: Taking a preset number of frames of continuous first signals as an energy block, and calculating a block energy value corresponding to each of the energy blocks according to the first energy value calculation result; Determine a target frame number from the energy block with the largest block energy value; Performing time domain conversion on the target frame sequence number to obtain a time domain index, and determining a sampling point sequence number according to the time domain index and a preset sampling range; The energy block with the largest block energy value is cut and screened according to the sampling point sequence number and the preset data length to obtain the first data.

4. The method for training a sound recognition model according to claim 1, characterized in that: The performing random merging processing on the first data, and combining at least one of the gain processing result, the filtering frequency domain enhancement processing result, and the random merging processing result to obtain the second data, includes: Calculating the number of sound events according to the preset number of sampling points and the total number of sampling points of the first data; Performing a second energy value calculation on the first data, and performing a second screening process a number of times according to the second energy value calculation result to obtain the number of sound events, that is, sound events; Randomly sorting and splicing the sound events to obtain a random merging processing result; At least one of the gain processing result, the filtering frequency domain enhancement processing result and the out-of-order merging processing result is randomly combined to obtain second data.

5. The method for training a sound recognition model according to claim 1, characterized in that: The performing Mel spectrum calculation on the second data to obtain a training sample includes: Performing a second frame windowing process on the second data, performing a third discrete Fourier transform on the result of the second frame windowing process, and calculating a power spectrum according to the result of the third discrete Fourier transform; Determine a Mel filter function according to a preset frequency upper limit and a preset frequency lower limit; A mel spectrum is obtained according to the mel filter function and the power spectrum; and the training sample includes the mel spectrum.

6. A recognition method, characterized in that: include: Get the data to be identified. Inputting the data to be recognized into a sound recognition model to obtain a recognition result; The sound recognition model is trained by the sound recognition model training method according to any one of claims 1 to 5.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-6.

Images