Audio feature extraction model training method and audio classification method

Abstract

The present application discloses a training method for an audio feature extraction model and an audio classification method, which belongs to the field of computer technology. The method comprises: obtaining original spectrum data and a benchmark recognition result corresponding to the sample audio; performing element occlusion processing on the original spectrum data to obtain fuzzy spectrum data corresponding to the original spectrum data; performing feature extraction on the fuzzy spectrum data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio; performing result recognition on the audio features by the audio recognition model to be trained to obtain predicted recognition results corresponding to the sample audio; and adjusting parameters of the audio feature extraction model to be trained and the audio recognition model to be trained based on the benchmark recognition result and the predicted recognition result. By adopting the present application, the anti-noise performance of the trained audio feature extraction model can be improved, so that it can be applied to audio in more complex scenarios, thereby obtaining more accurate audio features.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method for an audio feature extraction model and an audio classification method. Background Art

[0002] Many current music applications have functions such as song identification. The method of song identification is usually to use a trained audio feature extraction model to extract features from the acquired audio, obtain the audio features corresponding to the audio, and then compare the audio features with the audio features corresponding to each original audio in the database to determine the original audio with a higher similarity with the audio features corresponding to the audio.

[0003] Before performing the above processing, the audio feature extraction model needs to be trained to obtain a trained audio feature extraction model. Generally, the training method of the audio feature extraction model is as follows: obtain the spectral data corresponding to the sample audio and the benchmark recognition result corresponding to the sample audio, use the audio feature extraction model to be trained to extract features from the spectral data corresponding to the sample audio, obtain the audio features corresponding to the sample audio, and then use the audio recognition model to recognize the audio features to obtain the predicted recognition result corresponding to the sample audio. Finally, based on the predicted recognition classification result and the benchmark recognition result, adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained to obtain a trained audio feature extraction model.

[0004] However, the above training method is relatively crude. When the audio feature extraction model trained above is used to extract audio features, the obtained audio features may be relatively inaccurate. Summary of the Invention

[0005] An embodiment of the present application provides a training method for an audio feature extraction model, which can solve the problem that the training method for an audio feature extraction model in the prior art is relatively rough, resulting in relatively inaccurate audio features obtained based on the trained audio feature extraction model.

[0006] In a first aspect, a method for training an audio feature extraction model is provided, the method comprising:

[0007] Obtaining original spectrum data and a reference recognition result corresponding to the sample audio, wherein the reference recognition result is used to indicate the original audio corresponding to the sample audio;

[0008] Performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data;

[0009] Inputting the fuzzy spectrum data into an audio feature extraction model to be trained, and having the audio feature extraction model to be trained perform feature extraction on the fuzzy spectrum data to obtain audio features corresponding to the sample audio;

[0010] Inputting the audio features into an audio recognition model to be trained, and having the audio recognition model to be trained recognize the audio features to obtain a predicted recognition result corresponding to the sample audio, wherein the predicted recognition result is used to indicate a predicted audio corresponding to the sample audio;

[0011] Based on the benchmark recognition result and the predicted recognition result, the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained are adjusted; if the preset end condition is met, the audio feature extraction model after parameter adjustment is determined as the trained audio feature extraction model.

[0012] In a possible implementation, the original spectrum data is spectrum data in matrix form;

[0013] The performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data includes:

[0014] A first number of first elements and a second number of second elements are selected from the original spectrum data for masking to obtain blurred spectrum data corresponding to the original spectrum data, wherein each first element is located in a different column of the original spectrum data and each second element is located in a different row of the original spectrum data.

[0015] In a possible implementation, performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data includes:

[0016] Performing multiple element masking processes on the original spectrum data to obtain multiple different blurred spectrum data corresponding to the original spectrum data;

[0017] Inputting the fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the fuzzy spectrum data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio, includes:

[0018] The multiple different fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple different fuzzy spectrum data to obtain audio features corresponding to the sample audio.

[0019] In a possible implementation, after performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data, the method further includes:

[0020] Based on the same segmentation method, each of the fuzzy spectrum data is segmented to obtain fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data;

[0021] The step of inputting the plurality of different fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the plurality of different fuzzy spectrum data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio includes:

[0022] The fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the fuzzy spectrum sub-data to obtain audio features corresponding to the sample audio.

[0023] In one possible implementation, the audio feature extraction model includes a block embedding layer, a first activation layer, a first normalization layer, a convolutional layer, a global average pooling layer, and a fully connected layer;

[0024] The step of inputting the fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the fuzzy spectrum sub-data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio includes:

[0025] Using the block embedding layer, the first activation layer and the first normalization layer, sequentially process each fuzzy spectrum sub-data to obtain first feature information of each fuzzy spectrum sub-data;

[0026] The first convolution processing unit of the convolution layer processes the first feature information of each fuzzy spectrum sub-data to obtain target feature information of each fuzzy spectrum sub-data; and the second convolution processing unit of the convolution layer processes the target feature information of the plurality of fuzzy spectrum sub-data corresponding to each position area to obtain target feature information of each position area;

[0027] Based on the global average pooling layer, the fully connected layer and the target feature information of each position area, an audio feature corresponding to the sample audio is determined.

[0028] In a possible implementation, the first convolution processing unit of the convolution layer includes a depth-wise convolution layer, a second activation layer, and a second normalization layer;

[0029] The processing of the first feature information of each blurred spectrum sub-data by the convolution layer to obtain the target feature information of each blurred spectrum sub-data includes:

[0030] Inputting the first feature information of each blurred spectrum sub-data into the depth-by-depth convolution layer, and performing convolution processing on the first feature information of each blurred spectrum sub-data by the depth-by-depth convolution layer to obtain the second feature information of each blurred spectrum sub-data;

[0031] inputting the second feature information of each fuzzy spectrum sub-data into the second activation layer, and performing nonlinear activation processing on the second feature information of each fuzzy spectrum sub-data by the second activation layer to obtain third feature information of each fuzzy spectrum sub-data;

[0032] inputting the third feature information of each fuzzy spectrum sub-data into the second normalization layer, and performing normalization processing on the third feature information of each fuzzy spectrum sub-data by the second normalization layer to obtain fourth feature information of each fuzzy spectrum sub-data;

[0033] For each of the fuzzy spectrum sub-data, the fourth feature information of the fuzzy spectrum sub-data and the first feature information of the fuzzy spectrum sub-data are combined and processed to obtain target feature information of the fuzzy spectrum sub-data.

[0034] In a possible implementation, the second convolution processing unit of the convolution layer includes a point-by-point convolution layer, a third activation layer, and a third normalization layer;

[0035] The convolution layer processes the target feature information of the multiple blurred spectrum sub-data corresponding to each position area to obtain the target feature information of each position area, including:

[0036] Inputting the target feature information of each blurred spectrum sub-data into the point-by-point convolution layer, and performing convolution processing on the target feature information of multiple blurred spectrum sub-data corresponding to each position area by the point-by-point convolution layer to obtain first feature information of each position area;

[0037] Inputting the first feature information of each position area into the third activation layer, and performing nonlinear activation processing on the first feature information of each position area by the third activation layer to obtain the second feature information of each position area;

[0038] The second feature information of each location area is input into the third normalization layer, and the third normalization layer performs normalization processing on the second feature information of each location area to obtain the target feature information of each location area.

[0039] In a possible implementation, there are multiple convolutional layers, and each convolutional layer outputs target feature information of each position area;

[0040] The determining the audio features of the sample audio based on the global average pooling layer, the fully connected layer, and the target feature information of each location area includes:

[0041] The target feature information of the same position area output by each convolutional layer is spliced ​​to obtain the combined feature information of each position area; the combined feature information of each position area is input into the global average pooling layer to obtain the pooled feature information of each position area; the pooled feature information of multiple position areas is input into the fully connected layer for connection to obtain the audio features corresponding to the sample audio.

[0042] In a possible implementation, the spectrum data includes at least one of MFCC (Mel-Frequency Cepstral Coefficients) feature information and CQT (Constant Q Transform) feature information.

[0043] In a second aspect, a method for audio classification is provided, the method comprising:

[0044] Input each audio in the audio library into the trained audio feature extraction model as described in any one of the above items, and use the trained audio feature extraction model to extract features from each audio to obtain audio features corresponding to each audio;

[0045] Calculate the similarity between the audio features corresponding to each two audios;

[0046] An audio group corresponding to at least one song is determined based on the similarity, wherein the similarity between each two audios in the same audio group is greater than or equal to a similarity threshold.

[0047] In a third aspect, a training device for an audio feature extraction model is provided, the device comprising:

[0048] An acquisition module, configured to acquire original spectrum data and a reference recognition result corresponding to the sample audio, wherein the reference recognition result is used to indicate the original audio corresponding to the sample audio;

[0049] an occlusion module, configured to perform element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data;

[0050] a first determining module, configured to input the fuzzy spectrum data into an audio feature extraction model to be trained, and have the audio feature extraction model to be trained perform feature extraction on the fuzzy spectrum data to obtain audio features corresponding to the sample audio;

[0051] a second determination module, configured to input the audio features into an audio recognition model to be trained, and have the audio recognition model to be trained recognize the audio features to obtain a predicted recognition result corresponding to the sample audio, wherein the predicted recognition result is used to indicate a predicted audio corresponding to the sample audio;

[0052] A parameter adjustment module is used to adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained based on the benchmark recognition result and the predicted recognition result; if the preset end condition is met, the audio feature extraction model after parameter adjustment is determined as the trained audio feature extraction model.

[0053] In a possible implementation, the original spectrum data is spectrum data in matrix form;

[0054] The shielding module is used to:

[0055] A first number of first elements and a second number of second elements are selected from the original spectrum data for masking to obtain blurred spectrum data corresponding to the original spectrum data, wherein each first element is located in a different column of the original spectrum data and each second element is located in a different row of the original spectrum data.

[0056] In a possible implementation, the shielding module is configured to:

[0057] Performing multiple element masking processes on the original spectrum data to obtain multiple different blurred spectrum data corresponding to the original spectrum data;

[0058] The first determining module is configured to:

[0059] The multiple different fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple different fuzzy spectrum data to obtain audio features corresponding to the sample audio.

[0060] In a possible implementation, the apparatus further includes a segmentation module configured to:

[0061] Based on the same segmentation method, each of the fuzzy spectrum data is segmented to obtain fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data;

[0062] The first determining module is configured to:

[0063] The fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the fuzzy spectrum sub-data to obtain audio features corresponding to the sample audio.

[0064] In one possible implementation, the audio feature extraction model includes a block embedding layer, a first activation layer, a first normalization layer, a convolutional layer, a global average pooling layer, and a fully connected layer;

[0065] The first determining module is configured to:

[0066] Using the block embedding layer, the first activation layer and the first normalization layer, sequentially process each fuzzy spectrum sub-data to obtain first feature information of each fuzzy spectrum sub-data;

[0067] The first convolution processing unit of the convolution layer processes the first feature information of each fuzzy spectrum sub-data to obtain target feature information of each fuzzy spectrum sub-data; and the second convolution processing unit of the convolution layer processes the target feature information of the plurality of fuzzy spectrum sub-data corresponding to each position area to obtain target feature information of each position area;

[0068] Based on the global average pooling layer, the fully connected layer and the target feature information of each position area, an audio feature corresponding to the sample audio is determined.

[0069] In a possible implementation, the first convolution processing unit of the convolution layer includes a depth-wise convolution layer, a second activation layer, and a second normalization layer;

[0070] The first determining module is configured to:

[0071] Inputting the first feature information of each blurred spectrum sub-data into the depth-by-depth convolution layer, and performing convolution processing on the first feature information of each blurred spectrum sub-data by the depth-by-depth convolution layer to obtain the second feature information of each blurred spectrum sub-data;

[0072] inputting the second feature information of each fuzzy spectrum sub-data into the second activation layer, and performing nonlinear activation processing on the second feature information of each fuzzy spectrum sub-data by the second activation layer to obtain third feature information of each fuzzy spectrum sub-data;

[0073] inputting the third feature information of each fuzzy spectrum sub-data into the second normalization layer, and performing normalization processing on the third feature information of each fuzzy spectrum sub-data by the second normalization layer to obtain fourth feature information of each fuzzy spectrum sub-data;

[0074] For each of the fuzzy spectrum sub-data, the fourth feature information of the fuzzy spectrum sub-data and the first feature information of the fuzzy spectrum sub-data are combined and processed to obtain target feature information of the fuzzy spectrum sub-data.

[0075] In a possible implementation, the second convolution processing unit of the convolution layer includes a point-by-point convolution layer, a third activation layer, and a third normalization layer;

[0076] The first determining module is configured to:

[0077] Inputting the target feature information of each blurred spectrum sub-data into the point-by-point convolution layer, and performing convolution processing on the target feature information of multiple blurred spectrum sub-data corresponding to each position area by the point-by-point convolution layer to obtain first feature information of each position area;

[0078] Inputting the first feature information of each position area into the third activation layer, and performing nonlinear activation processing on the first feature information of each position area by the third activation layer to obtain the second feature information of each position area;

[0079] The second feature information of each location area is input into the third normalization layer, and the third normalization layer performs normalization processing on the second feature information of each location area to obtain the target feature information of each location area.

[0080] In a possible implementation, there are multiple convolutional layers, and each convolutional layer outputs target feature information of each position area;

[0081] The first determining module is configured to:

[0082] The target feature information of the same position area output by each convolutional layer is spliced ​​to obtain the combined feature information of each position area; the combined feature information of each position area is input into the global average pooling layer to obtain the pooled feature information of each position area; the pooled feature information of multiple position areas is input into the fully connected layer for connection to obtain the audio features corresponding to the sample audio.

[0083] In a possible implementation, the spectrum data includes at least one of MFCC feature information and CQT feature information.

[0084] According to a fourth aspect, an audio classification device is provided, the device comprising:

[0085] A feature extraction module, configured to input each audio in the audio library into a trained audio feature extraction model as described above, and perform feature extraction on each audio by the trained audio feature extraction model to obtain audio features corresponding to each audio;

[0086] A calculation module, used to calculate the similarity between audio features corresponding to each two audios;

[0087] The grouping module is configured to determine an audio group corresponding to at least one song based on the similarity, wherein the similarity between each two audios in the same audio group is greater than or equal to a similarity threshold.

[0088] In a fifth aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the training method of the audio feature extraction model or the audio classification method.

[0089] In a sixth aspect, a computer-readable storage medium is provided, in which at least one instruction is stored. The instruction is loaded and executed by a processor to implement the operations performed by the training method of the audio feature extraction model or the audio classification method.

[0090] In a seventh aspect, a computer program product is provided, which includes at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the training method of the audio feature extraction model or the audio classification method.

[0091] The beneficial effects of the technical solution provided by the embodiment of the present application are as follows: the solution mentioned in the embodiment of the present application can perform element occlusion processing on the original spectrum data corresponding to the sample audio, thereby obtaining fuzzy spectrum data corresponding to the original spectrum data, and then, based on the audio feature extraction model to be trained, perform feature extraction on the fuzzy spectrum data to obtain the audio features corresponding to the sample audio, and then, based on the audio recognition model to be trained, perform result recognition on the audio features, thereby obtaining the output predicted recognition results corresponding to the sample audio, and then, based on the benchmark recognition results and the predicted recognition results, adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained, and when the preset end conditions are met, determine the audio feature extraction model after parameter adjustment as the trained audio feature extraction model. Using the present application, the data input to the audio feature extraction model is the fuzzy spectrum data obtained after element occlusion processing, so that the noise resistance of the trained audio feature extraction model can be improved, so that it can be applied to audio in more complex scenes, thereby obtaining more accurate audio features. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0093] Figure 1 This is a flow chart of a training method for an audio feature extraction model provided in an embodiment of the present application;

[0094] Figure 2 This is a flow chart of a method for performing multiple element occlusion processing on spectrum data provided by an embodiment of the present application;

[0095] Figure 3 This is a flow chart of a method for segmenting multiple fuzzy spectrum data provided by an embodiment of the present application;

[0096] Figure 4 This is a flow chart of a feature extraction method provided in an embodiment of the present application;

[0097] Figure 5 This is a flow chart of a feature extraction method provided in an embodiment of the present application;

[0098] Figure 6 This is a flow chart of an audio classification method provided by an embodiment of the present application;

[0099] Figure 7 Schematic diagram of a training device for an audio feature extraction model provided in an embodiment of the present application;

[0100] Figure 8 This is a schematic diagram of the structure of an audio classification device provided in an embodiment of the present application;

[0101] Figure 9 This is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0103] The embodiment of the present application provides a method for training an audio feature extraction model, which can be implemented by a server. The server can be a single server or a server cluster consisting of multiple servers.

[0104] The server may include a processor, a memory, a communication component, etc., and the processor is connected to the memory and the communication component respectively.

[0105] The processor can be a central processing unit (CPU), which can be used to read instructions and process data, for example, performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data, performing feature extraction on the blurred spectrum data based on the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio, performing result recognition on the audio features based on the audio recognition model to be trained to obtain predicted recognition results corresponding to the sample audio, adjusting parameters of the audio feature extraction model to be trained and the audio recognition model to be trained based on the predicted recognition results and the benchmark recognition results, and so on.

[0106] The memory may be various volatile memories or non-volatile memories, such as a solid state disk (SSD), a dynamic random access memory (DRAM), etc. The memory may be used for data storage, for example, storage of original spectrum data and benchmark recognition result data corresponding to the acquired sample audio, storage of fuzzy spectrum data corresponding to the acquired original spectrum data, storage of data corresponding to the audio feature extraction model to be trained, storage of data corresponding to the audio recognition model to be trained, storage of intermediate data generated in the process of feature extraction of fuzzy spectrum data based on the audio feature extraction model to be trained, storage of audio feature data corresponding to the acquired sample audio, storage of intermediate data generated in the process of result recognition of audio features based on the audio recognition model to be trained, storage of predicted recognition result data corresponding to the acquired sample audio, storage of intermediate data generated in the process of parameter adjustment of the audio feature extraction model to be trained and the audio recognition model to be trained based on the benchmark recognition result and the predicted recognition result, storage of data corresponding to the acquired trained audio feature extraction model, etc.

[0107] The communication component may be a wired network connector, a wireless fidelity (WiFi) module, a Bluetooth module, a cellular network communication module, etc. The communication component may be used to perform data transmission with other devices.

[0108] Figure 1 This is a flowchart of a method for training an audio feature extraction model provided in an embodiment of the present application.

[0109] See also Figure 1 , the embodiment includes:

[0110] 101. Obtain original spectrum data and benchmark recognition results corresponding to the sample audio.

[0111] The benchmark recognition result is used to indicate the original audio corresponding to the sample audio.

[0112] In implementation, a training sample set can be first obtained, which includes multiple sample audios and corresponding benchmark recognition results. One specific implementation method for obtaining the sample audios is to obtain, for each song, the original audio and multiple corresponding cover audios of the song, and each of the original audio and the multiple cover audios is used as a sample audio.

[0113] After obtaining a training sample set from multiple different songs, you can proceed to model training. When training the model, you need to use the original spectrum data corresponding to the sample audio and the benchmark recognition results.

[0114] 102. Perform element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data.

[0115] The original spectrum data corresponding to the sample audio may be represented in a matrix form or a vector form. The original spectrum data includes multiple elements, wherein each element is used to represent the amplitude or energy at different time points and different frequency points in the sample audio.

[0116] In implementation, after obtaining the original spectrum data corresponding to the sample audio, some elements in the original spectrum data can be masked, that is, the values ​​of some elements in the original spectrum data can be modified, and they can be modified to any preset value or any preset character, so as to obtain the blurred spectrum data corresponding to the original spectrum data.

[0117] 103. Input the fuzzy spectrum data into the audio feature extraction model to be trained. The audio feature extraction model to be trained performs feature extraction on the fuzzy spectrum data to obtain audio features corresponding to the sample audio.

[0118] Among them, the audio feature extraction model can be a machine learning model such as a CNN model or a DNN model, which extracts information from the fuzzy spectrum data to obtain a set of feature parameters that can be used to characterize the changes of the audio signal over time, that is, the audio features corresponding to the sample audio.

[0119] 104. Input the audio features into the audio recognition model to be trained, and the audio recognition model to be trained recognizes the audio features to obtain a predicted recognition result corresponding to the sample audio.

[0120] The prediction recognition result is used to indicate the predicted audio corresponding to the sample audio.

[0121] In a specific embodiment, each original audio stored in the database can be numbered, and each original audio and the original audio number can be stored in correspondence. Thus, after the audio features corresponding to the sample audio are input into the audio recognition model to be trained, multiple predicted probability values ​​output by the audio recognition model can be obtained, and each predicted probability value is associated with a number recognized by the audio recognition model. The predicted probability value is used to represent the possibility that the sample audio is the original audio (or an imitation of the original audio) corresponding to the number associated with the predicted probability value.

[0122] The prediction recognition result may be a predicted probability value and a number associated with the predicted probability value.

[0123] 105. Based on the benchmark recognition results and the predicted recognition results, adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained.

[0124] The benchmark recognition result may be the number of the original audio corresponding to the sample audio, and a benchmark probability value corresponding to the number of the sample audio. The benchmark probability value is used to indicate the likelihood that the sample audio is the original audio corresponding to the number. The specific value of the benchmark probability value may be any reasonable value. For example, the benchmark probability value may be directly set to 1. Of course, other values ​​are also possible, and this embodiment of the application is not limited to this.

[0125] In implementation, after obtaining the predicted recognition result, the baseline recognition result and the predicted recognition result can be input into the loss function to obtain a loss value, and based on the loss value, the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained are adjusted simultaneously.

[0126] 106. If the preset end condition is met, the audio feature extraction model after parameter adjustment is determined as the trained audio feature extraction model.

[0127] In implementation, based on the method of steps 101-105, the audio feature extraction model to be trained and the audio recognition model to be trained can be adjusted multiple times using each sample audio in the training sample set and the corresponding benchmark recognition result. After each parameter adjustment, the following judgment can be made: whether the current training result meets the preset end condition. If so, the audio feature extraction model after the last parameter adjustment can be determined as the trained audio feature extraction model.

[0128] In the audio feature extraction model training method proposed in the above steps 101-106, the data input to the audio feature extraction model is the blurred spectrum data obtained after element occlusion processing. In this way, the noise resistance performance of the trained audio feature extraction model can be improved, making it applicable to audio in more complex scenarios, thereby obtaining more accurate audio features.

[0129] Optionally, the following is an example of a preset end condition:

[0130] The first is when the number of parameter adjustments for the audio feature extraction model to be trained and the audio recognition model to be trained reaches a parameter adjustment threshold. During implementation, staff can pre-set the parameter adjustment threshold. When the number of parameter adjustments reaches the threshold, training can be stopped and the audio feature extraction model obtained after the last parameter adjustment can be determined as the trained audio feature extraction model. The parameter adjustment threshold can be any reasonable value, such as 300 or 350, and is not limited in this embodiment of the present application.

[0131] The second type is that the loss values ​​obtained from a preset number of consecutive training runs are all less than a preset loss threshold. The preset number and the preset loss threshold can both be any reasonable values. For example, the preset number can be 4 or 5, and the preset loss threshold can be 0.06, etc. This embodiment of the application is not limited to this.

[0132] The third type is that the number of parameter adjustments reaches the parameter adjustment threshold, and the loss values ​​obtained from the preset number of consecutive training times are all less than the preset loss value threshold.

[0133] The preset end condition can be any one of the three above, or other end conditions, which is not limited in the embodiment of the present application.

[0134] In one possible implementation, the original spectrum data in step 101 includes at least one of MFCC feature information and CQT feature information. Of course, the original spectrum data can also be any other reasonable data, which can be selected as needed and is not limited in this embodiment of the present application.

[0135] MFCC feature information

[0136] MFCC feature information is the cepstral parameters extracted in the Mel scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear frequency.

[0137] The method for obtaining MFCC feature information can be as follows: the sample audio is sequentially subjected to pre-emphasis processing, framing processing, windowing processing, FFT (Fast Fourier Transform) processing, filtering processing through a Mel filter bank, logarithmic operation processing and DCT (Discrete Cosine Transform) processing, thereby obtaining the MFCC feature information corresponding to the sample audio.

[0138] CQT characteristic information

[0139] Since in music, all the notes are composed of 12 equal temperaments of several octaves, these 12 equal temperaments correspond to the twelve semitones on an octave in the piano. The frequency ratio between these semitones is 2 1 / 12 Obviously, for two octaves of the same pitch, the frequency of the higher octave is twice that of the lower octave.

[0140] Therefore, in music, sound is exponentially distributed. If you want to obtain an audio feature extraction model for extracting features of music audio, you can set all the sample audio in the training sample set to music audio, and set the original spectrum data corresponding to the sample audio as the CQT feature information corresponding to the sample audio.

[0141] CQT feature information refers to a filter bank with exponentially distributed center frequencies and varying filter bandwidths, but with a constant center frequency to bandwidth ratio of Q. The horizontal frequency axis of CQT feature information is not linear, but rather log2-based, and the filter window length can be varied based on the spectral line frequencies to achieve better performance. Thus, using CQT feature information to train an existing audio feature extraction model can make the trained audio feature extraction model more accurate, thereby yielding more accurate audio features.

[0142] Specifically, the CQT characteristic information can be obtained by the following formula:

[0143]

[0144]

[0145]

[0146]

[0147] f k =2 k / b f min ,k=0,1,···,K-1 (5)

[0148] Among them, x(n) is the audio signal, k is the frequency number, N k is the window length that varies with k, Is of length N k The window function, j is the imaginary unit, e is the natural constant, n is the variable, b is the number of frequency lines contained in an octave, Q is the constant factor, Indicates rounding up, f max is the center frequency of the highest filter, f min is the center frequency of the minimum filter, f s is the sampling frequency.

[0149] In a possible implementation, there may be multiple ways to process element occlusion in step 102. The following two processing methods are used as examples for illustration:

[0150] The first

[0151] When performing element occlusion processing on the original spectrum data, a third number of elements may be randomly selected from the original spectrum data for occlusion, thereby obtaining the original spectrum data subjected to element occlusion processing, that is, blurred spectrum data corresponding to the original spectrum data.

[0152] Among them, the third number can be any reasonable number, the third number can be set at will, or it can be equal to a preset proportion of the number of all elements included in the original spectrum data. For example, the number of all elements included in the original spectrum data is 1000, and the preset proportion is 30%, then the third number is 300, and so on. The selection method and specific value of the third number are not limited in the embodiments of this application.

[0153] The second

[0154] In order to distribute the obscured elements in the blurred spectrum data more evenly, the following processing can be performed when performing element obscuration processing on the original spectrum data: a first number of first elements and a second number of second elements are randomly selected in the original spectrum data for obscuration, thereby obtaining blurred spectrum data corresponding to the original spectrum data, wherein each first element is located in a different column of the original spectrum data, and each second element is located in a different row of the original spectrum data.

[0155] For example, the original spectrum data is 30×20 matrix data, the set first number is 10, and the set second number is 5. Then, when performing element masking processing on the original spectrum data, 10 first elements located in different columns and 5 second elements located in different rows can be randomly selected from the multiple elements contained in the original spectrum data for masking, thereby obtaining blurred spectrum data corresponding to the original spectrum data.

[0156] Optionally, there may be identical elements between the first elements of the first number and the second elements of the second number (i.e., the first element may be the same as the second element), or there may not be any identical elements between the first elements of the first number and the second elements of the second number (i.e., the first element is different from the second element). In this regard, corresponding settings can be made according to requirements, and the embodiments of the present application are not limited to this.

[0157] Optionally, a first number of first elements are randomly selected from the original spectrum data for masking, and a second number of second elements are randomly selected from the original spectrum data for masking. These two processing actions can be performed simultaneously or one after the other. The embodiment of the present application does not limit this.

[0158] In one possible implementation, to improve the accuracy of model training, the following processing can be performed:

[0159] When performing the above step 102, the original spectrum data can be subjected to multiple element occlusion processes to obtain multiple different blurred spectrum data corresponding to the original spectrum data. This is equivalent to increasing the spatial dimension of the blurred spectrum data. For example, Figure 2 As shown, the original spectrum data corresponding to the sample audio can be subjected to element masking processing three times respectively, thereby obtaining three different blurred spectrum data corresponding to the original spectrum data.

[0160] Correspondingly, when performing the processing of the above step 103, multiple different fuzzy spectrum data can be input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple different fuzzy spectrum data to obtain audio features corresponding to the sample audio.

[0161] In practice, when training an audio feature extraction model, multiple different fuzzy spectrum data corresponding to the original spectrum data can be input into the audio feature extraction model to be trained. The model then extracts features from these multiple fuzzy spectrum data, thereby obtaining audio features corresponding to the sample audio. This approach increases the spatial dimensionality of the fuzzy spectrum data. Thus, when the audio feature extraction model extracts features from the fuzzy spectrum data, it can comprehensively consider information from multiple spatial dimensions, thereby enhancing the accuracy of feature extraction. Furthermore, experimental results show that this approach is more conducive to the feature extraction effect of the audio feature extraction model, resulting in more accurate output audio features.

[0162] In one possible implementation, in order to enable the audio feature extraction model to perform more refined feature extraction on the fuzzy spectrum data, the fuzzy spectrum data can be segmented first and then feature extraction can be performed. The corresponding processing can be as follows:

[0163] like Figure 3 As shown, after obtaining multiple different fuzzy spectrum data corresponding to the original spectrum data, each fuzzy spectrum data can be segmented based on the same segmentation method to obtain multiple fuzzy spectrum sub-data corresponding to different position areas in each fuzzy spectrum data.

[0164] Here, the position region refers to the position of the blurred spectrum sub-data in the corresponding blurred spectrum data. For example, when the fuzzy spectrum data is in matrix form, the position region refers to the row and column positions of the matrix corresponding to the blurred spectrum sub-data in the matrix of the fuzzy spectrum data.

[0165] In practice, the segmentation method refers to the size information of each fuzzy spectrum data, or it can also be the location area of ​​each fuzzy spectrum sub-data in the fuzzy spectrum data when the fuzzy spectrum data is segmented. The staff can set the segmentation method and segment each fuzzy spectrum data according to the segmentation method. For example, the staff can set the size information of each fuzzy spectrum sub-data, and then segment each fuzzy spectrum data according to the set size information, thereby obtaining multiple fuzzy spectrum sub-data. For example, if the size of the fuzzy spectrum sub-data is 100×120, and the size information of each fuzzy spectrum sub-data is set to 25×30, then when segmenting each fuzzy spectrum data, the segmentation can be performed based on the size information, thereby segmenting each fuzzy spectrum data into fuzzy spectrum sub-data corresponding to 16 different location areas.

[0166] Correspondingly, when performing the processing of step 103 , feature extraction may be performed on the multiple fuzzy spectrum sub-data corresponding to the multiple fuzzy spectrum data based on the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio.

[0167] In implementation, when training the audio feature extraction model to be trained, multiple fuzzy spectrum sub-data corresponding to multiple fuzzy spectrum data can be input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple fuzzy spectrum sub-data to obtain audio features corresponding to the sample audio.

[0168] From the experimental results, it can be seen that more accurate audio features can be obtained by segmenting the fuzzy spectrum sub-data.

[0169] In one possible implementation, the structure of the audio feature extraction model may have multiple possibilities. For example, the audio feature extraction model may include a block embedding layer, a first activation layer, a first normalization layer, a convolutional layer, a global average pooling layer, and a fully connected layer.

[0170] Correspondingly, such as Figure 4 As shown, when the audio feature extraction model to be trained performs feature extraction on multiple fuzzy spectrum sub-data corresponding to multiple fuzzy spectrum data, the following processing can be performed:

[0171] Each fuzzy spectrum sub-data is processed sequentially using a block embedding layer, a first activation layer, and a first normalization layer to obtain the first feature information of each fuzzy spectrum sub-data. The first convolution processing unit of the convolution layer processes the first feature information of each fuzzy spectrum sub-data to obtain the target feature information of each fuzzy spectrum sub-data. The second convolution processing unit of the convolution layer also processes the target feature information of multiple fuzzy spectrum sub-data corresponding to each position region to obtain the target feature information corresponding to each position region. Based on the global average pooling layer, the fully connected layer, and the target feature information corresponding to each position region, comprehensive processing is performed to obtain the audio features corresponding to the sample audio.

[0172] Each fuzzy spectrum data includes multiple fuzzy spectrum sub-data, each fuzzy spectrum sub-data corresponds to a position area in the fuzzy spectrum data, and since the dimensions of these multiple fuzzy spectrum data are the same, in these multiple fuzzy spectrum data, each same position area corresponds to multiple fuzzy spectrum sub-data. For example, three different fuzzy spectrum data are segmented, and each fuzzy spectrum data is segmented into four fuzzy spectrum sub-data, namely the upper left corner, the lower left corner, the upper right corner, and the lower right corner. Then, there are four same position areas (upper left corner, lower left corner, upper right corner, and lower right corner) in these three fuzzy spectrum data, and each position area corresponds to three fuzzy spectrum sub-data.

[0173] In implementation, each fuzzy spectrum sub-data can be input into a block embedding layer, and the feature information output by the block embedding layer can be input into the first activation layer, and the feature information output by the first activation layer can be input into the first normalization layer, thereby obtaining the first feature information of each fuzzy spectrum sub-data output by the first normalization layer. Then, the first feature information of each fuzzy spectrum data can be input into the first convolution unit of the convolution layer for convolution processing to obtain the target feature information of each fuzzy spectrum sub-data output, and then the target feature information of the multiple fuzzy spectrum sub-data corresponding to each position area can be input into the second convolution processing unit of the convolution layer for convolution processing to obtain the target feature information of each position area output. Then, based on the obtained target feature information of each position area, the global average pooling layer, and the fully connected layer, the audio features corresponding to the sample audio are determined.

[0174] In one possible implementation, there are multiple methods for determining the audio features of the sample audio based on the global average pooling layer, the fully connected layer, and the target feature information of each location region. The following describes one of them:

[0175] The target feature information of the same position area output by each convolutional layer is spliced ​​to obtain the combined feature information of each position area, and the combined feature information of each position area is input into the global average pooling layer to obtain the pooled feature information of each position area. The pooled feature information of multiple position areas is input into the fully connected layer for connection to obtain the audio features corresponding to the sample audio.

[0176] In implementation, the audio feature extraction model may include only one convolution layer or multiple convolution layers, and the parameters of these multiple convolution layers are different. When the audio feature extraction model includes only one convolution layer, the target feature information of the same position area output by each convolution layer is spliced, and the combined feature information of each position area is the target feature information of the position area output by the only convolution layer. When the audio feature extraction model includes multiple convolution layers, each convolution layer outputs target feature information of multiple position areas, that is, multiple target feature information corresponding to each position area can be obtained. Then, for each position area, the following processing can be performed: the multiple target feature information of the position area is spliced ​​to obtain the combined feature information of the position area.

[0177] The combined feature information of each location area can be obtained in the above manner.

[0178] Regardless of whether it includes only one convolutional layer or multiple convolutional layers, after obtaining the combined feature information of each position area, it can be input into the global average pooling layer to perform pooling processing on the combined feature information of the same position, thereby obtaining the pooled feature information of each position area. The pooled feature information of each position area is then input into the fully connected layer to obtain the audio features corresponding to the output sample audio.

[0179] The above process is illustrated by an example: the size of the blurred spectrum data is 90×90, and the set size information is 30×30. When the original spectrum data is subjected to element occlusion processing, the element occlusion processing is performed on the original spectrum data three times, thereby obtaining three blurred spectrum data corresponding to the original spectrum data. Then, the three blurred spectrum data are segmented using the set size to obtain nine blurred spectrum sub-data corresponding to each blurred spectrum data, wherein each blurred spectrum sub-data corresponds to its own corresponding position area in the blurred spectrum data. After sequentially processing each blurred spectrum sub-data using the block embedding layer, the first activation layer, and the first normalization layer, the first feature information corresponding to 27 blurred spectrum sub-data can be obtained. Then, these 27 first feature information are input into one or more convolutional layers. The first convolution processing unit of each convolutional layer performs convolution processing on these 27 first feature information, outputting the target feature information of 27 fuzzy spectrum sub-data. The corresponding second convolution processing unit then performs convolution processing on the target feature information of the 3 fuzzy spectrum sub-data corresponding to the 9 position areas, thereby obtaining the target feature information of the output 9 position areas. That is, each convolution layer can output the target feature information of 9 position areas. Then, the target feature information of the same position area can be spliced ​​to obtain the combined feature information of the 9 position areas. The combined feature information of these 9 position areas is then input into the global average pooling layer to obtain the pooled feature information corresponding to the 9 position areas after dimensionality reduction. The pooled feature information corresponding to these 9 position areas is then input into the fully connected layer to obtain the audio features corresponding to the output sample audio.

[0180] Optionally, the audio feature extraction model in the embodiment of the present application may be a ConvMixer model, such as Figure 5 As shown, the first convolution processing unit of the above-mentioned convolution layer may include a depth-wise convolution layer, a second activation layer and a second normalization layer, and the second convolution processing unit of the convolution layer may include a point-wise convolution layer, a third activation layer and a third normalization layer.

[0181] Among them, the depth-by-depth convolution layer can also be called Depthwise Convolution, the point-by-point convolution layer can also be called Pointwise Convolution, the above-mentioned first activation layer and the second activation layer can both use GELU (Gaussian Error Linerar Units), the first normalization layer and the second normalization layer can both use BatchNorm (Batch Normalization), the above-mentioned block embedding layer can also be called PatchEmbedding, the above-mentioned global average pooling layer can also be called Golbal Average Pooling, and the above-mentioned fully connected layer can also be called Fully Connected.

[0182] Next, the processing flow of the first convolution processing unit of the convolution layer is introduced:

[0183] After obtaining the first feature information of each fuzzy spectrum sub-data output by the first normalization layer, the first feature information of each fuzzy spectrum sub-data can be input into the depth-by-depth convolution layer, and the depth-by-depth convolution layer performs convolution processing on the first feature information of each fuzzy spectrum sub-data respectively, so as to obtain the second feature information of each fuzzy spectrum sub-data.

[0184] The second characteristic information of each fuzzy spectrum sub-data is then input into the second activation layer, and the second activation layer performs nonlinear activation processing on the second characteristic information of each fuzzy spectrum sub-data to obtain the third characteristic information of each fuzzy spectrum sub-data as output.

[0185] The third characteristic information of each fuzzy spectrum sub-data is then input into the second normalization layer, which normalizes the third characteristic information of each fuzzy spectrum sub-data respectively, thereby obtaining the fourth characteristic information of each fuzzy spectrum sub-data as output.

[0186] Then, for each fuzzy spectrum sub-data, the following processing is performed: the fourth feature information of the fuzzy spectrum sub-data is combined with the first feature information of the fuzzy spectrum sub-data to obtain the target feature information of the fuzzy spectrum sub-data. This combination can be performed in a variety of ways, such as performing a bitwise addition process on the fourth feature information and the first feature information, or performing a concatenation process, etc., which is not limited in this embodiment of the present application.

[0187] Next, the processing flow of the second convolution processing unit of the convolution layer is introduced:

[0188] After obtaining the target feature information of each fuzzy spectrum sub-data, the target feature information of each fuzzy spectrum sub-data can be input into a point-by-point convolution layer. The point-by-point convolution layer performs convolution processing on the target feature information of multiple fuzzy spectrum sub-data corresponding to each position area, thereby obtaining the first feature information of each position area.

[0189] The first feature information of each position area is then input into the third activation layer, and the third activation layer performs nonlinear activation processing on the first feature information of each position area to obtain the second feature information of each position area.

[0190] The second feature information of each location area is then input into the third normalization layer, which performs normalization on the second feature information of each location area to obtain the target feature information of each location area.

[0191] Typically, each channel in the convolution layer in the prior art has three convolution kernels, while only one convolution kernel is set in each channel of the depth-wise convolution layer of the above structure, which can effectively reduce the amount of data calculation.

[0192] Optionally, the number of convolutional layers (i.e. Figure 5 In the embodiment of the present application, the audio feature extraction model when the number of convolutional layers is 5, 10 and 20 is experimentally verified. The experiment shows that when the number of convolutional layers is 20, the output result of the audio feature extraction model is more accurate.

[0193] Of course, the number of convolutional layers can also be other numbers, which is not limited in the embodiments of the present application.

[0194] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0195] The solution mentioned in the embodiment of the present application can perform element occlusion processing on the original spectrum data corresponding to the sample audio, thereby obtaining fuzzy spectrum data corresponding to the original spectrum data, and then perform feature extraction on the fuzzy spectrum data based on the audio feature extraction model to be trained to obtain the audio features corresponding to the sample audio, and then perform result recognition on the audio features based on the audio recognition model to be trained to obtain the predicted recognition results corresponding to the sample audio, and then, based on the baseline recognition result and the predicted recognition result, adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained, and when the preset end condition is met, determine the audio feature extraction model after parameter adjustment as the trained audio feature extraction model. Using the present application, the data input to the audio feature extraction model is the fuzzy spectrum data obtained after element occlusion processing, so that the noise resistance of the trained audio feature extraction model can be improved, so that it can be applied to audio in more complex scenes, thereby obtaining more accurate audio features.

[0196] There are multiple audios stored in the audio library, and these audios include the original audio and cover audio of the same song. In order to group each song and determine all the audios corresponding to each song, the embodiment of the present application provides an audio classification method, such as Figure 6 As shown, the method includes:

[0197] 601. Input each audio in the audio library into the trained audio feature extraction model respectively, and use the trained audio feature extraction model to extract features from each audio to obtain audio features corresponding to each audio.

[0198] In implementation, the trained audio feature extraction model obtained by the above training method is used to extract features from each audio in the audio library, thereby obtaining the audio features corresponding to each audio, which can then be stored in the database.

[0199] 602. Calculate the similarity between audio features corresponding to every two audios.

[0200] In implementation, the audio features corresponding to each audio stored in the database may be obtained, and then the similarity between the audio features corresponding to every two audios may be calculated.

[0201] There are many methods for calculating the similarity between audio features. For example, the similarity between two audio features can be calculated using the Euclidean Distance method, the angle cosine method, etc. Of course, other calculation methods can also be used, and this embodiment of the present application does not limit this.

[0202] 603. Determine an audio group corresponding to at least one song based on the similarity.

[0203] The similarity between every two audios in the same audio group is greater than or equal to a similarity threshold.

[0204] In implementation, staff can pre-set the similarity threshold.

[0205] After calculating the similarity between the audio features corresponding to each pair of audio files, audio files with a similarity greater than or equal to a similarity threshold can be classified as belonging to the same song. This method can be used to obtain an audio group for each song corresponding to the audio files in the audio library. In an audio group for a song, one or more audio files included are either the original audio file or a cover version of the song.

[0206] Optionally, the similarity threshold may be any reasonable value, for example, it may be a value such as 0.95, and this embodiment of the present application does not limit this.

[0207] The present application embodiment provides a training device for an audio feature extraction model, which may be the computer device in the above embodiment, such as Figure 6 As shown, the device includes:

[0208] An acquisition module 710 is configured to acquire original spectrum data corresponding to the sample audio and a reference recognition result, wherein the reference recognition result indicates the original audio corresponding to the sample audio;

[0209] The masking module 720 is configured to perform element masking on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data;

[0210] A first determining module 730 is configured to input the fuzzy spectrum data into an audio feature extraction model to be trained, and have the audio feature extraction model to be trained perform feature extraction on the fuzzy spectrum data to obtain audio features corresponding to the sample audio;

[0211] a second determination module 740 configured to input the audio features into an audio recognition model to be trained, and have the audio recognition model to be trained recognize the audio features to obtain a predicted recognition result corresponding to the sample audio, wherein the predicted recognition result indicates a predicted audio corresponding to the sample audio;

[0212] The parameter adjustment module 750 is used to adjust the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained based on the benchmark recognition result and the predicted recognition result; if the preset end condition is met, the audio feature extraction model after parameter adjustment is determined as the trained audio feature extraction model.

[0213] In a possible implementation, the original spectrum data is spectrum data in matrix form;

[0214] The shielding module 720 is used to:

[0215] A first number of first elements and a second number of second elements are selected from the original spectrum data for masking to obtain blurred spectrum data corresponding to the original spectrum data, wherein each first element is located in a different column of the original spectrum data and each second element is located in a different row of the original spectrum data.

[0216] In a possible implementation, the shielding module 720 is configured to:

[0217] Performing multiple element masking processes on the original spectrum data to obtain multiple different blurred spectrum data corresponding to the original spectrum data;

[0218] The first determining module 730 is configured to:

[0219] The multiple different fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple different fuzzy spectrum data to obtain audio features corresponding to the sample audio.

[0220] In a possible implementation, the apparatus further includes a segmentation module configured to:

[0221] Based on the same segmentation method, each of the fuzzy spectrum data is segmented to obtain fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data;

[0222] The first determining module 730 is configured to:

[0223] The fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the fuzzy spectrum sub-data to obtain audio features corresponding to the sample audio.

[0224] In one possible implementation, the audio feature extraction model includes a block embedding layer, a first activation layer, a first normalization layer, a convolutional layer, a global average pooling layer, and a fully connected layer;

[0225] The first determining module 730 is configured to:

[0226] Using the block embedding layer, the first activation layer and the first normalization layer, sequentially process each fuzzy spectrum sub-data to obtain first feature information of each fuzzy spectrum sub-data;

[0227] The first convolution processing unit of the convolution layer processes the first feature information of each fuzzy spectrum sub-data to obtain target feature information of each fuzzy spectrum sub-data; and the second convolution processing unit of the convolution layer processes the target feature information of the plurality of fuzzy spectrum sub-data corresponding to each position area to obtain target feature information of each position area;

[0228] Based on the global average pooling layer, the fully connected layer and the target feature information of each position area, an audio feature corresponding to the sample audio is determined.

[0229] In a possible implementation, the first convolution processing unit of the convolution layer includes a depth-wise convolution layer, a second activation layer, and a second normalization layer;

[0230] The first determining module 730 is configured to:

[0231] Inputting the first feature information of each blurred spectrum sub-data into the depth-by-depth convolution layer, and performing convolution processing on the first feature information of each blurred spectrum sub-data by the depth-by-depth convolution layer to obtain the second feature information of each blurred spectrum sub-data;

[0232] inputting the second feature information of each fuzzy spectrum sub-data into the second activation layer, and performing nonlinear activation processing on the second feature information of each fuzzy spectrum sub-data by the second activation layer to obtain third feature information of each fuzzy spectrum sub-data;

[0233] inputting the third feature information of each fuzzy spectrum sub-data into the second normalization layer, and performing normalization processing on the third feature information of each fuzzy spectrum sub-data by the second normalization layer to obtain fourth feature information of each fuzzy spectrum sub-data;

[0234] For each of the fuzzy spectrum sub-data, the fourth feature information of the fuzzy spectrum sub-data and the first feature information of the fuzzy spectrum sub-data are combined and processed to obtain target feature information of the fuzzy spectrum sub-data.

[0235] In a possible implementation, the second convolution processing unit of the convolution layer includes a point-by-point convolution layer, a third activation layer, and a third normalization layer;

[0236] The first determining module 730 is configured to:

[0237] Inputting the target feature information of each blurred spectrum sub-data into the point-by-point convolution layer, and performing convolution processing on the target feature information of multiple blurred spectrum sub-data corresponding to each position area by the point-by-point convolution layer to obtain first feature information of each position area;

[0238] Inputting the first feature information of each position area into the third activation layer, and performing nonlinear activation processing on the first feature information of each position area by the third activation layer to obtain the second feature information of each position area;

[0239] The second feature information of each location area is input into the third normalization layer, and the third normalization layer performs normalization processing on the second feature information of each location area to obtain the target feature information of each location area.

[0240] In a possible implementation, there are multiple convolutional layers, and each convolutional layer outputs target feature information of each position area;

[0241] The first determining module 730 is configured to:

[0242] The target feature information of the same position area output by each convolutional layer is spliced ​​to obtain the combined feature information of each position area; the combined feature information of each position area is input into the global average pooling layer to obtain the pooled feature information of each position area; the pooled feature information of multiple position areas is input into the fully connected layer for connection to obtain the audio features corresponding to the sample audio.

[0243] In a possible implementation, the spectrum data includes at least one of MFCC feature information and CQT feature information.

[0244] It should be noted that the training device for the audio feature extraction model provided in the above embodiment only uses the division of the above functional modules as an example when training the audio feature extraction model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the audio feature extraction model provided in the above embodiment and the training method embodiment of the audio feature extraction model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0245] The embodiment of the present application provides an audio classification device, which can be the computer device in the above embodiment, such as Figure 8 As shown, the device includes:

[0246] A feature extraction module 810 is configured to input each audio in the audio library into a trained audio feature extraction model as described above, and perform feature extraction on each audio by the trained audio feature extraction model to obtain audio features corresponding to each audio;

[0247] A calculation module 820 is used to calculate the similarity between audio features corresponding to each two audios;

[0248] The grouping module 830 is configured to determine an audio group corresponding to at least one song based on the similarity, wherein the similarity between each two audios in the same audio group is greater than or equal to a similarity threshold.

[0249] It should be noted that the audio classification device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the classification of audio. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio classification device provided in the above embodiment and the audio classification method embodiment are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0250] Figure 9 9 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 900 may vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 901 and one or more memories 902. The memories 902 store at least one instruction, which is loaded and executed by the CPU 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which are not described in detail here.

[0251] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions. The instructions can be executed by a processor in a terminal to complete the training method of the audio feature extraction model in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (read-only memory), RAM (random access memory), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0252] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0253] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the "raw spectrum data corresponding to the sample audio" involved in this application was obtained with full authorization.

[0254] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an audio feature extraction model, characterized in that: The method comprises: Obtaining original spectrum data and a reference recognition result corresponding to the sample audio, wherein the reference recognition result is used to indicate the original audio corresponding to the sample audio; Performing element masking processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data, wherein the element masking processing comprises: selecting a first number of first elements and a second number of second elements in the original spectrum data for masking, each first element being located in a different column of the original spectrum data, and each second element being located in a different row of the original spectrum data; Inputting the fuzzy spectrum data into an audio feature extraction model to be trained, and having the audio feature extraction model to be trained perform feature extraction on the fuzzy spectrum data to obtain audio features corresponding to the sample audio; Inputting the audio features into an audio recognition model to be trained, and having the audio recognition model to be trained recognize the audio features to obtain a predicted recognition result corresponding to the sample audio, wherein the predicted recognition result is used to indicate a predicted audio corresponding to the sample audio; Based on the benchmark recognition result and the predicted recognition result, the parameters of the audio feature extraction model to be trained and the audio recognition model to be trained are adjusted; if the preset end condition is met, the audio feature extraction model after parameter adjustment is determined as the trained audio feature extraction model.

2. The method according to claim 1, characterized in that The performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data includes: Performing multiple element masking processes on the original spectrum data to obtain multiple different blurred spectrum data corresponding to the original spectrum data; Inputting the fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the fuzzy spectrum data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio, includes: The multiple different fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the multiple different fuzzy spectrum data to obtain audio features corresponding to the sample audio.

3. The method according to claim 2, characterized in that After performing element occlusion processing on the original spectrum data to obtain blurred spectrum data corresponding to the original spectrum data, the method further includes: Based on the same segmentation method, each of the fuzzy spectrum data is segmented to obtain fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data; The step of inputting the plurality of different fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the plurality of different fuzzy spectrum data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio includes: The fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data are input into the audio feature extraction model to be trained, and the audio feature extraction model to be trained performs feature extraction on the fuzzy spectrum sub-data to obtain audio features corresponding to the sample audio.

4. The method according to claim 3, characterized in that The audio feature extraction model includes a block embedding layer, a first activation layer, a first normalization layer, a convolutional layer, a global average pooling layer and a fully connected layer; The step of inputting the fuzzy spectrum sub-data corresponding to different position areas in each of the fuzzy spectrum data into the audio feature extraction model to be trained, and extracting features from the fuzzy spectrum sub-data by the audio feature extraction model to be trained to obtain audio features corresponding to the sample audio includes: Using the block embedding layer, the first activation layer and the first normalization layer, sequentially process each fuzzy spectrum sub-data to obtain first feature information of each fuzzy spectrum sub-data; The first convolution processing unit of the convolution layer processes the first feature information of each fuzzy spectrum sub-data to obtain target feature information of each fuzzy spectrum sub-data; and the second convolution processing unit of the convolution layer processes the target feature information of the plurality of fuzzy spectrum sub-data corresponding to each position area to obtain target feature information of each position area; Based on the global average pooling layer, the fully connected layer and the target feature information of each position area, an audio feature corresponding to the sample audio is determined.

5. The method according to claim 4, characterized in that The first convolution processing unit of the convolution layer includes a depth-wise convolution layer, a second activation layer, and a second normalization layer; The processing of the first feature information of each blurred spectrum sub-data by the convolution layer to obtain the target feature information of each blurred spectrum sub-data includes: Inputting the first feature information of each blurred spectrum sub-data into the depth-by-depth convolution layer, and performing convolution processing on the first feature information of each blurred spectrum sub-data by the depth-by-depth convolution layer to obtain the second feature information of each blurred spectrum sub-data; inputting the second characteristic information of each fuzzy spectrum sub-data into the second activation layer, and performing nonlinear activation processing on the second characteristic information of each fuzzy spectrum sub-data by the second activation layer to obtain third characteristic information of each fuzzy spectrum sub-data; inputting the third feature information of each fuzzy spectrum sub-data into the second normalization layer, and performing normalization processing on the third feature information of each fuzzy spectrum sub-data by the second normalization layer to obtain fourth feature information of each fuzzy spectrum sub-data; For each of the fuzzy spectrum sub-data, the fourth feature information of the fuzzy spectrum sub-data and the first feature information of the fuzzy spectrum sub-data are combined and processed to obtain target feature information of the fuzzy spectrum sub-data.

6. The method according to claim 4, characterized in that The second convolution processing unit of the convolution layer includes a point-by-point convolution layer, a third activation layer and a third normalization layer; The convolution layer processes the target feature information of the multiple blurred spectrum sub-data corresponding to each position area to obtain the target feature information of each position area, including: Inputting the target feature information of each blurred spectrum sub-data into the point-by-point convolution layer, and performing convolution processing on the target feature information of the multiple blurred spectrum sub-data corresponding to each position area by the point-by-point convolution layer to obtain the first feature information of each position area; Inputting the first feature information of each position area into the third activation layer, and performing nonlinear activation processing on the first feature information of each position area by the third activation layer to obtain the second feature information of each position area; The second feature information of each location area is input into the third normalization layer, and the third normalization layer performs normalization processing on the second feature information of each location area to obtain the target feature information of each location area.

7. The method according to claim 4, characterized in that There are multiple convolutional layers, and each convolutional layer outputs target feature information of each position area; The determining the audio features of the sample audio based on the global average pooling layer, the fully connected layer, and the target feature information of each location area includes: splicing the target feature information of the same position area output by each convolutional layer to obtain combined feature information of each position area; The combined feature information of each of the position areas is input into the global average pooling layer to obtain the pooled feature information of each of the position areas; the pooled feature information of multiple position areas is input into the fully connected layer for connection to obtain the audio features corresponding to the sample audio.

8. An audio classification method, characterized in that: The method comprises: Inputting each audio in the audio library into the trained audio feature extraction model according to any one of claims 1 to 7, respectively, and performing feature extraction on each audio by the trained audio feature extraction model to obtain audio features corresponding to each audio; Calculate the similarity between the audio features corresponding to each two audios; An audio group corresponding to each audio is determined based on the similarity, wherein the similarity between every two audios in the same audio group is greater than or equal to a similarity threshold.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operation performed by the training method of the audio feature extraction model according to any one of claims 1 to 7 or the audio classification method according to claim 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the training method of the audio feature extraction model according to any one of claims 1 to 7 or the audio classification method according to claim 8.

11. A computer program product, characterized in that The computer program product includes at least one instruction, which is loaded and executed by a processor to implement the operations performed by the training method of the audio feature extraction model according to any one of claims 1 to 7 or the audio classification method according to claim 8.

Images