Audio classification model training method and classification method based on spectrum position coding Transform, storage medium and equipment

By introducing spectral positional encoding Transformer and relative positional encoding techniques into the audio classification model and optimizing attention weight calculation, the problem of insufficient ability of existing models to capture positional relationships is solved, and the accuracy of audio classification is significantly improved.

CN118918919APending Publication Date: 2024-11-08HARBIN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410983886.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing audio classification models are weak in capturing positional relationships, resulting in insufficient classification accuracy.

Method used

An audio classification model based on spectral position coding Transformer is adopted. By introducing a multi-head self-attention module and relative position coding technology, attention weight calculation is optimized and local structural features are extracted.

Benefits of technology

It significantly improved the accuracy of audio classification, especially on the ESC-50 dataset, where the accuracy increased from 95.7% to 96.65%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918919A_ABST
    Figure CN118918919A_ABST
Patent Text Reader

Abstract

The invention discloses an audio classification model training method and classification method based on spectrum position coding Transform, a storage medium and equipment, and belongs to the technical field of voice signal processing. In order to solve the problem that an attention mechanism is weak in position relation capturing capability when an existing model is used for audio classification, an audio classification model based on a frequency spectrum position code Transform is used for processing a Mel-frequency spectrum feature map of audio data; a spectrum block embedding layer block in the audio classification model cuts spectrum features into M rows and N columns of spectrum blocks, the spectrum blocks are flattened into a one-dimensional sequence to obtain a feature sequence, then a multi-head self-attention module of a Transform layer is fused with spectrum block relative position offset values, and the spectrum block relative position offset values are in one-to-one correspondence with the relative positions of the two spectrum blocks; and predicting according to the output of the Transform layer to obtain a classification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal processing, and in particular relates to an audio classification model training method, a classification method, a storage medium and a device. Background Art

[0002] Audio classification is a task that uses machine learning and deep learning techniques to automatically identify and classify the information contained in audio signals. It has applications in many fields, including music retrieval, environmental sound monitoring, speech recognition, sentiment analysis, etc. With the massive growth of digital media, audio content has become increasingly rich, including music, podcasts, sound tracks of movies and TV shows, environmental sounds, etc. Manually tagging and classifying these audio data is not only inefficient but also costly. Therefore, the demand for automatic audio classification is growing. In the era of globalization and digitalization, the generation and dissemination of audio content is rapid, and users are faced with a large amount of audio information from different sources. Audio classification technology can enable users to quickly find valuable information in a large amount of audio content, improve work efficiency and quality of life. On the technical level, audio classification usually relies on traditional audio processing technology and modern machine learning and deep learning methods. The processing flow generally includes steps such as audio preprocessing, feature extraction, model training, and classification prediction. With the advancement of deep learning technology, people continue to develop complex models for audio, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer models, which have shown excellent performance in audio classification tasks.

[0003] With the emergence of deep neural networks, audio classification research has shifted from models based on handcrafted features to end-to-end models that directly map audio spectrograms to corresponding labels over the past decade. Existing recognition models are inadequate in capturing positional relationships due to the original attention mechanism, which affects the classification accuracy of classification models used for audio classification. Summary of the invention

[0004] The present invention aims to solve the problem that the attention mechanism of the existing model is weak in capturing positional relationships during audio classification, and the resulting problem that the model classification accuracy needs to be further improved.

[0005] A method for training an audio classification model based on a spectrum position encoding Transformer, comprising:

[0006] Step 1: For the audio data in the audio data set, obtain the corresponding Mel-spectrum feature map, record it as the spectrum feature, and use it as a training set;

[0007] Step 2: training an audio classification model based on a spectrum position encoding Transformer, wherein the audio classification model comprises at least a spectrum slicing embedding layer and a plurality of Transformer encoder layers;

[0008] The training process includes the following steps:

[0009] Step 2-1: Send a batch of spectrum features to the spectrum block embedding layer to cut it into M rows and N columns of spectrum blocks, and record the row and column positions of the spectrum blocks; then flatten it into a one-dimensional sequence to obtain the feature sequence X n =[x 1 ′,x 2 ′,...,x i ′,...,x′ K ]; where x i ′ is the feature sequence X n The vector corresponding to the i-th spectrum block after the spectrum is cut, K = M × N is the number of vectors after the spectrum is cut;

[0010] Step 2-2: transform the feature sequence x 1 ′,x′ 2 ,...,x i ′,...,x′ K It is fed into multiple Transformer encoder layers, each of which includes a multi-head self-attention module and a feedforward neural network to integrate the features to form features X after being processed by the attention mechanism. n ';

[0011] The multi-head self-attention module is as follows:

[0012]

[0013] Where Q, K, V are the query, key, and value matrices in the attention mechanism; B represents the spectrum block x i The relative position offset values ​​correspond to the relative position offset table, which is set as a learnable parameter during the training process and updated through the model training process. When the model is trained, the offset table B composed of the offset values ​​is obtained; spectrum block x i The relative position offset value corresponds one-to-one to the relative position of the two spectrum blocks;

[0014] Step 2-3: For feature sequence X n 'Get the prediction results;

[0015] Step 2-4: Repeat steps 2-1, 2-2, and 2-3 for the spectral features corresponding to all batches of samples in the training set until the preset maximum number of iterations is reached, thereby obtaining a trained audio classification model based on spectral position encoding Transformer.

[0016] Furthermore, the spectrum block x i The one-to-one correspondence between the relative position offset value and the relative position of the two spectrum blocks is determined by the following method:

[0017] The spectrum slice embedding layer records the row and column positions of the spectrum blocks. For K = M × N spectrum blocks, for two spectrum blocks, recorded as the i-th and j-th slices, the relative positions are described by their row offset δx and column offset δy:

[0018] δx=x j -x i ,δy=y i -y j (3)

[0019] Among them, x i ,y i and x j ,y j are the row and column coordinates of slice i and slice j in the spectrum slice, respectively. Each coordinate offset (δx, δy) needs to be converted into a single index value Index;

[0020] The (δx, δy) correspondence is implemented through the Index index; specifically, the bias value B is retrieved from the bias table using the following formula:

[0021] B Index =B[Index] (5)

[0022] Among them, B[Index] represents the Index position offset in the offset table, B Index The bias value corresponding to the position is set as a learnable parameter during the training process and updated through the model training process. When the model is trained, the bias table B composed of the bias values ​​is obtained.

[0023] Furthermore, each coordinate offset (δx, δy) needs to be converted into a single index value Index = (δx + M - 1) × (2N - 1) + (δy + N - 1).

[0024] Furthermore, the audio classification model further includes a mean pooling layer, a fully connected layer and a softmax layer after the Transformer encoder layer; in step 2-3, for the feature sequence X n 'When getting the prediction result, first convert the feature sequence X n'It is then sent to the mean pooling layer to generate the speech feature vector μ, and finally μ is input into the fully connected layer and the softmax layer for prediction to produce the final prediction result.

[0025] An audio classification method based on spectral position encoding Transformer, for audio data to be classified, obtains the corresponding Mel spectrum feature map, recorded as spectrum feature, and then uses an audio classification model based on spectral position encoding Transformer for recognition;

[0026] An audio classification model processing process based on spectral position encoding Transformer includes the following steps:

[0027] Step 2-1: Send the spectrum features to the spectrum block embedding layer to cut them into M rows and N columns of spectrum blocks, and record the row and column positions of the spectrum blocks; then flatten them into a one-dimensional sequence to obtain the feature sequence X n =[x 1 ′,x 2 ′,...,x i ′,...,x′ K ]; where x i ′ is the feature sequence X n The vector corresponding to the i-th spectrum block after the spectrum is cut, K = M × N is the number of vectors after the spectrum is cut;

[0028] Step 2-2: transform the feature sequence x 1 ′,x′ 2 ,...,x i ′,...,x′ K It is fed into multiple Transformer encoder layers, each of which includes a multi-head self-attention module and a feedforward neural network to integrate the features to form features X after being processed by the attention mechanism. n ';

[0029] The multi-head self-attention module is as follows:

[0030]

[0031] Where Q, K, V are the query, key, and value matrices in the attention mechanism; B represents the spectrum block x i The relative position offset values ​​correspond to the relative position offset table, spectrum block x i The relative position offset value corresponds to the relative position of the two spectrum blocks one by one, and the relative position offset table is determined during the training process of an audio classification model based on a spectrum position encoding Transformer;

[0032] Step 2-3: For feature sequence X n'Get the prediction results.

[0033] Furthermore, the spectrum block x i The one-to-one correspondence between the relative position offset value and the relative position of the two spectrum blocks is determined by the following method:

[0034] The spectrum slice embedding layer records the row and column positions of the spectrum blocks. For K = M × N spectrum blocks, for two spectrum blocks, recorded as the i-th and j-th slices, the relative positions are described by their row offset δx and column offset δy:

[0035] δx=x j -x i ,δy=y i -y j (3)

[0036] Among them, x i ,y i and x j ,y j are the row and column coordinates of slice i and slice j in the spectrum slice, respectively. Each coordinate offset (δx, δy) needs to be converted into a single index value Index;

[0037] The (δx, δy) correspondence is implemented through the Index index; specifically, the bias value B is retrieved from the bias table using the following formula:

[0038] B Index =B[Index] (5)

[0039] Among them, B[Index] represents the Index position offset in the offset table, B Index Indicates the offset value of the corresponding position.

[0040] Furthermore, each coordinate offset (δx, δy) needs to be converted into a single index value Index = (δx + M - 1) × (2N - 1) + (δy + N - 1).

[0041] Furthermore, the audio classification model further includes a mean pooling layer, a fully connected layer and a softmax layer after the Transformer encoder layer; in step 2-3, for the feature sequence X n 'When getting the prediction result, first convert the feature sequence X n 'It is then sent to the mean pooling layer to generate the speech feature vector μ, and finally μ is input into the fully connected layer and the softmax layer for prediction to produce the final prediction result.

[0042] A computer storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the audio classification method based on spectrum position coding Transformer.

[0043] An audio classification device based on spectral position coding Transformer, the device includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the audio classification method based on spectral position coding Transformer.

[0044] Beneficial effects:

[0045] The present invention adopts an innovative attention module position encoding technology, which optimizes the calculation of attention weight by considering the relative position difference between elements rather than the absolute position of a single element, thereby effectively extracting local structural features, and then realizing an audio classification model based on spectral position encoding Transformer, which can effectively improve the classification accuracy of the audio spectrum Transformer network. For the ESC-50 dataset, the highest accuracy of the present invention is 96.65%. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the present invention.

[0047] Figure 2 This is the network architecture diagram of the audio classification model based on the spectral position encoding Transformer.

[0048] Figure 3 To compare the performance of the present invention and the comparative method on the ESC-50 dataset, the performance is evaluated using classification accuracy (ACC).

[0049] Figure 4 It is a curve diagram of the accuracy change of the position encoding method and the non-position encoding method proposed in the present invention at different training times;

[0050] Figure 5 The present invention proposes a loss variation curve diagram of the position encoding method and the non-position encoding method at different training times. DETAILED DESCRIPTION

[0051] The present invention proposes a spectral position encoding Transformer, which is a convolution-free model purely based on the attention mechanism, and is designed to directly process the audio spectrogram and effectively capture long-distance global context information. The spectral position encoding Transformer proposed in the present invention encodes the relative position relationship of elements in the spectrogram by introducing an innovative attention-based position encoding technology, thereby optimizing the calculation of attention weights and extracting local structural features. This method effectively makes up for the shortcomings of the original attention mechanism in capturing positional relationships and significantly improves the accuracy of audio classification. The following is an explanation in conjunction with a specific implementation method.

[0052] Specific implementation method 1: Combination Figure 1 , Figure 2 To explain this embodiment,

[0053] This embodiment is an audio classification method based on spectral position encoding Transformer, including:

[0054] Step 1: Collect different types of audio data sets, and preprocess and enhance the audio data; for unified processing, cut the enhanced audio data sets to the same length.

[0055] Collect various audios and preprocess the speech data, and then perform data enhancement, which includes Mixup and frequency / time masking. Mixup creates new training samples by linear interpolation in the feature space, aiming to enhance the data set and enable the model to learn smoother decision boundaries. Frequency / time masking randomly applies masks on the time and frequency dimensions of the spectrogram to generate new training samples to improve the generalization and robustness of the model.

[0056] Subsequently, the audio is processed using a Hamming window Fourier transform, which is calculated every 10 milliseconds to generate a 128-dimensional Mel-spectrum feature map, which is recorded as the spectrum feature and used as a training set.

[0057] Step 2: Build a deep learning model and input the spectral features obtained in step 1 into the audio spectrum Transformer model for training. Use the cross entropy loss function to update the network parameters until the preset maximum number of iterations is reached to obtain a trained network model. The audio spectrum Transformer model includes a spectrum block embedding layer, multiple Transformer encoder layers, a mean pooling layer, a fully connected layer, and a softmax layer. In the spectrum block embedding layer, a block of size (m, m) is used to divide the spectrum feature map into K blocks, which are flattened into a one-dimensional sequence and input into the Transformer encoder layer.

[0058] The specific processing process of the audio spectrum Transformer model includes the following steps:

[0059] Step 2-1: Send the spectrum features of a batch into the spectrum block embedding layer to cut it into M rows and N columns of spectrum blocks, and then flatten it into a one-dimensional sequence to obtain the feature sequence X n ;

[0060] X n =[x 1 ′,x 2 ′,...,x i ′,...,x′ K ] (1)

[0061] Among them, x i ′∈R F is the feature sequence X n The vector corresponding to the i-th spectrum block after the block is cut, F is the dimension of the speech feature, and K = M × N is the number of vectors after the spectrum is cut;

[0062] Step 2-2: transform the feature sequence x 1 ′,x′ 2 ,...,x i ′,...,x′ K It is fed into multiple Transformer encoder layers, each of which includes a multi-head self-attention module and a feedforward neural network to integrate the features and form the feature X processed by the attention mechanism. n '.

[0063] The present invention improves on the basis of the multi-head self-attention module of the traditional Transformer encoder. The multi-head self-attention module of the present invention introduces the proposed position encoding information. The specific calculation formula is as follows:

[0064]

[0065] Among them, Attention(Q, K, V) is the weighted sum of the attention weights on the values, Q, K, V are the query, key, and value matrices in the attention mechanism, and B is the relative position bias. The specific explanation is: In order to use the attention module, the spectral feature map is flattened into a one-dimensional sequence X after being sent to the spectral block embedding layer. n, since the relative position relationship between the feature sequences in the original space is lost, and this position relationship is indispensable in the attention mechanism. For this reason, a relative position encoding method is proposed: there are K = M × N (M rows and N columns) spectrum blocks. For two spectrum blocks, denoted as the i-th and j-th blocks, their relative positions in the spectrum blocks can be described by their row offset δx and column offset δy:

[0066] δx=x j -x i ,δy=y i -y j (3)

[0067] Among them, x i ,y i and x j ,y j are the row and column coordinates of slice i and slice j in the spectrum slice respectively (row coordinates belong to [0, M-1], column coordinates belong to [0, N-1]), and each coordinate offset (δx, δy) needs to be converted to a single index value.

[0068] First, convert the row offset δx and column offset δy to non-negative numbers, and add the maximum row offset M-1 to δx and the maximum column offset N-1 to δy (M rows and N columns mentioned above) to make δx and δy non-negative. The index is calculated using the following formula:

[0069] Index=(δx+M-1)×(2N-1)+(δy+N-1) (4)

[0070] Here, (δx+M-1) and (δy+N-1) are the above-mentioned addition of the maximum row offset and column offset. The key to mapping from the two-dimensional relative coordinates (δx, δy) to the one-dimensional index is to ensure that each pair of coordinate differences (δx, δy) can uniquely correspond to an index value. After multiplying by 2N-1 here, the index will not be repeated, because it is the maximum number that the column offset can reach plus 1 (that is, the maximum number that (δy+N-1) can reach plus 1), ensuring that the offset value of each row (when δx increases) does not overlap with all possible values ​​of the previous row (determined by δy). That is, (δy+N-1) can change within [0, 2N-2], and each time (δx+M-1) increases by 1, the Index increases by 2N-1, which just exceeds the maximum range of (δy+N-1) changes, thereby avoiding any duplication. Therefore, the Index realizes that each pair of coordinate differences is mapped to a unique index value. At this point, a unique one-dimensional index Index has been calculated for each relative position coordinate difference (δx, δy). The following describes how to use this index to retrieve the offset value from a pre-trained relative position offset table B.

[0071] The bias table B is initialized before model training. It is initialized to all 0s and has a length of (2N-1)×(2M-1). The bias table B is set as a learnable parameter in the model. Its value is updated in the model training in an end-to-end manner. This length is the sum of all possible relative positions. Each bias value corresponds to a specific position difference (δx, δy). This correspondence is implemented through the Index index. Specifically, the bias value can be retrieved from the bias table using the following formula:

[0072] B Index =B[Index] (5)

[0073] Where Index = (δx+M-1)×(2N-1)+(δy+N-1), and each Index corresponds to a unique set of (δx, δy); B[Index] represents the index position in the bias table, B Index Indicates the bias value of the corresponding position. It is set as a learnable parameter during the training process. Its value is updated in the model training in an end-to-end manner. When the model is trained, the bias table B composed of the bias values ​​is obtained. In the subsequent actual prediction, the bias table B is directly used to determine the bias value of the corresponding position according to the index Index.

[0074] Step 2-3: The feature sequence X n 'It is then sent to the mean pooling layer to generate the speech feature vector μ, and finally μ is input into the fully connected layer and the softmax layer for prediction to produce the final prediction result.

[0075] Step 2-4: Repeat steps 2-1, 2-2, and 2-3 for the spectral features corresponding to all batches of samples in the training set until the preset maximum number of iterations is reached, thereby obtaining a fully trained audio spectrum Transformer model.

[0076] Step 3: Perform spectral feature conversion on the audio data to be tested to obtain a Mel-spectrogram feature map, and input it into the trained audio spectrum Transformer model to obtain the category of the test audio.

[0077] Example:

[0078] The technical solution adopted by the present invention is an audio classification method based on audio spectrum Transformer, which comprises the following steps:

[0079] Step 1: Collect different types of audio data sets, and perform preprocessing and data enhancement on the audio data. For unified processing, cut the enhanced audio data sets to the same length and use them as training sets.

[0080] Subsequently, the audio is processed using a Hamming window Fourier transform, which is calculated every 10 milliseconds to generate a 128-dimensional Mel-spectrogram feature map for use in the training set.

[0081] Step 2, build a deep learning model, and input the features obtained in step 1 into the audio spectrum Transformer model for training. The audio spectrum Transformer model includes a spectrum block embedding layer, multiple Transformer encoder layers, a mean pooling layer, and a fully connected layer. In the spectrum block embedding layer, the spectrum feature map is divided into K blocks using blocks of size (m, m) in the spectrum block embedding layer. These blocks are flattened into a one-dimensional sequence and input into the Transformer encoder layer. When training the model, the cross entropy loss function is used to update the network parameters until the preset maximum number of iterations is reached to obtain a trained network model. In this embodiment, the maximum number of iterations is set to 50 times, and the deep learning model learning rate is 0.00001. After the iteration is completed, the deep learning model is obtained.

[0082] Step 3: Perform spectral feature conversion on the audio data to be tested and input it into the trained deep learning model to obtain the category of the test audio and use accuracy (ACC) to evaluate the model performance.

[0083] The performance of this invention is verified on the ESC-50 dataset, which contains 50 different types of audio. The performance evaluation index is the classification accuracy (ACC), and the larger the value, the better the performance. In the audio classification task, the lowest average loss achieved by this method is 3.396, and the highest accuracy is 96.65%. Figure 3 As shown in Figure 2, compared with EAT-S, ACDNet, GCL, EARNN-2-5, sepTr, and XDC, the performance on the ESC-50 dataset is significantly improved, and the highest accuracy of the present invention is 96.65%. Figure 4 As shown in FIG. 1 , compared with the same system without the position encoding of the present invention, the accuracy rate is improved from 95.7% to 96.65%. Figure 5 As shown, the test set loss of the present invention decreases faster and the model converges more quickly. The above results show that, through experimental verification, the deep learning model using relative position encoding shows faster convergence speed and higher classification accuracy on the ESC-50 dataset, proving the effectiveness and superiority of this method. Specific implementation method 2:

[0085] This implementation is a computer storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the audio classification method based on spectral position encoding Transformer.

[0086] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0088] This embodiment is an audio classification device based on spectral position coding Transformer, the device includes a processor and a memory. It should be understood that including any device including a processor and a memory described in the present invention, the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions;

[0089] The memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the audio classification method based on spectrum position coding Transformer.

[0090] Those skilled in the art will appreciate that at least one instruction stored is a computer program product corresponding to the method or system. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0091] The present application is described with reference to the flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present application, and may also be used for corresponding devices. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0095] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0096] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for training an audio classification model based on a spectral position encoding Transformer, characterized in that: include: Step 1: For the audio data in the audio data set, obtain the corresponding Mel-spectrum feature map, record it as the spectrum feature, and use it as a training set; Step 2: training an audio classification model based on a spectrum position encoding Transformer, wherein the audio classification model comprises at least a spectrum slicing embedding layer and a plurality of Transformer encoder layers; The training process includes the following steps: Step 2-1: Send a batch of spectrum features to the spectrum block embedding layer to cut it into M rows and N columns of spectrum blocks, and record the row and column positions of the spectrum blocks; then flatten it into a one-dimensional sequence to obtain the feature sequence X n =[x1′,x2′,...,x i ′,...,x′ K ]; where x i ′ is the feature sequence X n The vector corresponding to the i-th spectrum block after the spectrum is cut, K = M × N is the number of vectors after the spectrum is cut; Step 2-2: Substitute the feature sequence x1′, x′2, ..., x i ′,...,x′ K It is fed into multiple Transformer encoder layers, each of which includes a multi-head self-attention module and a feedforward neural network to integrate the features to form features X after being processed by the attention mechanism. n '; The multi-head self-attention module is as follows: Where Q, K, V are the query, key, and value matrices in the attention mechanism; B represents the spectrum block x i The relative position offset values ​​correspond to the relative position offset table, which is set as a learnable parameter during the training process and updated through the model training process. When the model is trained, the offset table B composed of the offset values ​​is obtained; spectrum block x i The relative position offset value corresponds one-to-one to the relative position of the two spectrum blocks; Step 2-3: For feature sequence X n 'Get the prediction results; Step 2-4: Repeat steps 2-1, 2-2, and 2-3 for the spectral features corresponding to all batches of samples in the training set until the preset maximum number of iterations is reached, thereby obtaining a trained audio classification model based on spectral position encoding Transformer.

2. According to the audio classification model training method based on spectral position encoding Transformer according to claim 1, it is characterized in that: Spectrum Block x i The one-to-one correspondence between the relative position offset value and the relative position of the two spectrum blocks is determined by the following method: The spectrum slice embedding layer records the row and column positions of the spectrum blocks. For K = M × N spectrum blocks, for two spectrum blocks, recorded as the i-th and j-th slices, the relative positions are described by their row offset δx and column offset δy: δx=x j -x i ,δy=y i -y j (3) Among them, x i ,y i and x j ,y j are the row and column coordinates of slice i and slice j in the spectrum slice, respectively. Each coordinate offset (δx, δy) needs to be converted into a single index value Index; The (δx, δy) correspondence is implemented through the Index index; specifically, the bias value B is retrieved from the bias table using the following formula: B Index =B[Index] (5) Among them, B[Index] represents the Index position offset in the offset table, B Index The bias value corresponding to the position is set as a learnable parameter during the training process and updated through the model training process. When the model is trained, the bias table B composed of the bias values ​​is obtained.

3. According to claim 2, the audio classification model training method based on spectral position encoding Transformer is characterized in that: Each coordinate offset (δx, δy) needs to be converted into a single index value Index = (δx + M - 1) × (2N - 1) + (δy + N - 1).

4. The audio classification model training method based on spectral position encoding Transformer according to any one of claims 1 to 3, characterized in that: The audio classification model further includes a mean pooling layer, a fully connected layer and a softmax layer after the Transformer encoder layer; in step 2-3, for the feature sequence X n 'When getting the prediction result, first convert the feature sequence X n 'It is then sent to the mean pooling layer to generate the speech feature vector μ, and finally μ is input into the fully connected layer and the softmax layer for prediction to produce the final prediction result.

5. An audio classification method based on spectral position encoding Transformer, characterized in that: For the audio data to be classified, the corresponding Mel-spectrum feature map is obtained and recorded as the spectrum feature, and then an audio classification model based on the spectrum position encoding Transformer is used for recognition; An audio classification model processing process based on spectral position encoding Transformer includes the following steps: Step 2-1: Send the spectrum features to the spectrum block embedding layer to cut them into M rows and N columns of spectrum blocks, and record the row and column positions of the spectrum blocks; then flatten them into a one-dimensional sequence to obtain the feature sequence X n =[x1′,x2′,...,x i ′,...,x′ K ]; where x i ′ is the feature sequence X n The vector corresponding to the i-th spectrum block after the spectrum is cut, K = M × N is the number of vectors after the spectrum is cut; Step 2-2: Substitute the feature sequence x1′, x′2, ..., x i ′,...,x′ K It is fed into multiple Transformer encoder layers, each of which includes a multi-head self-attention module and a feedforward neural network to integrate the features to form features X after being processed by the attention mechanism. n '; The multi-head self-attention module is as follows: Where Q, K, V are the query, key, and value matrices in the attention mechanism; B represents the spectrum block x i The relative position offset values ​​correspond to the relative position offset table, spectrum block x i The relative position offset value corresponds to the relative position of the two spectrum blocks one by one, and the relative position offset table is determined during the training process of an audio classification model based on a spectrum position encoding Transformer; Step 2-3: For feature sequence X n 'Get the prediction results.

6. The audio classification method based on spectral position coding Transformer according to claim 5, characterized in that: Spectrum Block x i The one-to-one correspondence between the relative position offset value and the relative position of the two spectrum blocks is determined by the following method: The spectrum slice embedding layer records the row and column positions of the spectrum blocks. For K = M × N spectrum blocks, for two spectrum blocks, recorded as the i-th and j-th slices, the relative positions are described by their row offset δx and column offset δy: δx=x j -x i ,δy=y i -y j (3) Among them, x i ,y i and x j ,y j are the row and column coordinates of slice i and slice j in the spectrum slice, respectively. Each coordinate offset (δx, δy) needs to be converted into a single index value Index; The (δx, δy) correspondence is implemented through the Index index; specifically, the bias value B is retrieved from the bias table using the following formula: B Index =B[Index] (5) Among them, B[Index] represents the Index position offset in the offset table, B Index Indicates the offset value of the corresponding position.

7. The audio classification method based on spectral position coding Transformer according to claim 6, characterized in that: Each coordinate offset (δx, δy) needs to be converted into a single index value Index = (δx + M - 1) × (2N - 1) + (δy + N - 1).

8. The audio classification method based on spectral position coding Transformer according to any one of claims 5 to 7, characterized in that: The audio classification model further includes a mean pooling layer, a fully connected layer and a softmax layer after the Transformer encoder layer; in step 2-3, for the feature sequence X n 'When getting the prediction result, first convert the feature sequence X n 'It is then sent to the mean pooling layer to generate the speech feature vector μ, and finally μ is input into the fully connected layer and the softmax layer for prediction to produce the final prediction result.

9. A computer storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the audio classification method based on spectral position coding Transformer as described in any one of claims 5 to 7.

10. An audio classification device based on spectral position encoding Transformer, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement an audio classification method based on spectral position coding Transformer as described in any one of claims 5 to 7.