A speaker recognition method based on dual attention mechanism dynamic convolution

Abstract

The present invention claims a dynamic convolution speaker recognition method based on channel-spatial attention, which includes the following steps: S1, cutting a sound signal into multiple segments and extracting acoustic features of the sound signal; S2, using a dual attention method of channel attention and spatial attention fusion to extract the attention weight matrix in order to solve the problem of insufficient information in the attention weight matrix in static convolution and general dynamic convolution methods; S3, fusing the channel attention and spatial attention weight matrices as the attention weight matrix of dynamic convolution and adding them with multiple static convolution kernels according to weights to obtain the final dynamic convolution kernel. S4, convolving the input speech data with the dynamic convolution kernel to obtain the speaker's speech frame-level features.

Description

Technical Field

[0001] The invention belongs to the field of speech recognition and is characterized by a speaker recognition method based on dual attention mechanism dynamic convolution. Background Art

[0002] Speech is an effective way for humans to communicate and obtain information. Since each speaker has a unique pronunciation, speaking habits, and throat size, the speaker's voice contains a wealth of personal information. This makes it possible for a computer to identify a speaker through his or her voice. This topic studies automatic speaker recognition (ASR), which is to identify the speaker's identity through the speaker's characteristics contained in a piece of speech.

[0003] Speaker recognition technology is an important branch of speech recognition technology, and is also an important module for realizing human-computer interaction. It has a wide range of applications in many fields. For example, the speaker's identity can be identified during a call to prevent fraud. This technology can be used for remote identity recognition in banking systems. In personal smart devices or smart homes, it can be unlocked and commands can be issued without touching or requiring close-range facial recognition, making human-computer interaction more convenient and intelligent. However, speaker recognition technology is difficult to cope with the complex and changing environment in actual engineering applications, and incorrect judgment results may bring more serious consequences to the speaker himself. In some areas with high security requirements, such as identity confirmation in banking systems and voiceprint locks for private smart devices, if the device collects incorrect commands, it will have a serious impact on personal privacy, life and property safety.

[0004] Although there have been many very successful commercial products, the performance of speaker recognition systems in actual applications is still far behind human auditory recognition capabilities. On the one hand, deep learning is data-driven, and human speech has very diverse variations. People of different ages, genders, languages, and different emotions and speech speeds of the same person will cause changes in speech signals, thus affecting the stability of the speaker recognition system. On the other hand, because the speaker information in the speaker's speech signal is mixed with various interference information, such as interference from channel differences between different acquisition devices, environmental noise in different scenarios, etc., the stability of the speaker recognition system is reduced.

[0005] Therefore, how to extract more discriminative speaker features and how to make the extracted speaker features more robust in speaker recognition tasks in various scenarios are the current main research directions.

[0006] Invention patent CN113763965A proposes a speaker recognition method with multiple attention feature fusion. The invention performs weighted fusion on multiple branch features of input data through multiple attention feature fusion. The multiple attention feature fusion includes a spatial attention mechanism and a channel attention mechanism, which enhances the effective information in each branch and obtains more robust speaker recognition performance. However, the attention fusion method proposed in the invention only performs feature fusion on multiple extracted data, and the convolution method used in the feature extraction process is still a static convolution with fixed parameters. It is still not flexible when extracting features from input data, and cannot capture the phoneme change information between multiple different input speech data. Therefore, the present invention adopts a dual attention dynamic convolution method combining spatial and channel attention. Dynamic convolution can overcome the defect of fixed static convolution parameters, and it can dynamically adjust the parameters of the convolution kernel according to the different inputs, so that the phoneme change information between different input data can be flexibly captured during the feature extraction process. Summary of the invention

[0007] The present invention aims to solve the above problems of the prior art. A speaker recognition method based on dual attention mechanism dynamic convolution is proposed. The technical solution of the present invention is as follows:

[0008] A speaker recognition method based on dual attention mechanism dynamic convolution, comprising the following steps:

[0009] S1, obtaining a speaker's voice signal and cutting the voice signal into multiple segments, extracting the Melp features of the voice signal;

[0010] S2, uses the dual attention method of channel attention and spatial attention fusion to extract the attention weight matrix;

[0011] S3, the channel attention and spatial attention weight matrices are fused as the attention weight matrix of dynamic convolution and added to multiple static convolution kernels according to the weights to obtain the final dynamic convolution kernel;

[0012] S4, convolving the input speech data with the dynamic convolution kernel to obtain frame-level feature data of the speaker's speech signal. The frame-level data is used as the input of the subsequent fully connected layer for the speaker classification network.

[0013] Furthermore, the step S1 acquires the speaker's voice signal and cuts the voice signal into multiple segments to extract the Melp features of the voice signal, specifically including: dividing the voice signal into frames, windowing, and Fourier transforming to obtain the Melp features;

[0014] S21: Set the length of the Hamming window to 25ms and the step size to 10ms;

[0015] S22: Set the dimension of extracting frame features to 40 dimensions.

[0016] Furthermore, S2 uses a dual attention method of channel attention and spatial attention fusion to extract the attention weight matrix, and adds each element thereof accordingly. The weight matrix generation model establishment step includes:

[0017] S31:M avg =conv(c in ,k,1)(AvgPool(X)), X is the input data, AvgPool is the global average pooling layer, conv is the convolutional layer, and the parameter c in is the input channel of the data, k is the number of output channels, which is also the number of static convolution kernels set, and 1 is the convolution kernel size; after global average pooling of the input data, convolution is performed to obtain the attention weight parameter M of each channel avg ;

[0018] S32:M channel =conv(k,k,1)(ReLU(M avg )) The attention weight parameter matrix M avg After activation by the nonlinear function ReLU, a k×k convolution is performed to obtain the final channel attention parameter matrix M channel ;

[0019] S33: After the input data passes through the maximum pooling layer (MaxPooling) and the global average pooling layer (AvgPooling), the dimension is reduced to 2×H×W, where H is the dimension of each frame of speech data and W is the number of frames of speech data. The multi-frame data is convolved to reduce its dimension to 1×H×W, and M is obtained. s Matrix M s =conv(c in ,1,1)(MaxPool,AvgPool(X))),M s The matrix is ​​the weight parameter matrix for each position in space;

[0020] S34: M s After the matrix is ​​flattened, its dimension is b×T, where b is the batch size of the input data and T is the length of the flattened data. In the convolution kernel parameters, k is the number of static convolution kernels, 1 is the convolution kernel size, and M is the convolution kernel size. F =conv(T,k,1)(Flatten(M s )), M F is the spatial weight matrix without nonlinear activation;

[0021] S35: M FAfter the matrix is ​​activated by the ReLu nonlinear function, it is convolved by k×k to obtain the final spatial attention weight parameter matrix M spatial , M spatial =conv(k,k,1)(ReLu(M F ).

[0022] Furthermore, step S3 fuses the channel attention and spatial attention weight matrices as the attention weight matrix of the dynamic convolution and adds them to the multiple static convolution kernels according to the weights to obtain the final dynamic convolution kernel, which specifically includes:

[0023] The dynamic convolution kernel generation model establishment step includes:

[0024] S41: First, the obtained channel attention and spatial attention parameters are added element-wise. Among them, M cs That is, the dual attention dynamic convolution kernel that integrates channel and spatial attention;

[0025] S42: M cs Perform a k×k convolution, and then normalize the output data to the range of 0-1 through the normalized exponential function (SoftMax), and the final dynamic convolution attention weight M is obtained. attention , M attention =SoftMax(conv(k,k,1)(M cs ));

[0026] S43: The final dynamic convolution kernel parameter calculation formula is: The calculation formula for the partial dynamic convolution kernel weighted data b is: in is the calculated dynamic convolution kernel parameter, is the weighted data of the dynamic convolution kernel, k is the number of static convolution kernels, are the parameters of multiple static convolution kernels, π k (x) is the dynamic convolution kernel attention parameter matrix;

[0027] Furthermore, the step S4 convolves the input voice data with the dynamic convolution kernel to obtain frame-level feature data of the speaker's voice signal, specifically including:

[0028] S51: Convolve the input speech data with the obtained dynamic convolution kernel to obtain the frame-level feature data of the speaker's speech signal. The calculation formula is: Where W T For input data, is the output data after dynamic convolution, g is the dynamic convolution layer, x is the dynamic convolution kernel parameter, and b is the dynamic convolution kernel offset data;

[0029] The advantages and beneficial effects of the present invention are as follows:

[0030] In the text-independent speaker recognition task, since each speech is composed of phonemes, the traditional static convolutional neural network cannot flexibly capture the change information of phonemes between different input speech due to its own fixed parameters, thus limiting the performance of the speaker recognition network. Using dynamic convolution method is a common method to overcome this problem. However, the common dynamic convolution method only has channel attention, and in the speaker recognition task, the phoneme changes are more reflected in the frequency domain, that is, in space. Therefore, the main innovation of the present invention is described in step 3, which combines space and channel attention to generate dynamic convolution. The dynamic convolution kernel generated by dual attention is more flexible in speaker recognition tasks than the traditional static convolutional network model, and captures richer information than the general single-attention dynamic convolutional network. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention provides a preferred embodiment of a dual-attention dynamic convolutional network graph;

[0032] Figure 2 It is the channel attention module diagram;

[0033] Figure 3 is the spatial attention module diagram;

[0034] Figure 4 Generate graphs for dynamic convolution kernels. DETAILED DESCRIPTION

[0035] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0036] The technical solution of the present invention to solve the above technical problems is:

[0037] In view of the shortcomings of the prior art, a speaker recognition method based on channel-space dual attention dynamic convolution is proposed, which can obtain a higher recognition rate and capture richer information about phoneme changes between different speech signals. The technical solution of the present invention is as follows: the channel and spatial attention weight matrix are fused as a dynamic convolution kernel generation matrix, and the generated dynamic convolution kernel is used to perform convolution operation with the input data, which includes the following steps:

[0038] S1, cutting the sound signal into multiple segments and extracting the Melp features of the sound signal;

[0039] S2, in order to solve the problem of insufficient information in the attention weight matrix in static convolution and general dynamic convolution methods, a dual attention method of channel attention and spatial attention fusion is used to extract the attention weight matrix:

[0040] First, the channel attention matrix M is extracted. avg =conv(c in ,k,1)(AvgPool(X)). X is the input data, which is globally averaged and then convolved to obtain the attention weight parameter M for each channel. avg . The parameter c in is the input channel of the data, k is the number of output channels, which is also the number of static convolution kernels that need to be set in the method of this article, and 1 is the convolution kernel size.

[0041] M avg The matrix undergoes a k×k convolution to obtain the final channel attention parameter matrix M channel Then, we extract the spatial attention. We first perform maximum and average pooling on the data, and then perform convolution to obtain a spatial attention weight matrix with a dimension of 1×H×W. Then, we perform a Flatten operation on the data. To obtain the final spatial attention parameters, we convolve the flattened data to reduce its dimension to b×k.

[0042] S3, the channel attention and spatial attention weight matrices are fused as the attention weight matrix of the dynamic convolution and added to multiple static convolution kernels according to the weights to obtain the final dynamic convolution kernel.

[0043] 4. Dynamically generate a weight matrix based on the input, and add it to multiple static convolution kernels by weight. The steps of establishing the dynamic convolution kernel generation model include:

[0044] S41: First, the obtained channel attention and spatial attention parameters are added element-wise to obtain M cs ,

[0045] S42: M cs Perform a k×k convolution, and then normalize the output data to the range of 0-1 through SoftMax, and the final dynamic convolution attention weight M is obtained. attention , M attention =SoftMax(conv(k,k,1)(M cs )).

[0046] S43: The final dynamic convolution kernel model calculation formula is, y = g (W T x+b).

[0047] S44: W T is the weight, which is composed of the weight matrix of multiple static convolution kernels and M attention Calculated, The weight b is composed of the weights of multiple static convolution kernels and M attention Calculated,

[0048] S4, uses the generated dynamic convolution kernel to replace the static convolution kernel in the convolutional neural network ResNet, and uses the network as the speaker identity feature extraction network.

[0049] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0050] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0051] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0052] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A speaker recognition method based on dual attention mechanism dynamic convolution, characterized in that: The following steps are involved: S1, obtaining a speaker's voice signal and cutting the voice signal into multiple segments, extracting the Melp features of the voice signal; S2, uses the dual attention method of channel attention and spatial attention fusion to extract the attention weight matrix; S3, the channel attention and spatial attention weight matrices are fused as the attention weight matrix of dynamic convolution and added to multiple static convolution kernels according to the weights to obtain the final dynamic convolution kernel; S4, convolving the input speech data with the dynamic convolution kernel to obtain frame-level feature data of the speaker's speech signal, and the frame-level data is used as the input of the subsequent fully connected layer for the speaker classification network; The S2 uses a dual attention method of channel attention and spatial attention fusion to extract the attention weight matrix, and adds each element thereof accordingly. The weight matrix generation model establishment step includes: S31:M avg =conv(c in ,k,1)(AvgPool(X)), X is the input data, AvgPool is the global average pooling layer, conv is the convolutional layer, and the parameter c in is the input channel of the data, k is the number of output channels, which is also the number of static convolution kernels set, and 1 is the convolution kernel size; after global average pooling of the input data, convolution is performed to obtain the attention weight parameter M of each channel avg ; S32:M channel =conv(k,k,1)(ReLU(M avg )) The attention weight parameter matrix M avg After activation by the nonlinear function ReLU, a k×k convolution is performed to obtain the final channel attention parameter matrix M channel ; S33: After the input data passes through the maximum pooling layer (MaxPooling) and the global average pooling layer (AvgPooling), the dimension is reduced to 2×H×W, where H is the dimension of each frame of speech data, and W is the number of frames of speech data. The multi-frame data is convolved to reduce its dimension to 1×H×W, and M is obtained. s Matrix M s =conv(c in ,1,1)(MaxPool,AvgPool(X))),M s The matrix is ​​the weight parameter matrix for each position in space; S34: M s After the matrix is ​​flattened, its dimension is b×T, where b is the batch size of the input data and T is the length of the flattened data. In the convolution kernel parameters, k is the number of static convolution kernels, 1 is the convolution kernel size, and M is the convolution kernel size. F =conv(T,k,1)(Flatten(M s )), M F is the spatial weight matrix without nonlinear activation; S35: M F After the matrix is ​​activated by the ReLu nonlinear function, it is convolved by k×k to obtain the final spatial attention weight parameter matrix M spatial , M spatial =conv(k,k,1)(ReLu(M F ); The step S3 combines the channel attention and spatial attention weight matrices as the attention weight matrix of the dynamic convolution and adds them to the multiple static convolution kernels according to the weights to obtain the final dynamic convolution kernel, which specifically includes: The dynamic convolution kernel generation model establishment step includes: S41: First, add the obtained channel attention and spatial attention parameters to each element Among them, M cs That is, the dual attention dynamic convolution kernel that integrates channel and spatial attention; S42: M cs Perform a k×k convolution, and then normalize the output data to the range of 0-1 through the normalized exponential function SoftMax, and the final dynamic convolution attention weight M is obtained. attention , M attention =SoftMax(conv(k,k,1)(M cs )); S43: The final dynamic convolution kernel parameter calculation formula is: The calculation formula for the partial dynamic convolution kernel weighted data b is: in is the calculated dynamic convolution kernel parameter, is the weighted data of the dynamic convolution kernel, k is the number of static convolution kernels, are the parameters of multiple static convolution kernels, π k (x) is the dynamic convolution kernel attention parameter matrix.

2. According to claim 1, a speaker recognition method based on dual attention mechanism dynamic convolution is characterized in that: The step S1 acquires the speaker's voice signal and cuts the voice signal into multiple segments to extract the Melp features of the voice signal, specifically including: dividing the voice signal into frames, adding windows, and Fourier transforming to obtain the Melp features; S21: Set the length of the Hamming window to 25ms and the step size to 10ms; S22: Set the dimension of extracting frame features to 40 dimensions.

3. According to claim 1, a speaker recognition method based on dual attention mechanism dynamic convolution is characterized in that: The step S4 convolves the input speech data with the dynamic convolution kernel to obtain frame-level feature data of the speaker's speech signal, specifically including: S51: Convolve the input speech data with the obtained dynamic convolution kernel to obtain the frame-level feature data of the speaker's speech signal. The calculation formula is W~ T =g(W T x+b), where W T is the input data, W~ T is the output data after dynamic convolution, g is the dynamic convolution layer, x is the dynamic convolution kernel parameter, and b is the dynamic convolution kernel offset data.

Images