Acoustic attention focus detection method based on tensor decomposition and convolutional neural network

Abstract

The invention discloses an acoustic attention focus detection method based on tensor decomposition and a convolutional neural network, and the method comprises the following steps: firstly, collecting a multi-channel audio signal and an electroencephalogram signal through a multi-array microphone and an electroencephalogram collection device, and carrying out the preprocessing operation, including re-reference, downsampling and normalization processing; then representing the signal as a time-frequency multi-dimensional tensor, extracting low-dimensional features of the signal through tensor decomposition, and constructing multi-channel feature representation; thirdly, inputting the low-dimensional features into a convolutional neural network optimized by a lightweight part, and extracting deep-level features related to the acoustic attention focus of the user in combination with an attention mechanism; and finally, detecting the acoustic attention focus of the user through full-connection layer classification. According to the method, the tensor decomposition and deep learning technologies are combined, the core feature information in the audio and electroencephalogram signals can be effectively extracted, the accuracy of acoustic attention focus detection in a complex environment is remarkably improved, and the method is suitable for real-time processing scenes such as hearing aids, intelligent voice assistants and brain-computer interfaces.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and particularly to an acoustic attention focus detection method based on the combination of tensor decomposition and deep learning. Background Art

[0002] In recent years, with the rapid development of intelligent devices and human-computer interaction technologies, acoustic signal processing has become a research field that has attracted much attention. In a complex acoustic environment, how to accurately determine the user's auditory focus has become a core issue in the technical applications such as hearing aids, intelligent voice assistants, virtual reality, and brain-computer interfaces. The acoustic attention detection technology has emerged. It analyzes the audio signal and the user's electroencephalogram signal to help the device determine the user's acoustic attention focus, thereby improving the perception effect of specific sound source signals. However, although the acoustic attention detection technology has made some progress, the existing methods still face many challenges when dealing with multi-source, background noise interference, and real-time processing in resource-constrained devices.

[0003] The traditional linear filtering method is one of the early main solutions. Such methods attempt to separate the target sound in a multi-source environment through simple filtering operations in the frequency domain or time domain. However, due to the inability to capture the non-linear characteristics of the acoustic attention focus, its detection accuracy in a complex background noise is very limited. With the rise of deep learning technologies, models such as convolutional neural networks, recurrent neural networks, and long short-term memory networks have been introduced into the field of acoustic signal processing. These deep learning models can effectively extract the non-linear characteristics of audio signals, adapt to complex acoustic environments, and have made some progress in detecting the user's acoustic attention focus. However, deep learning models usually have a high computational complexity and a large number of model parameters, making it difficult to achieve real-time processing in resource-limited devices (such as hearing aids or embedded devices).

[0004] To solve these problems, tensor decomposition technology has been introduced into the field of acoustic attention detection. As an effective tool for processing multi-dimensional data, tensor decomposition can jointly represent multi-channel audio signals and electroencephalogram signals as high-dimensional tensors, and extract the latent features related to attention through decomposition. Tensor decomposition methods, such as higher-order singular value decomposition and non-negative tensor decomposition, show good feature extraction capabilities in a multi-source environment, can effectively simplify complex data, and extract the core feature information related to different sound sources and electroencephalogram activities.

[0005] However, algorithms that solely rely on tensor decomposition techniques still face challenges when dealing with complex noise environments and dynamically changing sound sources. Existing tensor decomposition techniques mainly rely on linear models and are unable to fully extract the non-linear features in audio and EEG signals. Especially in the case of overlapping multi-source sounds or dynamically changing noise, the effect of tensor decomposition is limited. Therefore, how to combine tensor decomposition techniques with deep learning models to improve the accuracy of signal feature extraction and the mining of non-linear information has become the key to enhancing the performance of acoustic focus detection. Summary of the Invention

[0006] To solve the problems existing in the above-mentioned prior art, the object of the present invention is to propose an acoustic attention focus detection method based on tensor decomposition and convolutional neural network, combined with multi-channel audio signals and EEG signals. Through this method, the acoustic attention focus of the user can be efficiently and accurately detected in a complex acoustic environment, taking into account the lightweight characteristics of the model suitable for portable devices to achieve the enhancement of the target sound source.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] An acoustic attention focus detection method based on tensor decomposition and convolutional neural network, the specific steps are as follows:

[0009] Step 1: Preprocessing of acoustic and EEG signals

[0010] Collect using multi-array microphones and multi-channel EEG acquisition devices, and process the collected data for subsequent analysis of the attention focus; perform re-referencing, downsampling, filtering on the collected signals, and perform normalization at the same scale to provide support for synchronous feature extraction under tensor decomposition and neural network prediction;

[0011] Step 2: Multi-channel and tensor construction of acoustic and EEG signals

[0012] Apply frequency-slice wavelet transform to the EEG signal to generate a three-dimensional tensor T is the time length, F is the frequency resolution, and E is the number of channels; perform short-time Fourier transform on the audio speech envelope to generate a three-dimensional tensor M is the dual-channel; splice the two signals along the decision window dimension D to form a fourth-order joint tensor

[0013] Step 3: Tensor construction and decomposition

[0014] Decompose the fourth-order joint tensor X into a low-dimensional representation:

[0015] X = G ×1A (1) ×2A (2) ×3A (3) ×4A (4)

[0016] where is the core tensor, where K1, K2, K3, and K4 are the four dimensions of the low-dimensional core tensor G respectively; is the factor matrix; the loss function with nuclear norm regularization is minimized by the gradient descent method, specifically as follows, to extract sparse low-dimensional features across time, frequency, channels, and decision windows;

[0017]

[0018] is the loss function, λ is the regularization coefficient, ‖·‖ * is the nuclear norm;

[0019] Step 4: Convolutional neural network feature extraction

[0020] The factor matrices A (1) , A (2) , A (3) , A (4) are concatenated by channels into a three-dimensional feature map and input into a convolutional neural network to extract spatio-temporal features, where T' is the updated time step, F' is the updated frequency step, and C is the channel; the convolutional neural network includes three layers of 2D convolution, max pooling, and fully connected layers;

[0021] Step 5: Lightweight deployment optimization of the convolutional neural network

[0022] To achieve real-time processing of devices such as hearing aids, the following optimizations are performed on the convolutional neural network in Step 4:

[0023] Pruning the weights W of the convolutional layer according to the channel importance:

[0024] Keep the channels = {m | ‖W m ‖1 > 0.1max(‖W‖1)}

[0025] where m is the output channel index, and W m is the convolutional weight of channel m;

[0026] At the same time, integerization is performed on the pruned weights:

[0027]

[0028] where W int8 is an 8-bit integer weight with a value range of [-127, 127], and round(·) is the rounding operation.

[0029] Step 6: Cross-modal attention weighting

[0030] For the time-frequency feature map of the convolution output Calculate the attention weights along the time and frequency dimensions, where T ″ is the time-step slice after output, and F ″ is the frequency-step slice after output:

[0031]

[0032] Among them, α t (i) is the attention weight in the time dimension, and α f (j) is the attention weight in the frequency dimension. W t , V t , b t and W f , V f , b f are learnable parameters. H[i, :, :] is the extracted two-dimensional time-dimensional feature slice, and H[:, j, :] is the extracted two-dimensional frequency-dimensional feature slice. The attention weights are fused with the feature map through tensor multiplication:

[0033]

[0034] To enhance the spatio-temporal-spectral feature focus on the sound source of user concern. Among them, H att is the weighted feature map, and H[i, j, :] is the extracted time-frequency feature slice;

[0035] Step 7: Classification and decision-making of the fully connected layer

[0036] Flatten the weighted feature map H att into a vector D' is the slice length of the flattened vector, and input it into the fully connected layer to output the classification probability:

[0037]

[0038] Among them, w c , b c are the parameters of the fully connected layer, and w k , b k are the weights and biases corresponding to the k-th class in the fully connected layer, which directly participate in the calculation of the classification probability. h is the vector representation of the weighted spatio-temporal-spectral features. The loss function uses cross-entropy plus L2 regularization. Finally, c ∈ {left, right} is the calculated acoustic attention focus direction.

[0039] The present invention provides an acoustic attention focus detection method based on tensor decomposition and convolutional neural network, which aims to solve the problem of detecting the user's acoustic attention focus in a multi-source environment. By constructing a multi-dimensional tensor to represent the time-frequency feature information of audio signals and electroencephalogram signals, and combining tensor decomposition technology and convolutional neural network, this method can extract potential features related to acoustic attention focus, effectively improving the detection accuracy under complex noise and non-linear feature conditions. At the same time, this method takes into account computational efficiency and is applicable to resource-constrained embedded devices to achieve real-time processing. This method provides technical support for people with hearing defects, analyzes and obtains their acoustic focus, and provides emerging technical support and development directions for hearing aids. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0041] Figure 1 It is a flowchart of the acoustic attention focus detection method based on tensor decomposition and convolutional neural network of the present invention;

[0042] Figure 2 It is a process diagram of data tensor decomposition of the present invention;

[0043] Figure 3 It is a schematic diagram of the minimum expected switching duration of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the data set in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0045] The data used in this embodiment are speech and EEG data. The 64-channel EEG data of 17 subjects were recorded using the BrainAmp system at a sampling rate of 8192 Hz, and the international 10 / 20 system was adopted. The speech stimuli included 64 short Chinese stories, which were narrated by a female and a male. The frequency band was limited to 4 kHz and the volume was 60 dB. The subjects wore a pair of insert earphones, and the two segments of speech were played from the positions of 90° on the left and 90° on the right respectively. Each subject was required to randomly select one audio to listen to attentively and ignore the other in each trial. The position of the attention focus and the gender of the narrator were also random. Each trial lasted for 55 to 90 seconds. A total of 17 subjects completed 32 trials, with a total duration of 20 minutes, and 10.8 hours of EEG signals were recorded cumulatively.

[0046] The following is an explanation of the use and experimental results of the acoustic attention focus detection method based on tensor decomposition and convolutional neural network:

[0047] As Figure 1 shown, an acoustic attention focus detection method based on tensor decomposition and convolutional neural network of the present invention includes the following steps:

[0048] Step 1: Preprocessing of acoustic and EEG signals

[0049] In the preparation process of EEG data, a systematic method was implemented in two datasets to optimize the signal quality and retain key information. First, a 50 Hz notch filter was applied to reduce power line interference, thereby ensuring the integrity of the data. At the same time, additional artifact cleaning operations such as eye movement or muscle activity artifact removal were avoided to not affect the retention of important information. Subsequently, all electrode signals were re-referenced based on the average value, the data was downsampled to 64 Hz, and band-pass filtering of 1 - 32 Hz was performed. Finally, the data of all channels was normalized for normalization processing.

[0050] The first step in speech data preparation is to apply a low-pass filter with a cut-off frequency of 8 kHz to remove frequency components above 8 kHz. Subsequently, the audio signal was downsampled to a sampling rate of 16 kHz. This process aims to reduce high-frequency noise and unnecessary details to make the signal more suitable for subsequent analysis and processing;

[0051] Step 2: Multi-channel and tensor construction of acoustic and EEG signals

[0052] Apply frequency-slice wavelet transform to the EEG signal to generate a three-dimensional tensor T is the time length, F is the frequency resolution with F = 1 Hz, E is the number of channels with E = 64; perform short-time Fourier transform on the audio speech envelope to generate a three-dimensional tensor M is two channels with M = 2; splice the two signals along the decision window dimension D to form a fourth-order joint tensor The time resolution is 1 ms, the frequency resolution is 1 Hz, and the construction method and decomposition order of the tensor are as follows Figure 2 shown

[0053] Step 3: Tensor construction and decomposition

[0054] Decompose the fourth-order joint tensor X into a low-dimensional representation:

[0055] X = G ×1A (1) ×2A (2) ×3A (3) ×4A (4)

[0056] where is the core tensor, where K1, K2, K3, and K4 are the four dimensions of the low-dimensional core tensor G respectively; are the factor matrices; minimize the loss function with nuclear norm regularization through the gradient descent method, specifically as follows, to extract sparse low-dimensional features across time, frequency, channels, and decision windows;

[0057]

[0058] is the loss function, λ is the regularization coefficient, ‖·‖ * is the nuclear norm;

[0059] Step 4: Convolutional neural network feature extraction

[0060] Concatenate the factor matrices A (1) , A (2) , A (3) , A (4) along the channels to form a three-dimensional feature map Input it into a convolutional neural network. The network contains three layers of 2D convolution, with a kernel size of 3×3, the number of channels 64→128→256, ReLU activation, a maximum pooling 2×2 window, and a fully connected layer, 1024 dimensions, a Dropout rate of 0.5, to extract spatio-temporal features;

[0061] Step 5: Optimization of lightweight deployment of convolutional neural network

[0062] To achieve real-time processing of devices such as hearing aids, perform the following optimizations on the convolutional neural network in Step 4:

[0063] Prune the weights W of the convolutional layer according to the channel importance:

[0064] Keep the channels = {m∣‖W m ‖1>0.1max(‖W‖1)}

[0065] where m is the output channel index, and W m is the convolution weight of channel m.

[0066] At the same time, integerization is performed on the pruned weights:

[0067]

[0068] where W int8 is the 8-bit integer weight with a value range of [-127, 127], and round(·) is the rounding operation; Step 6: Cross-modal attention weighting

[0069] Calculate the attention weights for the time-frequency feature map of the convolution output along the time and frequency dimensions. Here, T ″ is the time step slice after output, and F ″ is the frequency step slice after output:

[0070]

[0071] where α t (i) is the attention weight in the time dimension, and α f (j) is the attention weight in the frequency dimension. W t , V t , b t and W f , V f , b f are learnable parameters. H[i, :, :] is the extracted two-dimensional time-dimensional feature slice, and H[:, j, :] is the extracted two-dimensional frequency-dimensional feature slice; The attention weights are fused with the feature map through tensor multiplication:

[0072]

[0073] to enhance the spatio-temporal-spectral feature focus on the sound source of user concern. Here, H att is the weighted feature map, and H[i, j, :] is the extracted time-frequency feature slice;

[0074] Step 7: Classification and decision-making of the fully connected layer

[0075] Flatten the weighted feature map H att into a vector where D' is the slice length of the flattened vector, and input it into the fully connected layer to output the classification probability:

[0076]

[0077] where w c , b c are the parameters of the fully connected layer, and w k , bk They are the weights and biases corresponding to the k-th class in the fully connected layer, directly participating in the calculation of classification probabilities. h is the vector representation of the weighted spatio-temporal - spectral features. The loss function uses cross-entropy plus L2 regularization (weight decay 1 e-4 ). Finally, c ∈ {left, right} is the calculated direction of the acoustic attention focus.

[0078] Furthermore, in this embodiment, the effect of acoustic attention focus detection on the speech and EEG datasets is verified.

[0079] The method proposed in the present invention is supervised trained using the PyTorch framework. The optimization method adopts stochastic gradient descent, and the initial learning rate is 1×10 -3 to minimize the cost function. The batch size is set to 50, and the training is carried out within 100 epochs, and the learning rate is adaptively adjusted according to the number of epochs. The cross-entropy loss function is used in the calculation. To prevent overfitting, a Dropout layer is applied after the pooling layer and the fully connected layer, and a batch normalization layer is inserted after the convolutional layer. The Softmax output layer provides binary classification predictions: 0 represents "left", 1 represents "right", and is compared with the actual attention labels to evaluate the accuracy. To ensure the robustness of the model, each subject is trained ten times with different initializations, and the results are averaged. The method aims to establish a reliable and transferable attention prediction model, mitigating the impact of random initialization and minimizing overfitting through appropriate regularization strategies.

[0080] The linear model CCA, the non-linear models DNN, CNN, and the state-of-the-art model CMA are simultaneously applied to this dataset, and the effects are compared with the method proposed in the present invention. For the linear model, the CCA model is directly reproduced on the dataset, and its results serve as the benchmark for the linear model. For the non-linear models, the models are reconstructed and reproduced using the same parameters as the proposed model to establish the benchmarks for the non-linear models.

[0081] In the following table, the average detection accuracy of each model on all subjects is shown. The method proposed in the present invention is significantly better than the linear model baseline (P < 0.001) in all decision windows of the dataset. The comparison with the non-linear models further confirms the superior performance of the proposed model. In the dataset, the method proposed in the present invention is better than the CMA model in all performance metrics. Specifically, the performance of the method proposed in the present invention is improved by 3.2, 4.3, 3.6, 5.6, and 7.2 percentage points respectively compared with the CMA model in each metric. In addition, the method proposed in the present invention is better than the other two non-linear baseline models, namely DNN (P < 0.001) and CNN (P < 0.01) in all metrics and datasets. These results consistently confirm the advantages of tensor decomposition in attention focus detection, thus providing higher accuracy under different conditions.

[0082] Model Accuracy at 0.1s Accuracy at 0.2s Accuracy at 0.5s Accuracy at 1s Accuracy at 2s CCA 52.1 54.3 56.5 58.3 63.2 DNN 62.4 66.1 70.3 73.5 75.2 CNN 66.3 70.8 73.2 76.4 79.6 CMAA 71.0 74.3 75.8 80.6 83.9 The method of the present invention 74.2 78.6 79.4 86.2 91.1

[0083] The minimum expected switching duration can be used to evaluate the efficiency of focus detection. The method of the present invention is compared with the performance of the CMAA model.

[0084] As shown in Figure 3, the results of the minimum expected switching duration calculated by the fixed window length and the corresponding decoding accuracy are shown. These results cover the performance of each subject in each dataset, so that the performance of the CMAA model and the method proposed in the present invention can be directly compared. The minimum expected switching duration values of the CMAA model range from 0.46 to 5.53 seconds, and the median is 0.89 seconds, while the minimum expected switching duration values of the method proposed in the present invention range from 0.39 to 4.38 seconds, and the median is 0.66 seconds, which is significantly better than the CMAA model (P < 0.01). This shows that the method proposed in the present invention shows a shorter switching duration in the actual auditory scene, thus achieving faster acoustic decoding to determine the attention focus.

Claims

1. An acoustic attention focus detection method based on tensor decomposition and convolutional neural network, characterized in that Including the following steps: Step 1: Preprocessing of acoustic and electroencephalogram signals Collect using multi-array microphones and multi-channel electroencephalogram acquisition devices, and process the collected data for subsequent analysis of the attention focus; perform re-referencing, downsampling, filtering on the collected signals, and normalization at the same scale to provide support for synchronous feature extraction and neural network prediction under tensor decomposition; Step 2: Multi-channel and tensor construction of acoustic and electroencephalogram signals Apply frequency-slice wavelet transform to EEG signals to generate a three-dimensional tensor T is the time length, F is the frequency resolution, and E is the number of channels; perform short-time Fourier transform on the audio speech envelope to generate a three-dimensional tensor M is for stereo channels; Concatenate the two signals along the decision window dimension D to form a fourth-order joint tensor Step 3: Tensor construction and decomposition Decompose the fourth-order joint tensor X into a low-dimensional representation: X = G × 1 A (1) × 2 A (2) × 3 A (3) × 4 A (4) Among them is the core tensor, where K1, K2, K3, and K4 are the four dimensions of the low-dimensional core tensors G respectively; is the factor matrix; minimizing the loss function with nuclear norm regularization by the gradient descent method is as follows, to extract sparse low-dimensional features across time, frequency, channels, and decision windows; is the loss function, λ is the regularization coefficient, and ‖·‖ * is the nuclear norm; Step 4: Feature extraction of convolutional neural network The factor matrix A obtained by decomposition (1) , A (2) , A (3) , A (4) are concatenated by channels into a three-dimensional feature map Input into a convolutional neural network to extract spatio-temporal features, where T' is the updated time step, F' is the updated frequency step, and C is the channel; the convolutional neural network includes three layers of 2D convolution, max pooling, and fully connected layers; Step 5: Lightweight deployment optimization of convolutional neural network To achieve real-time processing of hearing aids, perform the following optimizations on the convolutional neural network in Step 4: Prune the weights W of the convolutional layer according to the channel importance: Reserved channel = {m | ||W m ||1 > 0.1 max(||W||1)} where m is the output channel index, and W m is the convolution weight of channel m; At the same time, perform integerization on the pruned weights: where W int8 is an 8-bit integer weight with a value range of [-127, 127], and round(·) is a rounding operation; Step 6: Cross-modal attention weighting The time-frequency feature map of the convolution output Calculate the attention weights along the time and frequency dimensions, where T ″ is the time-step slice after output, and F ″ is the frequency-step slice after output: Among them, α t (i) is the attention weight in the time dimension, and α f (j) is the attention weight in the frequency dimension. W t , V t , b t and W f , V f , b f are learnable parameters. H[i, :, :] is the extracted two-dimensional time-dimensional feature slice, and H[:, j, :] is the extracted two-dimensional frequency-dimensional feature slice. The attention weight and the feature map are fused through tensor multiplication: To enhance the spatio-temporal and spectral feature focusing on the sound source of interest to the user, where H att is the weighted feature map, and H[i, j, :] is the extracted time-frequency feature slice; Step 7: Classification and decision-making of fully connected layer Flatten the weighted feature map H att into a vector where D' is the slice length of the flattened vector, and input it into the fully connected layer to output the classification probability: where w c , b c are the parameters of the fully connected layer, w k , b k are the weights and biases corresponding to the k-th class in the fully connected layer, directly participating in the calculation of the classification probability. h is the vector representation of the weighted spatio-temporal - spectral features. The loss function uses cross-entropy plus L2 regularization. Finally, c ∈ {left, right} is the calculated direction of the acoustic attention focus.

Images