Method, device and electronic device for generating cross-scene auditory attention decoding model

Abstract

The present application relates to a method, device and electronic device for generating a cross-scene auditory attention decoding model, wherein the generating method comprises: obtaining source domain EEG signals and target domain EEG signals, the source domain EEG signals and the target domain EEG signals are EEG signals generated by subjects in different scenes respectively; pre-training the auditory attention decoding model through the source domain EEG signals, and performing domain adaptation training on the pre-trained auditory attention decoding model through the target domain EEG signals. The generating method of the cross-scene auditory attention decoding model proposed in the present invention adopts the regularization loss (SRL) as the domain adaptation loss, which can minimize the data difference between different scenes, thereby reducing the domain offset, and can generate an auditory attention decoding model adapted to different scenes, solving the problem that the generating method of the traditional auditory attention decoding model cannot generate an auditory attention decoding model that can adapt to different scenes.

Description

Technical Field

[0001] The present application relates to the field of auditory attention decoding, and in particular to a method, device and electronic device for generating a cross-scene auditory attention decoding model. Background Art

[0002] EEG signals have been widely used in various auditory attention decoding applications, including neural control devices such as hearing aids, due to their non-invasiveness and high temporal resolution. Therefore, it is crucial to develop an effective scheme for decoding auditory attention. The traditional method of generating auditory attention decoding models usually uses a single dataset (from the same scene or similar scenes) to train the auditory attention decoding model. There are many limitations, the main problems include limited model generalization ability, difficulty in ensuring external validity, data bias, overfitting risk, lack of diversity, experimental design bias, insufficient evaluation accuracy, poor adaptability, lack of simulation of real-world complexity, and limited research depth. These shortcomings mean that the model may overfit the characteristics of specific scenes, have difficulty adapting to the changing real environment, and may not accurately reflect the situation of different populations or conditions.

[0003] There is currently no effective solution to the problem that traditional auditory attention decoding model generation methods are unable to generate auditory attention decoding models that can adapt to different scenarios. Summary of the invention

[0004] The present invention provides a method for generating a cross-scene auditory attention decoding model to solve the problem that the traditional method for generating an auditory attention decoding model cannot generate an auditory attention decoding model that can adapt to different scenes.

[0005] The present invention provides a method for generating a cross-scene auditory attention decoding model, comprising:

[0006] Acquire a source domain EEG signal and a target domain EEG signal, wherein the source domain EEG signal and the target domain EEG signal are respectively EEG signals generated by a subject in different scenarios;

[0007] The auditory attention decoding model is pre-trained by using source domain EEG signals, wherein the auditory attention decoding model includes a feature extraction module, a classification module and a domain adaptation module, and the pre-training steps include:

[0008] Extracting features of the source domain EEG signal by the feature extraction module to obtain a first EEG feature;

[0009] Predicting and classifying the first EEG feature by the classification module to obtain a first classification loss;

[0010] Optimizing the first classification loss to pre-train the auditory attention decoding model;

[0011] The pre-trained auditory attention decoding model is subjected to domain adaptation training using the target domain EEG signal. The steps of the domain adaptation training include:

[0012] Extracting features of the target domain EEG signal by the feature extraction module to obtain a second EEG feature;

[0013] Predicting and classifying the second EEG feature by the classification module to obtain a second classification loss;

[0014] The domain adaptation module performs singular value decomposition on the first EEG feature and the second EEG feature and retains the first m maximum singular values ​​and calculates the regularization loss. L sr :

[0015]

[0016] in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i represents the ith singular value among the first m largest singular values ​​of the second EEG feature;

[0017] The first classification loss and the regularization loss are optimized to perform domain adaptation training on the auditory attention decoding model.

[0018] Compared with the related art, the method for generating a cross-scene auditory attention decoding model proposed in the present invention adopts a regularization loss (SRL) as a domain adaptation loss, which can minimize the data differences between different scenes, thereby reducing domain shift, and can generate an auditory attention decoding model that adapts to different scenes. This solves the problem that the traditional method for generating an auditory attention decoding model cannot generate an auditory attention decoding model that can adapt to different scenes.

[0019] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of the structure of an auditory attention decoding model in some embodiments of the present invention;

[0021] Figure 2 It is a schematic diagram of the model generation process in some embodiments of the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0024] In an embodiment of the present invention, a method for generating a cross-scene auditory attention decoding model is provided, and the process includes step S110, step S120, and step S130.

[0025] Step S110, obtaining a source domain EEG signal and a target domain EEG signal, where the source domain EEG signal and the target domain EEG signal are respectively EEG signals generated by a subject in different scenarios.

[0026] In order to train the auditory attention decoding model, sample data must be collected first. In order to make the model more generalizable, the present invention adopts a domain adaptation method for model training. This step requires collecting the EEG signals generated by the subject in at least two different scenarios, and using the EEG signals generated in one of the scenarios as the source domain EEG signals (source domain sample data), and the EEG signals generated in at least one other scenario can be used as different target domain EEG signals (target domain sample data), respectively, depending on the scenarios that the model needs to adapt to. Exemplarily, different scenarios include audio stimulation scenarios and audio-video stimulation scenarios. In the audio stimulation scenario, only audio is played to the subject, while in the audio-video stimulation scenario, audio and video are played to the subject at the same time.

[0027] In order to ensure a certain amount of data, there should be multiple subjects, and the EEG signals generated by each subject in different scenarios need to be obtained.

[0028] In order to improve the model training effect, the EEG signal can also be preprocessed to remove noise or other unnecessary components in the signal.

[0029] Therefore, in some embodiments, the method further comprises:

[0030] Step S100, performing data preprocessing on the source domain EEG signal and the target domain EEG signal, the data preprocessing step includes:

[0031] Step S101, filtering the EEG signal by using a power frequency notch filter and a finite impulse response filter.

[0032] Step S102, downsampling the filtered EEG signal to a target frequency.

[0033] Step S103, performing independent component analysis on the downsampled EEG signal and retaining the principal component.

[0034] Illustratively, during initial data processing, a power frequency notch filter is applied to reduce 50 Hz power line noise, followed by high-pass and low-pass finite impulse response filtering to remove voltage drift and high-frequency noise. The signal is downsampled to 128 Hz to retain the necessary frequency components while reducing computational requirements. Independent component analysis is then used to isolate and remove artifacts such as eye movements and muscle activity. These preprocessing steps are critical to obtaining accurate EEG data, which is essential for analyzing neural activity patterns and evaluating auditory attention decoding accuracy under different experimental conditions.

[0035] Step S120, pre-training the auditory attention decoding model using source domain EEG signals.

[0036] Reference Figure 1, the auditory attention decoding model includes a feature extraction module (refer to Figure 1 a, b), classification module and domain adaptation module (refer to Figure 1 (Part c) The pre-training steps include:

[0037] Step S121, extracting features from the source domain EEG signal through a feature extraction module to obtain a first EEG feature.

[0038] Step S122: predict and classify the first EEG feature through a classification module to obtain a first classification loss.

[0039] Step S123, optimizing the first classification loss to pre-train the auditory attention decoding model.

[0040] In the domain adaptation method, the model is first trained with source domain sample data. The classification loss of the model can be calculated using the cross entropy function, which is mainly used to improve the model's prediction classification accuracy for the source domain.

[0041] Step S130, domain adaptation training is performed on the pre-trained auditory attention decoding model using the target domain EEG signal. The domain adaptation training steps include:

[0042] Step S131, extracting features of the target domain EEG signal through a feature extraction module to obtain a second EEG feature.

[0043] Step S132: predict and classify the second EEG feature through a classification module to obtain a second classification loss.

[0044] Step S133, performing singular value decomposition on the first EEG feature and the second EEG feature through the domain adaptation module and retaining the first m largest singular values ​​and calculating the regularization loss respectively L sr :

[0045]

[0046] in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i Represents the i-th singular value among the first m largest singular values ​​of the second EEG feature.

[0047] Step S134, optimizing the first classification loss and the regularization loss to perform domain adaptation training on the auditory attention decoding model.

[0048] For domain adaptation training, on the one hand, the second classification loss corresponding to the target domain data is calculated, and on the other hand, the regularization loss between the source domain data and the target domain data is calculated. The two losses can be combined by preset weights as the total loss of the model in the target domain training.

[0049] The first classification loss and the second classification loss are both calculated by the cross entropy loss function, which is:

[0050]

[0051] in, L ce represents the classification loss, y represents the true label, p Represents the probability that the sample predicted by the model belongs to the true category in the decision window.

[0052] The present invention introduces a specific regularization loss (SRL) method to minimize the data differences between different scenarios, thereby reducing domain shift.

[0053] Features with large singular values ​​in the eigenvectors are crucial to the feature matrix and its transferability, and they usually contain the most important information in the data. However, although these eigenvectors play a significant role in transferability, other eigenvectors with smaller singular values ​​cannot be ignored in distinguishing different categories, because most discriminative classifiers rely on information in multiple dimensions.

[0054] In order to prevent the eigenvector of a certain dimension from excessively affecting the result due to its largest singular value, it needs to be compressed and decomposed. The SRL method first applies SVD to the source domain feature matrix Fs (first EEG feature) and the target domain feature matrix Ft (second EEG feature), extracts the largest m singular values, and then uses SRL as a regularization term to act on these singular values, which helps to balance the importance of the eigenvector. This method helps to improve the generalization ability and distinguishability of the model while maintaining the transferability of the features.

[0055] In summary, the present invention proposes a method for generating a cross-scene auditory attention decoding model, which mainly adopts the regularization loss (SRL) as the domain adaptation loss, which can minimize the data differences between different scenes, thereby reducing domain shift, and can generate an auditory attention decoding model that adapts to different scenes. It solves the problem that the traditional method for generating an auditory attention decoding model cannot generate an auditory attention decoding model that can adapt to different scenes.

[0056] In addition, the SRL method can be applied directly between the feature vectors of each batch, which makes it more flexible and effective in dealing with cross-scenario tasks. In this way, it can better adapt to the data characteristics in different scenarios and improve the adaptability and accuracy of the model. The application of this method is not only innovative in theory, but also shows great potential in practical applications, especially in scenarios that need to process multimodal data and cross-domain tasks.

[0057] In some of the embodiments, the feature extraction module includes a time domain feature extraction branch (see Figure 1 Part a in the figure can also be called the time feature extraction branch, time domain feature branch, etc.) and the frequency domain feature extraction branch (refer to Figure 1 Part b in the figure can also be called the spectral residual branch); the input of the time domain feature extraction branch is the model input after common spatial pattern (CSP) filtering, and the input of the feature extraction module is the multi-band differential entropy feature extracted from the model input; the output of the time domain feature extraction branch (time domain feature) and the output of the frequency domain feature extraction branch (frequency domain feature) are feature fused to obtain the output of the feature extraction module.

[0058] The model input is the source domain EEG signal or the target domain EEG signal. The time domain feature extraction branch is used to extract the time domain features in the model input, and the frequency domain feature extraction branch is used to extract the frequency domain features in the model input. Due to the high dimensionality of the original EEG data, it contains a large number of channels and sampling points. Directly applying it to classification may result in suboptimal performance. In order to enhance the ability to distinguish between different brain states, this embodiment uses the features extracted by CSP, which are more focused on the features of the EEG signal. The input EEG data X={ x i |i=1,...,N}, for each x i ∈ R C×L The dimensionality reduction is performed through CSP filtering, thereby retaining the most useful spatial features for classification and reducing the computational complexity.

[0059] The basic principle of the CSP algorithm is to extract the features of EEG signals under two-classification tasks by spatial filtering. By diagonalizing the covariance matrix of the EEG signal, a set of filters that can maximize the inter-class variance is found to extract a feature vector with discriminability. This method can effectively extract features related to specific tasks from multi-channel EEG signals.

[0060] At the same time, the use of multi-band differential entropy features to assist in extracting frequency domain features has shown significant advantages in auditory attention decoding. This method combines the differential entropy's sensitive capture of signal complexity and irregularity with the frequency domain features' accurate description of the signal frequency distribution, achieving a more comprehensive and multi-dimensional analysis of auditory signals. This feature fusion not only improves the accuracy and robustness of decoding, reduces noise interference, but also simplifies the complexity of the decoding model and enhances its generalization ability. Therefore, the integration of differential entropy features and frequency domain features is of great value in improving the performance of auditory attention decoding, and is widely used in various occasions that require accurate identification of auditory attention.

[0061] Furthermore, in some embodiments, the time domain feature extraction branch includes a transformer, a KAN network, and a fully connected block connected in sequence, and the frequency domain feature extraction branch includes a first 3D convolution, a residual network, two second 3D convolutions, and an average pooling layer connected in sequence.

[0062] The expression of KAN network is:

[0063]

[0064] in, x ′ represents the network input, SplineOpt( x ′) represents the model output, BaseActivation() represents the baseline activation function, W base represents the trainable weight parameter matrix, B () represents B basis functions, W spline Represents the spline weight matrix.

[0065] The time domain feature extraction branch processes the model input through the transformer, KAN network and fully connected block in sequence. This embodiment innovatively introduces the combination of transformer and KAN network to extract time domain features.

[0066] The transformer adopts a multi-head attention mechanism, which allows the model to learn information in different representation subspaces in parallel.

[0067] The KAN network improves computational efficiency and reduces model parameters. The core feature of the KAN network is that it places activation functions on the weights of the network, and these activation functions are learnable and are usually parameterized using B-splines. This design enables the KAN network to have higher accuracy and interpretability in handling complex function fitting and solving partial differential equations.

[0068] KAN uses B-spline basis functions to replace traditional linear layers and adopts learnable activation functions, thereby reducing the number of model parameters. This makes the network more lightweight while maintaining performance, and is suitable for low computing resource scenarios; KAN uses nonlinear basis functions such as B-spline to enable the network to better capture complex nonlinear changes. This significantly improves the network's representation capabilities when processing high-dimensional nonlinear data such as EEG signals; because KAN utilizes the characteristics of local basis functions, it can perform fine modeling in local areas of the data, allowing the network to process signals with complex spatiotemporal distribution changes; the KAN network can effectively improve the generalization ability of the model in scenarios where data is scarce or distribution differences exist by reducing unnecessary parameters and improving the model's ability to capture complex nonlinear patterns.

[0069] The KAN network processes the output of the transformer, and the fully connected block flattens the KAN network to obtain the temporal features, that is, the feature extraction results of the time domain feature extraction branch.

[0070] Specifically, the transformer includes a multi-head attention block (used to implement the multi-head attention mechanism) and a residual connection & layer normalization layer (used to implement residual enhancement and layer normalization).

[0071] The formula of the multi-head attention mechanism is as follows:

[0072]

[0073] in, x represents the input of the transformer, x ′ represents the output of the multi-head attention block, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively. d k Represents the dimension of the key vector, and the softmax function normalizes the scaled dot product attention.

[0074] The formula for residual enhancement is as follows:

[0075]

[0076] Output x res As the input of the KAN network.

[0077] The fully connected block first flattens the output of the KAN network and then performs layer normalization.

[0078] Exemplarily, the residual network includes a first residual block, a second residual block, and a third residual block connected in sequence (the three residual blocks are respectively Figure 1The first 3D convolution uses a convolution kernel of size 7x7, a stride of 2, and a padding of 3, and uses a maximum pooling operation with a pooling window of 3x3, a stride of 2, and a padding of 1; the first 3D convolution is configured to expand the number of channels of the feature map from 1 to 32, so as to reduce the spatial size of the feature map; the first residual block is configured to expand the number of channels of the feature map from 32 to 64, so as to increase the depth of the network without causing the gradient to disappear; the second residual block is configured to expand the number of channels of the feature map from 64 to 128, so as to further reduce the spatial size of the feature map; the third residual block is configured to expand the number of channels of the feature map from 128 to 256.

[0079] The output of the residual network is used as the input of two 3D convolutions to further extract features on the channel.

[0080] The adaptive average pooling layer (AdaptiveAveragePoolingLayer) is used at the end of the frequency domain feature extraction branch to average each channel of the feature map and reduce the spatial size to 1x1x1.

[0081] As above, at least one embodiment of each part of the method for generating a cross-scene auditory attention decoding model provided by the present invention has been described separately. It should be noted that the various parts of the method for generating a cross-scene auditory attention decoding model can be combined using different embodiments to constitute different embodiments of the method for generating a cross-scene auditory attention decoding model.

[0082] Reference Figure 2 In a specific overall embodiment, a method for generating a cross-scene auditory attention decoding model includes:

[0083] S1: Obtain EEG data of multiple subjects in two experimental paradigms (audio-video scene and audio scene) and perform necessary preprocessing.

[0084] It should be noted that during initial data processing, a notch filter was applied to reduce 50Hz power line noise, followed by high-pass and low-pass finite impulse response filtering to remove voltage drift and high-frequency noise. The signal was downsampled to 128Hz to retain the necessary frequency components while reducing computational requirements. Independent component analysis was then used to isolate and remove artifacts such as eye movements and muscle activity. These preprocessing steps are critical to obtaining accurate EEG data, which is essential for analyzing neural activity patterns and evaluating auditory attention decoding accuracy under different experimental conditions.

[0085] S2: The dual-branch domain adaptation method consists of a time domain feature extraction branch and a frequency domain feature extraction branch.

[0086] S21: In the time domain feature extraction branch, CSP filtering is first used. The basic principle of the CSP algorithm is to extract spatial filtering features from EEG signals under two-classification tasks. By diagonalizing the covariance matrix of the EEG signal, a set of filters that can maximize the inter-class variance is found to extract feature vectors with discrimination. This method can effectively extract features related to specific tasks from multi-channel EEG signals. The multi-head attention mechanism allows the model to learn information in parallel in different representation subspaces. Secondly, the KAN network is used to improve computational efficiency and reduce model parameters. The core feature of the KAN network is that it places activation functions on the weights of the network, and these activation functions are learnable, usually parameterized using B-splines. This design makes the KAN network more accurate and interpretable in dealing with complex function fitting, partial differential equation solving, etc. KAN reduces the number of model parameters by using B-spline basis functions instead of traditional linear layers and adopting learnable activation functions. This makes the network more lightweight while maintaining performance, suitable for low computing resource scenarios;

[0087] KAN uses nonlinear basis functions such as B-spline to enable the network to better capture complex nonlinear changes. This significantly improves the network's representation capabilities when processing high-dimensional nonlinear data such as EEG signals.

[0088] Since KAN utilizes the characteristics of local basis functions, it can perform fine modeling in local areas of the data, allowing the network to process signals with complex temporal and spatial distribution changes;

[0089] The KAN network can effectively improve the generalization ability of the model in scenarios where data is scarce or distributed differently by reducing unnecessary parameters and improving the model's ability to capture complex nonlinear patterns.

[0090] There is a time feature branch of KAN, which is characterized by improving calculation efficiency and reducing calculation amount. The expression of KAN is:

[0091]

[0092] in, x ′ represents the network input, SplineOpt( x ′) represents the model output, BaseActivation() represents the baseline activation function, W base represents the trainable weight parameter matrix, B () represents B basis functions, W spline represents the spline weight matrix. Finally, the temporal feature Ft is output by the flattened fully connected block (FCB).

[0093] S22: In terms of frequency domain feature extraction, it combines convolutional layers and Transformer encoders specifically for processing EEG data. It extracts features through convolutional layers, and then further processes these features using Transformer encoders. On the other hand, the ConvLayer and ResBlock classes implement 3D convolutional layers and residual blocks, respectively, which are the basic components for building deep convolutional networks. This branch uses these convolutional layers and residual blocks to extract features and classifies them through linear layers. These model components can be used individually or combined into more complex network architectures to solve various machine learning and deep learning tasks.

[0094] The first 3D convolution converts the input single-channel image into 32 feature channels, using a 7x7 convolution kernel, a stride of 2, and a padding of 3. A maximum pooling operation is used with a pooling window of 3x3, a stride of 2, and a padding of 1 to reduce the spatial size of the feature map.

[0095] Layer 1, contains a residual block, which is used to increase the depth of the network without causing the gradient vanishing problem. It expands the feature map of 32 channels to 64 channels. Layer 2, the residual block of the second sequence, expands the feature map of 64 channels to 128 channels with a step size of 2, which is used to further reduce the spatial size of the feature map. Layer 3, the residual block of the third sequence, expands the feature map of 128 channels to 256 channels with a step size of 2. The output of the residual network is used as the input of the next two 3D convolutions for further feature extraction on the channel. In addition, it is used to normalize the features, which helps to speed up the training process and improve the generalization ability of the model.

[0096] Adaptive Average Pooling Layer, used at the end of the network, averages each channel of the feature map and reduces the spatial size to 1x1x1.

[0097] S3: The EEG data in audio and video scenes and audio scenes are spliced ​​through the dual-branch time domain and spectrum space domain features to extract the time-frequency-space features of the EEG data; feature fusion combines features from different data sources or extraction methods to improve the prediction accuracy, robustness and generalization ability of the model, while reducing information redundancy and enhancing the model's ability to understand and express complex data, thereby playing a key role in a variety of application scenarios.

[0098] S4: Description of the domain adaptation method and classification prediction. The EEG data collected under audio and video scene stimulation is subjected to dual-branch feature extraction and fusion to obtain the features and prediction labels under audio and video stimulation. Similarly, the EEG data collected under audio scene stimulation only obtains the features under audio stimulation.

[0099] S41: Based on the time-frequency dual-branch method, in order to adapt to the single-subject cross-scenario task, introducing a pre-adaptation method to deal with data differences is an effective strategy. Features with large singular values ​​in the feature vector are crucial to the feature matrix and its transferability, and they usually contain the most important information in the data. However, although these feature vectors play a significant role in transferability, other feature vectors with smaller singular values ​​cannot be ignored in distinguishing different categories, because most discriminative classifiers rely on information in multiple dimensions.

[0100] In order to prevent the eigenvector of a certain dimension from excessively affecting the results due to its largest singular value, it needs to be compressed and decomposed. The SRL method helps balance the importance of eigenvectors by first applying SVD to the source domain feature matrix Fs and the target domain feature matrix Ft, extracting the largest m singular values, and then using SRL as a regularization term to act on these singular values. This method helps improve the generalization ability and discriminability of the model while maintaining the transferability of the features.

[0101] A cross-scene auditory attention decoding method based on a dual-branch domain adaptation network is proposed. The cross entropy loss between the predicted result and the true label is used as the first loss function loss1, and the specific regularization loss calculated by the domain adaptation module is used as the second loss function loss2; the final loss Loss obtained by the weighted linear combination of loss1 and loss2 is used as the optimization function.

[0102] The dual-branch domain adaptation network is trained and verified using the training set and the validation set; the model parameters of the network with the best prediction result of the validation set are saved and tested using the test set.

[0103] The tested two-branch domain adaptation network is used as a prediction model to identify EEG signals and predict the corresponding auditory attention decoding results. The loss1 cross entropy loss function in the final loss is used to calculate the classification loss of the source domain labeled data, which is expressed as:

[0104]

[0105] in, L ce represents the classification loss, y represents the true label, p Represents the probability that the sample predicted by the model belongs to the true category in the decision window.

[0106] The loss2 specific regularization loss SRL loss function in the final loss is:

[0107]

[0108] in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i Represents the i-th singular value among the first m largest singular values ​​of the second EEG feature. In practice, m is usually set to 1.

[0109] The SRL method first applies singular value decomposition to the feature matrix Fav for audio-video scenes and the feature matrix Fao for audio-only scenes to obtain the first m largest singular values. The shape of the feature tensors Fav and Fao is (H,D), where H is the batch size and D is the dimension of the features extracted by the two-branch network. Subsequently, SRL is used as a regularization term for the first m singular values. Minimizing the domain differences of singular values ​​can promote the alignment of cross-domain feature spaces, thereby reducing domain shift.

[0110] This paper introduces a specific regularization loss (SRL) method to minimize the data differences between different scenarios, thereby reducing domain shift. This method achieves better matching of feature space by optimizing the alignment of source and target domains on singular values, further improving the robustness and adaptability of the model in cross-scenario tasks. Experimental results show that this method has significant performance improvement over traditional baseline models in single-subject cross-scenario tasks.

[0111] In addition, the SRL method can be applied directly between the feature vectors of each batch, which makes it more flexible and effective in dealing with cross-scenario tasks. In this way, it can better adapt to the data characteristics in different scenarios and improve the adaptability and accuracy of the model. The application of this method is not only innovative in theory, but also shows great potential in practical applications, especially in scenarios that need to process multimodal data and cross-domain tasks.

[0112] The method for generating the cross-scene auditory attention decoding model in the above embodiment has the following technical effects:

[0113] 1. The present invention can simultaneously capture the time domain and spectrum-space features in EEG signals by combining the time feature branch and the spectrum residual branch. This multi-branch design not only enriches the feature representation, but also effectively reduces the model parameters and improves the computational efficiency through the KAN and spectrum residual modules. This method significantly enhances the decoding ability of the model in complex and changing scenarios, and helps to improve the overall performance of the BCI system.

[0114] 2. This paper introduces a specific regularization loss (SRL) method to minimize the data differences between different scenarios, thereby reducing domain shift. This method achieves better matching of feature space by optimizing the alignment of source and target domains on singular values, further improving the robustness and adaptability of the model in cross-scenario tasks. Experimental results show that this method has significant performance improvement over traditional baseline models in single-subject cross-scenario tasks.

[0115] 3. This invention demonstrates its strong flexibility and scalability in processing complex and changeable EEG data. The model effectively copes with nonlinear changes and noise interference in the data by efficiently extracting multi-band differential entropy (MBDE) features and extracting spectral-spatial information with a residual network. This feature enables the dual-branch domain-adaptive cross-scenario network to more accurately cope with challenges in practical applications when evaluating model robustness and generalization ability.

[0116] 4. The present invention achieves a 31.4% improvement in accuracy under a 1-second decision window, fully demonstrating its effectiveness in cross-scenario attention decoding tasks. At the same time, by optimizing the model structure and introducing efficient feature extraction methods, this method has also achieved significant results in reducing computational costs, providing strong support for the realization of real-time BCI systems.

[0117] In summary, this method, through its unique multi-branch design, specialized regularization loss method and efficient feature extraction technology, has demonstrated significant advantages in the cross-scene attention decoding task of a single subject, providing an important way to improve the practicality and accuracy of BCI systems.

[0118] As follows, some experiments are conducted to illustrate the effectiveness of the generation method of the cross-scene auditory attention decoding model provided by the present invention.

[0119] The dataset MM-AAD collected by the present invention is used for experiments. In order to quantitatively evaluate the auditory attention decoding results across subjects, the average precision is used as the evaluation indicator.

[0120] Table 1 Comparison of decoding accuracy of different models

[0121]

[0122] Table 1 provides a comparison of the decoding accuracy of different models in the auditory attention task. The data in Table 1 shows the performance of various models under different decision window lengths (0.5 seconds, 1 second, and 2 seconds). These models include baseline models and some domain adaptation methods. The following is a detailed description of the table content:

[0123] Baseline model:

[0124] MBSSFCC: Under all three decision windows, the decoding accuracy of the MBSSFCC model is 50.0%, 50.4%, and 51.0%, respectively. This shows that the performance of the model under different decision windows has almost no significant change; the decoding accuracy of the DBPNet model under the decision windows of 0.5 seconds, 1 second, and 2 seconds is 51.0%, 51.4%, and 55.3%, respectively. It can be seen that as the decision window increases, the performance of the model improves; this is also the method adopted by the present invention, namely, dual-branch feature extraction.

[0125] Domain adaptation method:

[0126] The decoding accuracy of the MCC method under three decision windows is 56.7%, 63.7% and 68.1% respectively. This shows that the performance of the MCC method under different decision windows is significantly improved, especially under longer decision windows. The decoding accuracy of the CDAN method under decision windows of 0.5 seconds, 1 second and 2 seconds is 70.0%, 80.0% and 81.9% respectively. This shows that the CDAN method has achieved high performance under all decision windows, and the performance is further improved as the decision window increases. The decoding accuracy of the CORAL method under three decision windows is 76.9%, 81.1% and 81.9% respectively. Similar to CDAN, the CORAL method performs better under longer decision windows. The method of the DBDANet invention has the highest decoding accuracy under decision windows of 0.5 seconds, 1 second and 2 seconds, which are 79.6%, 82.8% and 83.2% respectively. This shows that:

[0127] DBPNet performs slightly better than MBSSFCC under all decision windows, which may indicate that DBPNet has better feature extraction ability or model architecture when processing auditory data.

[0128] The performance of these domain adaptation methods improves as the decision window increases, and they generally outperform the baseline models, which shows that domain adaptation can significantly improve the model's ability to identify the target sound source.

[0129] The performance of CDAN is significantly better than MCC under all decision windows, which may mean that CDAN has stronger adaptability and robustness in processing data from different domains.

[0130] Although the performance of CDAN and CORAL is very close, CDAN performs slightly better than CORAL under longer decision windows, which may indicate that CDAN is more effective in utilizing long temporal information.

[0131] DBDANet outperforms CORAL at all decision windows, especially at 1 second and 2 second decision windows, which indicates that DBDANet may adopt a more advanced domain adaptation strategy or a more complex model structure.

[0132] The performance of all models improves as the decision window increases, but the improvement is more significant for domain adaptation methods, suggesting that they are better able to exploit information over longer time windows.

[0133] All domain adaptation methods outperform the baseline models at all decision windows, indicating that domain adaptation techniques are crucial to improving the performance of models in the auditory attention detection task.

[0134] DBDANet has the best performance under all decision windows, which may mean that DBDANet combines effective feature extraction techniques and domain adaptation strategies to make it more effective in processing complex auditory scenes.

[0135] This shows that as the decision window increases, the decoding accuracy of all models improves. This may be because longer decision windows provide more temporal information, which helps the model better understand the auditory scene.

[0136] Domain adaptation methods (such as CDAN, CORAL, and DBDANet) outperform baseline models (MBSSFCC and DBPNet) under all decision windows, which shows that these methods are effective in dealing with domain differences and improving model generalization ability. DBDANet achieves the best performance in all cases, which may mean that it has advantages in feature extraction and domain adaptation.

[0137] Table 2 Ablation experimental results of the model of the present invention

[0138]

[0139] As shown in Table 2: SRB is the spectrum residual module (frequency domain feature extraction branch) in the present invention, which is used to extract spectrum space features. In the ablation experiment, the DBDANet model achieved 79.6%, 82.8% and 83.2% decoding accuracy in the decision window of 0.5 seconds, 1 second and 2 seconds, respectively, under the complete configuration. When the KAN component is removed, the model performance is slightly reduced, and the accuracy is reduced by 1.6, 0.6 and 0.5 percentage points, respectively; and when the SRB component is removed, the accuracy is reduced by 1.1, 0.5 and 0.3 percentage points, respectively, indicating that both components have a positive effect on improving the decoding accuracy of the model. When the KAN and SRB components are removed at the same time, the model performance is more significantly reduced, and the accuracy is reduced by 2.4, 1.4 and 0.6 percentage points, respectively. This result highlights the importance of the KAN and SRB components in the model and their complementarity in improving performance. Although removing these components will lead to performance degradation, the DBDANet model can still maintain a high accuracy even in the absence of key components, which reflects the overall robustness of the model. Both KAN and SRB components contribute positively to the performance of the DBDANet model. Removing either component results in a drop in accuracy, indicating that both components are important factors in achieving high model accuracy. The KAN component has a slightly greater impact on model performance than the SRB component. When removing KAN, the drop in model performance is slightly greater than when removing SRB. The drop in model performance is greatest when both KAN and SRB are removed, indicating that the KAN and SRB components may have complementary effects in improving model performance. Despite the drop in performance caused by removing these components, the DBDANet model is still able to maintain relatively high accuracy even in the absence of key components, which may indicate good overall robustness of the model.

[0140] Table 3 Comparison of the number of parameters of different models

[0141]

[0142] Table 3 compares the parameters of DBDANet with those of MBSSFCC and DBPNet, and finds that DBDANet requires a total of 0.74M trainable parameters, which is about 1.2 times less than DBPNet and 113 times less than MBSSFCC. This shows that KAN reduces the number of parameters for model training. The significant reduction in computational cost makes it more suitable for low-power hearing aids.

[0143] In an embodiment of the present invention, a cross-scene auditory attention decoding method is also provided, including:

[0144] The EEG signal of the subject is input into the trained auditory attention decoding model, and the auditory attention of the subject is obtained through the auditory attention decoding model. The auditory attention decoding model is obtained by the generation method of the cross-scene auditory attention decoding model provided by the present invention.

[0145] In an embodiment of the present invention, a device for generating a cross-scene auditory attention decoding model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below may implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0146] The generation device of the cross-scene auditory attention decoding model includes:

[0147] The signal acquisition module is used to acquire source domain EEG signals and target domain EEG signals, where the source domain EEG signals and the target domain EEG signals are respectively EEG signals generated by the subject in different scenarios.

[0148] The pre-training module is used to pre-train the auditory attention decoding model through the source domain EEG signal. The auditory attention decoding model includes a feature extraction module, a classification module and a domain adaptation module. The pre-training steps include:

[0149] Extract features of the source domain EEG signal through a feature extraction module to obtain a first EEG feature;

[0150] Predict and classify the first EEG feature through the classification module to obtain the first classification loss;

[0151] Pre-training the auditory attention decoding model by optimizing the first classification loss;

[0152] The domain adaptation training module is used to perform domain adaptation training on the pre-trained auditory attention decoding model through the target domain EEG signal. The steps of domain adaptation training include:

[0153] The feature extraction module is used to extract the features of the target domain EEG signal to obtain the second EEG feature;

[0154] The second EEG feature is predicted and classified through the classification module to obtain the second classification loss;

[0155] The domain adaptation module is used to perform singular value decomposition on the first EEG feature and the second EEG feature, and the first m largest singular values ​​are retained and the regularization loss is calculated. L sr :

[0156]

[0157] in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i represents the ith singular value among the first m largest singular values ​​of the second EEG feature;

[0158] The auditory attention decoding model is trained for domain adaptation by optimizing the first classification loss and regularization loss.

[0159] The present invention proposes a device for generating a cross-scene auditory attention decoding model, which mainly adopts a regularization loss (SRL) as a domain adaptation loss, can minimize the data differences between different scenes, thereby reducing domain shift, and can generate an auditory attention decoding model that adapts to different scenes. It solves the problem that the traditional method of generating an auditory attention decoding model cannot generate an auditory attention decoding model that can adapt to different scenes.

[0160] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0161] In an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for generating a cross-scene auditory attention decoding model proposed in the present invention.

[0162] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0163] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

Claims

1. A method for generating a cross-scene auditory attention decoding model, characterized in that: include: Acquire a source domain EEG signal and a target domain EEG signal, wherein the source domain EEG signal and the target domain EEG signal are respectively EEG signals generated by a subject in different scenarios; The auditory attention decoding model is pre-trained by using source domain EEG signals, wherein the auditory attention decoding model includes a feature extraction module, a classification module and a domain adaptation module, and the pre-training steps include: Extracting features of the source domain EEG signal by the feature extraction module to obtain a first EEG feature; Predicting and classifying the first EEG feature by the classification module to obtain a first classification loss; Optimizing the first classification loss to pre-train the auditory attention decoding model; The pre-trained auditory attention decoding model is subjected to domain adaptation training using the target domain EEG signal. The steps of the domain adaptation training include: Extracting features of the target domain EEG signal by the feature extraction module to obtain a second EEG feature; Predicting and classifying the second EEG feature by the classification module to obtain a second classification loss; The domain adaptation module performs singular value decomposition on the first EEG feature and the second EEG feature and retains the first m maximum singular values ​​and calculates the regularization loss. L sr : ， in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i represents the ith singular value among the first m largest singular values ​​of the second EEG feature; The first classification loss and the regularization loss are optimized to perform domain adaptation training on the auditory attention decoding model.

2. The method for generating a cross-scene auditory attention decoding model according to claim 1, characterized in that: Also includes: The source domain EEG signal and the target domain EEG signal are subjected to data preprocessing, wherein the data preprocessing step comprises: The EEG signal is filtered by a power frequency notch filter and a finite impulse response filter; Down-sampling the filtered EEG signal to the target frequency; Independent component analysis was performed on the downsampled EEG signals and the principal components were retained.

3. The method for generating a cross-scene auditory attention decoding model according to claim 1, characterized in that: The different scenarios include audio stimulation scenarios and audio-video stimulation scenarios.

4. The method for generating a cross-scene auditory attention decoding model according to claim 1, characterized in that: The feature extraction module includes a time domain feature extraction branch and a frequency domain feature extraction branch; The input of the time domain feature extraction branch is the model input after common spatial mode filtering, and the input of the feature extraction module is the multi-band differential entropy feature extracted from the model input; The output of the time domain feature extraction branch and the output of the frequency domain feature extraction branch are feature fused to obtain the output of the feature extraction module.

5. The method for generating a cross-scene auditory attention decoding model according to claim 4, characterized in that: The time domain feature extraction branch includes a transformer, a KAN network and a fully connected block connected in sequence. The expression of the KAN network is: ， in, x ′ represents the network input, SplineOpt( x ′) represents the model output, BaseActivation() represents the baseline activation function, W base represents the trainable weight parameter matrix, B () represents B basis functions, W spline represents the spline weight matrix; The frequency domain feature extraction branch includes a first 3D convolution, a residual network, two second 3D convolutions and an average pooling layer connected in sequence.

6. The method for generating a cross-scene auditory attention decoding model according to claim 5, characterized in that: The residual network includes a first residual block, a second residual block and a third residual block connected in sequence; The first 3D convolution uses a convolution kernel of size 7x7, a stride of 2, and a padding of 3 and a maximum pooling operation of a pooling window of 3x3, a stride of 2, and a padding of 1; The first 3D convolution is configured to expand the number of channels of the feature map from 1 to 32; The first residual block is configured to expand the number of channels of the feature map from 32 to 64; The second residual block is configured to expand the number of channels of the feature map from 64 to 128; The third residual block is configured to expand the number of channels of the feature map from 128 to 256.

7. The method for generating a cross-scene auditory attention decoding model according to claim 1, characterized in that: The first classification loss and the second classification loss are both calculated by a cross entropy loss function, and the cross entropy loss function is: ， in, L ce represents the classification loss, y represents the true label, p Represents the probability that the sample predicted by the model belongs to the true category in the decision window.

8. A cross-scene auditory attention decoding method, characterized in that: include: The EEG signal of the subject is input into the trained auditory attention decoding model, and the auditory attention of the subject is obtained through the auditory attention decoding model. The auditory attention decoding model is obtained by the generation method of the cross-scene auditory attention decoding model described in any one of claims 1-7.

9. A device for generating a cross-scene auditory attention decoding model, characterized in that: include: A signal acquisition module, used to acquire a source domain EEG signal and a target domain EEG signal, wherein the source domain EEG signal and the target domain EEG signal are respectively EEG signals generated by a subject in different scenarios; A pre-training module is used to pre-train an auditory attention decoding model using source domain EEG signals. The auditory attention decoding model includes a feature extraction module, a classification module, and a domain adaptation module. The pre-training steps include: Extracting features of the source domain EEG signal by the feature extraction module to obtain a first EEG feature; Predicting and classifying the first EEG feature by the classification module to obtain a first classification loss; Optimizing the first classification loss to pre-train the auditory attention decoding model; The domain adaptation training module is used to perform domain adaptation training on the pre-trained auditory attention decoding model through the target domain EEG signal. The steps of the domain adaptation training include: Extracting features of the target domain EEG signal by the feature extraction module to obtain a second EEG feature; Predicting and classifying the second EEG feature by the classification module to obtain a second classification loss; The domain adaptation module performs singular value decomposition on the first EEG feature and the second EEG feature and retains the first m maximum singular values ​​and calculates the regularization loss. L sr : ， in, σ v，i represents the ith singular value among the first m largest singular values ​​of the first EEG feature, σ o，i represents the ith singular value among the first m largest singular values ​​of the second EEG feature; The first classification loss and the regularization loss are optimized to perform domain adaptation training on the auditory attention decoding model.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for generating a cross-scene auditory attention decoding model according to any one of claims 1 to 7.

Images