Auditory attention decoding method, auditory enhancement method, hearing aid and sound system

Abstract

The invention discloses an auditory attention decoding method, an auditory enhancement method, a hearing aid and a sound system, and the decoding method comprises the steps: extracting a differential entropy feature tensor in an electroencephalogram signal, inputting the differential entropy feature tensor into a frequency domain branch of a dual-path time-frequency fusion neural network model, and inputting the electroencephalogram signal subjected to CSP processing into a time domain branch of the model; the model outputs an auditory attention decoding result; the frequency domain branch is configured with a two-dimensional convolution layer, a maximum pooling layer, a three-dimensional convolution residual module, an average pooling layer and a feature paving layer so as to finally output a first output feature; the time domain branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a splicing module and a linear layer so as to finally output a second output feature; and finally, fusing the first output feature and the second output feature, and comprehensively capturing multi-scale time sequence features and space-spectrum coupling information, thereby realizing accurate auditory attention recognition in a short time window.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an auditory attention decoding method, an auditory enhancement method, a hearing aid, and an audio system. Background Technology

[0002] In multi-source environments, the ability to selectively focus on and process a specific sound signal is a fundamental capability of the human cognitive system, a phenomenon often referred to as the "cocktail party effect." This ability is based on language comprehension and is crucial for human social interaction and information processing. However, in complex noisy environments, especially for people with hearing impairments or neurocognitive disorders, the ability to focus on specific sound signals is often significantly limited, greatly affecting their acquisition and understanding of external information.

[0003] Electroencephalography (EEG), as a high-temporal-resolution, non-invasive, and convenient method of recording brain activity, can accurately reflect the neural activity of the brain in the process of receiving and processing external information, and has become an important tool for studying the brain's auditory attention mechanism.

[0004] The application of electroencephalogram (EEG) signals in auditory attention decoding (AAD) is becoming increasingly widespread. Traditional AAD methods typically rely on linear models to process EEG signals. For example, Geirnaert et al. proposed a decoding method based on Common Spatial Pattern (CSP), which can effectively classify signals by extracting spatial features, but its ability to handle complex nonlinear mapping relationships is weak. Another example is the Temporal Response Function (TRF) method, which is widely used in temporal modeling of EEG. The TRF method proposed by Crosse et al. improves decoding accuracy to some extent by modeling the relationship between time-series signals and stimulus signals.

[0005] However, these traditional AAD methods typically require long decision windows to achieve accurate decoding in multi-source environments. Secondly, although time-domain EEG signals contain rich dynamic temporal information, their spectral-spatial characteristics in the time domain are limited and their correlation with human spatial attention is weak. For example, some studies have used models such as Long Short-Term Memory (LSTM) networks to capture long-term temporal dependencies in EEG signals, and these methods have achieved some decoding results when processing time-series data. However, these methods often neglect the spatial distribution characteristics of EEG signals—that is, the dynamic patterns of different brain regions in receiving, processing, and responding to auditory stimuli. Furthermore, the presence of significant noise and outliers in EEG signals makes decoding models susceptible to interference when capturing long-term potential dependencies, thus affecting decoding performance.

[0006] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0007] The purpose of this invention is to provide an effective auditory attention decoding method that combines time-domain features and frequency-domain features. It extracts the temporal dynamic information and spectral spatial features from the EEG signal, and fuses the two to comprehensively capture the multi-scale temporal features and spatial-spectral coupling information in the EEG signal, thereby achieving accurate auditory attention recognition within a short time window.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An auditory attention decoding method includes the following steps: Acquire real-time EEG signals; Extract the differential entropy feature tensor from the frequency band of the real-time EEG signal, and perform CSP processing on the EEG signal to obtain CSP features; The differential entropy feature tensor and CSP feature are input into a pre-constructed dual-path time-frequency fusion neural network model; The dual-path time-frequency fusion neural network model outputs the auditory attention decoding result; The dual-path time-frequency fusion neural network model is constructed in the following way: by constructing a time-domain feature extraction branch, a frequency-domain feature extraction branch, and a feature fusion module, wherein: The frequency domain feature extraction branch is configured with a two-dimensional convolutional layer, a max pooling layer, a three-dimensional convolutional residual module, an average pooling layer, and a feature flattening layer to finally output the first output feature; The temporal feature extraction branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a concatenation module, and a linear layer to finally output the second output feature; The feature fusion module is configured to fuse the first output feature and the second output feature to obtain a fused feature vector, which is related to the auditory attention decoding result.

[0009] Furthermore, following any or a combination of the aforementioned technical solutions, the two-dimensional convolutional layer is configured to extract spatial texture features from the two-dimensional spatial image obtained by unfolding the differential entropy feature tensor along the electrode channel dimension, thereby obtaining a spatial feature map; the max pooling layer is configured to embed the original channel dimension features into the spatial feature map, thereby obtaining a tensor representation adapted to the 3D convolutional input format; the three-dimensional convolutional residual module is stacked in multiple consecutive levels, and is configured to extract the coupling relationship between frequency domain features and spatial and spectral domain structures level by level, thereby obtaining the output feature tensor of the last residual module; the average pooling layer is configured to perform global average pooling through two dimensions of frequency and time, thereby obtaining the channel-compressed representation vector; the feature flattening layer is configured to linearly transform and map the channel-compressed representation vector to the first output feature of the frequency domain feature extraction branch.

[0010] Furthermore, following any or a combination of the aforementioned technical solutions, the feature embedding layer is configured to extract spatial structure features along the channel dimension of the CSP features and compress them to a set number of channels to form a unified temporal representation; the first attention decoding module receives the unified temporal representation and outputs the attention decoding result to the second attention decoding module; the splicing module splices the attention decoding result of the second attention decoding module with the attention decoding result of the first attention decoding module along the channel dimension to obtain a spliced ​​result; the linear layer linearly transforms and maps the spliced ​​result to the second output feature of the temporal feature extraction branch.

[0011] Furthermore, following any one or a combination of the aforementioned technical solutions, the three-dimensional convolutional residual modules are stacked continuously at N levels. The coupling relationship between the frequency domain features extracted by the three-dimensional convolutional residual modules in the spatial and spectral domain structures includes: , ,in, Indicates the first The output feature tensor of the residual module, Indicates the first The input feature tensor of the residual module, This represents the three-dimensional convolution function corresponding to a 1×1×1 convolution kernel. This represents the three-dimensional convolution function corresponding to a 3×3×3 convolution kernel. Represents a linear correction function. This represents the output feature tensor of the last-stage residual module.

[0012] Furthermore, following any one or a combination of the aforementioned technical solutions, the two-dimensional convolutional layer has a 7×7 convolutional kernel, which yields its spatial feature map using the following formula: ,in, This represents the spatial feature map output by the two-dimensional convolutional layer. The two-dimensional convolution function represents a two-dimensional convolutional layer. The feature tensor of the difference entropy is represented as: ,in, Indicates the number of electrode channels. Indicates the number of discrete frequency bands. Indicates the number of frames in the sliding time window; The spatial feature map pass The activation function is then max-pooled through a max-pooling layer to obtain the tensor representation of the adapted 3D convolutional input format, which serves as the input feature tensor of the first-level residual module. .

[0013] Furthermore, following any one or a combination of the aforementioned technical solutions, the average pooling layer obtains the channel-compressed representation vector using the following formula. : ,in, This represents a global average pooling function that performs pooling on both frequency and time dimensions. The feature tiling layer is linearly transformed and mapped using the following formula: ,in, This represents the first output feature of the frequency domain feature extraction branch. This represents the trainable parameters.

[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, the frequency domain feature extraction branch is further configured with a 1×1 convolutional layer and a batch normalization layer. The two-dimensional convolutional layer, the max pooling layer, the three-dimensional convolutional residual module, the 1×1 convolutional layer, the batch normalization layer, the average pooling layer, and the feature flattening layer are sequentially connected. The 1×1 convolutional layer is configured to perform a 1×1 convolution operation on the output feature tensor of the last-stage residual module, and the batch normalization layer is configured to perform a batch normalization operation on the result of the 1×1 convolution operation.

[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, both the first attention decoding module and the second attention decoding module obtain the corresponding attention decoding results in the following manner: The input timing sequence is subjected to layer normalization and a first gated projection branch and a second gated projection branch are generated, wherein the first gated projection branch is denoted as... The second gated projection branch is denoted as ,in, Indicates the layer normalization processing function, This indicates the timing of the input to the attention decoding module. Represents the projection function; First gated projection branch Local dependencies are extracted using one-dimensional convolution, and activation values ​​are obtained using a non-linear activation function. ,in, Represents a one-dimensional convolution function. Represents a nonlinear activation function; Second gated projection branch The activation value is obtained through a nonlinear activation function. ; The weighted fusion result is obtained by performing weighted fusion using the following gating mechanism. ,in, This represents element-wise multiplication. The weighted fusion result is then subjected to layer normalization and projection processing in sequence to obtain the attention decoding result.

[0016] Furthermore, following any one or a combination of the aforementioned technical solutions, a unified temporal representation is formed in the feature embedding layer of the temporal feature extraction branch, defined as follows: ,in, This indicates the number of channels to be set. Indicates the number of frames in the sliding time window; The unified timing representation As the timing input to the first attention decoding module, the attention decoding result of the first attention decoding module is calculated and denoted as... ; attention decoding results As the timing input to the second attention decoding module, the attention decoding result of the second attention decoding module is calculated and denoted as... ; The splicing module obtains the splicing result using the following formula. ,in, This represents the concatenation function.

[0017] Furthermore, following any one or a combination of the aforementioned technical solutions, the linear layer obtains the second output feature in the following manner: ,in, This represents the second output feature of the time-domain feature extraction branch. This represents the trainable parameters.

[0018] Furthermore, following any one or a combination of the aforementioned technical solutions, the feature fusion module will convert the first output feature... With the second output feature The fused feature vector is obtained by concatenation. ; The fused feature vector is related to the auditory attention decoding result, including obtaining the final classification result through a fully connected layer. ,in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This represents the predicted probability of the dual-path time-frequency fusion neural network model for the current input.

[0019] Furthermore, following any one or a combination of the aforementioned technical solutions, the dual-path time-frequency fusion neural network model is trained in the following manner: Multiple learning samples are collected. Each learning sample includes the differential entropy feature tensor corresponding to the same EEG signal and the CSP feature after CSP processing, as well as its corresponding classification label. Based on the collected learning samples, training sample sets and test sample sets are formed respectively; The dual-path time-frequency fusion neural network model is iteratively learned using the training sample set, and the network model is supervised training using the binary cross-entropy loss function; The prediction accuracy of the dual-path time-frequency fusion neural network model is calculated using the test sample set. When the accuracy reaches a preset threshold and / or a preset termination condition is met, the iterative learning stops.

[0020] According to another aspect of the present invention, a hearing enhancement method is provided, comprising the following steps: Using the auditory attention decoding method described above, the classification results of auditory attention decoding are determined, which characterize the location where auditory attention is more concentrated; Based on the classification results of the auditory attention decoding, the collected sound signals are enhanced: the enhancement level of sound signals from directions where auditory attention is more concentrated is higher than that of sound signals from other directions.

[0021] According to another aspect of the present invention, a hearing aid is provided, comprising a left hearing aid and / or a right hearing aid, wherein the gain and / or volume of the hearing aid are controlled using the hearing enhancement method described above, comprising: The classification results of auditory attention decoding are divided into left and right categories; If the classification result of the auditory attention decoding is a left classification result, then increase the gain and / or volume of the left hearing aid; And / or, if the classification result of the auditory attention decoding is a right classification result, then increase the gain and / or volume of the right hearing aid.

[0022] According to another aspect of the present invention, a multi-channel audio system is provided, wherein the audio system is sound-tuned using the auditory enhancement method described above, comprising: The audio system includes N channels, and the classification results of auditory attention decoding are divided into N classification results corresponding to the channels of the audio system. Based on the classification results of the auditory attention decoding, the channel level of the corresponding channel is increased.

[0023] As can be seen from the above, the construction method of the dual-path time-frequency fusion neural network model includes the following steps: A frequency domain feature extraction branch is constructed: a two-dimensional convolutional layer, a max pooling layer, a three-dimensional convolutional residual module, an average pooling layer, and a feature tiling layer are configured on this branch. The two-dimensional convolutional layer is configured to extract spatial texture features from the two-dimensional spatial image obtained by unfolding the differential entropy feature tensor extracted from the EEG signal along the electrode channel dimension, thereby obtaining a spatial feature map. The feature map is then sequentially processed by the max pooling layer, the convolutional processing of the three-dimensional convolutional residual module, the pooling processing of the average pooling layer, and the linear transformation mapping processing of the feature tiling layer to obtain the first output feature representing the frequency domain features. A temporal feature extraction branch is constructed: This branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a concatenation module, and a linear layer. The feature embedding layer is configured to extract spatial structural features along the channel dimension from the CSP-processed EEG signal and compress them to a set number of channels to form a unified temporal representation. The first attention decoding module receives the unified temporal representation and outputs the attention decoding result to the second attention decoding module. The concatenation module concatenates the attention decoding result of the second attention decoding module with the attention decoding result of the first attention decoding module along the channel dimension to obtain a concatenated result. The linear layer linearly transforms the concatenated result into a second output feature representing the temporal domain features. A feature fusion module is constructed, which is configured to fuse the first output feature and the second output feature to obtain a fused feature vector.

[0024] Furthermore, following any one or a combination of the aforementioned technical solutions, the first attention decoding module is constructed to include: The first-level normalization unit is configured to perform layer normalization on the unified time-series representation: ,in, Indicates the layer normalization processing function, This represents the temporal representation formed by the feature embedding layer; The first gated projection unit is configured to generate a first gated projection branch from the output of the first layer normalization unit, denoted as... ,in, Represents the projection function; The second gated projection unit is configured to generate a second gated projection branch from the output of the first layer normalization unit, denoted as... ; The first convolutional unit is configured to project a first gated branch. Perform one-dimensional convolution processing: ,in, This represents a one-dimensional convolution function; The first nonlinear activation unit is configured to perform nonlinear activation processing on the output of the first convolutional unit to obtain activation values. ,in, Represents a nonlinear activation function The second nonlinear activation unit is configured to perform nonlinear activation processing on the output of the second gated projection unit to obtain an activation value. ; The first summation unit is configured to perform weighted fusion through the following gating mechanism to obtain the weighted fusion result. ,in, This represents element-wise multiplication. The second-level normalization unit is configured to normalize the weighted fusion results. Perform layer normalization; The mapping unit is configured to project the output of the second-layer normalization unit to obtain the attention decoding result, denoted as... .

[0025] Furthermore, following any one or a combination of the aforementioned technical solutions, the second attention decoding module is constructed to include: The third-layer normalization unit is configured to process the attention decoding result output by the first attention decoding module. Perform layer normalization: ; The third gating projection unit is configured to generate a third gating projection branch from the output of the third-layer normalization unit, denoted as... ; The fourth gated projection unit, configured to generate a fourth gated projection branch from the output of the third-layer normalization unit, is denoted as... ; The second convolutional unit is configured to project the third gated branch. Perform one-dimensional convolution processing: ,in, This represents a one-dimensional convolution function; The third nonlinear activation unit is configured to perform nonlinear activation processing on the output of the second convolutional unit to obtain activation values. ,in, Represents a nonlinear activation function; The fourth nonlinear activation unit is configured to perform nonlinear activation processing on the output of the fourth gated projection unit to obtain the activation value. ; The second summing unit is configured to perform weighted fusion through the following gating mechanism to obtain the weighted fusion result. ,in, This represents element-wise multiplication. The fourth normalization unit is configured to normalize the weighted fusion results. Perform layer normalization; The mapping unit is configured to project the output of the fourth-layer normalization unit to obtain the attention decoding result, denoted as... .

[0026] Furthermore, following any one or a combination of the aforementioned technical solutions, the temporal representation formed by the feature embedding layer is defined as follows: ,in, This indicates the number of channels to be set. Indicates the number of frames in the sliding time window; The splicing module of the temporal feature extraction branch obtains the splicing result using the following formula. ,in, This represents the concatenation function.

[0027] Furthermore, following any or a combination of the aforementioned technical solutions, the temporal feature extraction branch is further configured with a first average pooling layer, a first linear layer, a second average pooling layer, and a second linear layer. The first average pooling layer and the first linear layer are positioned between the first attention decoding module and the second attention decoding module. The attention decoding result of the first attention decoding module is then subjected to average pooling and linear transformation processing in sequence, and the processed attention decoding result is then output to the second attention decoding module. The second average pooling layer and the second linear layer are set between the second attention decoding module and the splicing module. Then, the attention decoding result of the second attention decoding module is processed by average pooling and linear transformation in sequence. Finally, the processed attention decoding result output by the second linear layer is spliced ​​with the processed attention decoding result output by the first linear layer to obtain the spliced ​​result.

[0028] Furthermore, following any one or a combination of the aforementioned technical solutions, the linear layer of the temporal feature extraction branch processes the splicing result in the following manner. Perform a linear transformation mapping to obtain the second output feature: ,in, This represents the second output feature of the time-domain feature extraction branch. This represents the trainable parameters.

[0029] Furthermore, following any or a combination of the aforementioned technical solutions, the max pooling layer of the frequency domain feature extraction branch is configured to embed the original channel-dimensional features into the spatial feature map to obtain a tensor representation adapted to the 3D convolution input format; the three-dimensional convolutional residual module is stacked in multiple consecutive levels and is configured to extract the coupling relationship between the frequency domain features in the spatial structure and the spectral domain structure level by level to obtain the output feature tensor of the last residual module; the average pooling layer is configured to perform global average pooling through two dimensions of frequency and time to obtain the channel-compressed representation vector; the feature flattening layer is configured to linearly transform and map the channel-compressed representation vector to the first output feature of the frequency domain feature extraction branch.

[0030] Furthermore, following any one or a combination of the aforementioned technical solutions, the three-dimensional convolutional residual modules are stacked continuously at N levels. The coupling relationship between the frequency domain features extracted by the three-dimensional convolutional residual modules in the spatial and spectral domain structures includes: , ,in, Indicates the first The output feature tensor of the residual module, Indicates the first The input feature tensor of the residual module, This represents the three-dimensional convolution function corresponding to a 1×1×1 convolution kernel. This represents the three-dimensional convolution function corresponding to a 3×3×3 convolution kernel. Represents a linear correction function. This represents the output feature tensor of the last-stage residual module.

[0031] Furthermore, following any one or a combination of the aforementioned technical solutions, the two-dimensional convolutional layer has a 7×7 convolutional kernel, which yields its spatial feature map using the following formula: ,in, This represents the spatial feature map output by the two-dimensional convolutional layer. The two-dimensional convolution function represents a two-dimensional convolutional layer. The feature tensor of the difference entropy is represented as: ,in, Indicates the number of electrode channels. Indicates the number of discrete frequency bands. Indicates the number of frames in the sliding time window; The spatial feature map pass The activation function is then max-pooled through a max-pooling layer to obtain the tensor representation of the adapted 3D convolutional input format, which serves as the input feature tensor of the first-level residual module. .

[0032] Furthermore, following any one or a combination of the aforementioned technical solutions, the average pooling layer obtains the channel-compressed representation vector using the following formula. : ,in, This represents a global average pooling function that performs pooling on both frequency and time dimensions. The feature tiling layer is linearly transformed and mapped using the following formula: ,in, This represents the first output feature of the frequency domain feature extraction branch. This represents the trainable parameters.

[0033] Furthermore, following any one or a combination of the aforementioned technical solutions, the frequency domain feature extraction branch is further configured with a 1×1 convolutional layer and a batch normalization layer. The two-dimensional convolutional layer, the max pooling layer, the three-dimensional convolutional residual module, the 1×1 convolutional layer, the batch normalization layer, the average pooling layer, and the feature flattening layer are sequentially connected. The 1×1 convolutional layer is configured to perform a 1×1 convolution operation on the output feature tensor of the last-stage residual module, and the batch normalization layer is configured to perform a batch normalization operation on the result of the 1×1 convolution operation.

[0034] Furthermore, following any one or a combination of the aforementioned technical solutions, the feature fusion module will convert the first output feature... With the second output feature The fused feature vector is obtained by concatenation. ; The final classification result is obtained through a fully connected layer. ,in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This represents the predicted probability of the dual-path time-frequency fusion neural network model for the current input.

[0035] Accordingly, the training method for the dual-path time-frequency fusion neural network model includes the following steps: A dual-path time-frequency fusion neural network model is constructed using the method described above; Multiple learning samples are collected. Each learning sample includes the differential entropy feature tensor corresponding to the same EEG signal and the CSP feature after CSP processing, as well as its corresponding classification label. Based on the collected learning samples, training sample sets and test sample sets are formed respectively; The dual-path time-frequency fusion neural network model is iteratively learned using the training sample set, and the network model is supervised training using the binary cross-entropy loss function; The prediction accuracy of the dual-path time-frequency fusion neural network model is calculated using the test sample set. When the accuracy reaches a preset threshold and / or a preset termination condition is met, the iterative learning stops.

[0036] The beneficial effects of the technical solution provided by this invention are as follows: a. The neural network model for auditory attention decoding extracts temporal dynamic information and spectral spatial features from EEG signals by designing time-domain and frequency-domain branches in parallel, and then fuses the two to comprehensively capture multi-scale temporal features and spatial-spectral coupling information in EEG signals. This effectively overcomes the limitation of a single feature domain in traditional methods and avoids information loss caused by the independence of time-domain and frequency-domain information. b. Introduce a multi-scale Mamba module within the time-domain branch and adopt an efficient state-space modeling approach to better capture long-term dependencies in EEG signals, thereby achieving accurate auditory attention recognition within a short time window; c. Frequency domain branching based on 3D convolution and residual connections can effectively capture the spatial-spectral coupling characteristics in EEG signals, thereby improving decoding accuracy and model stability. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating an auditory attention decoding method provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of the architecture of a dual-path time-frequency fusion neural network model provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of the architecture of an attention decoding module on a temporal path branch provided as an exemplary embodiment of the present invention; Figure 4 A schematic flowchart of an auditory enhancement method provided as an exemplary embodiment of the present invention; Figure 5 A flowchart of a hearing enhancement method applied to a hearing aid, provided as an exemplary embodiment of the present invention; Figure 6 A flowchart illustrating a method for constructing a dual-path time-frequency fusion neural network model, provided as an exemplary embodiment of the present invention; Figure 7 This is a flowchart illustrating a training method for a dual-path time-frequency fusion neural network model provided as an exemplary embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0041] EEG signals contain rich information in both time and frequency, especially their frequency domain features, which can reveal different brain activity states and cognitive characteristics. In recent years, many studies have focused on the frequency domain features of EEG signals, utilizing deep learning methods such as Convolutional Neural Networks (CNNs) for spectral feature extraction and decoding. For example, CNNs are used to extract differential entropy (DE) values ​​in the EEG frequency bands and project them onto a two-dimensional topological map to reveal spatial brain activity, achieving good results. However, these methods often neglect the dynamic temporal information in EEG signals. Since EEG signals are essentially time-series data, some studies have begun to introduce attention mechanisms to capture temporal dependencies, thereby improving decoding performance.

[0042] In one embodiment of the present invention, an auditory attention decoding method is provided, such as... Figure 1 As shown, the decoding method includes the following steps: Acquire real-time EEG signals; Extract the differential entropy feature tensor from the frequency band of real-time EEG signals, and perform CSP processing on the EEG signals to obtain CSP features; Input the differential entropy feature tensor and CSP features into a pre-constructed dual-path time-frequency fusion neural network model; The dual-path time-frequency fusion neural network model outputs the auditory attention decoding result; The dual-path time-frequency fusion neural network model is constructed as follows: a time-domain feature extraction branch, a frequency-domain feature extraction branch, and a feature fusion module are built, such as... Figure 2 As shown: The frequency domain feature extraction branch is configured with a 2D convolutional layer, a max pooling layer, a 3D convolutional residual module, an average pooling layer, and a feature flattening layer to ultimately output the first output feature. Specifically, the 2D convolutional layer is configured to extract spatial texture features from the 2D spatial image obtained by unfolding the differential entropy feature tensor along the electrode channel dimension, resulting in a spatial feature map. The max pooling layer is configured to embed the original channel-dimensional features into the spatial feature map, resulting in a tensor representation adapted to the 3D convolutional input format. The 3D convolutional residual module is stacked in multiple consecutive levels and is configured to extract the coupling relationship between frequency domain features in the spatial and spectral domain structures level by level, resulting in the output feature tensor of the last residual module. The average pooling layer is configured to perform global average pooling through both frequency and time dimensions, resulting in a channel-compressed representation vector. The feature flattening layer is configured to linearly transform and map the channel-compressed representation vector to the first output feature of the frequency domain feature extraction branch, denoted as . .

[0043] The temporal feature extraction branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a concatenation module, and a linear layer to ultimately output a second output feature. Specifically, the feature embedding layer is configured to extract spatial structure features along the channel dimension of the CSP features and compress them to a set number of channels to form a unified temporal representation. The attention decoding result of the first attention decoding module is input to the second attention decoding module. The concatenation module concatenates the attention decoding results of the second and first attention decoding modules along the channel dimension to obtain the concatenated result. The linear layer linearly transforms the concatenated result to map it to the second output feature of the temporal feature extraction branch, denoted as... .

[0044] The feature fusion module is configured to combine the first output features With the second output feature The signals are fused to obtain a fused feature vector, which is correlated with the auditory attention decoding result. Multi-scale temporal information fusion is achieved through feature concatenation, comprehensively capturing the spatiotemporal coupling features in the EEG signal.

[0045] The working process of the frequency domain feature extraction branch is first described below: In the frequency domain branch, to improve the model's ability to model joint time-frequency-space features, this embodiment uses a continuously stacked N-level 3D convolutional residual module as the backbone structure. Each level of the 3D convolutional residual module consists of a set of continuous... With two The network is constructed using convolutional operations and employs residual connections to preserve original feature information and enhance the stability of network training. For example... Figure 2In the illustrated embodiment, the three-dimensional convolutional residual module is stacked in three consecutive levels, extracting the deep coupling relationship between frequency domain features and spatial and spectral domain structures layer by layer. , This represents the output feature tensor of the last-stage residual module. The introduction of the residual path not only avoids feature degradation and gradient vanishing, but also preserves and enhances spectral features at different scales, providing a more discriminative frequency domain representation for subsequent time-frequency fusion modules.

[0046] The output feature tensor of the first-level residual module is obtained in the following way: The two-dimensional convolutional layer of the frequency domain feature extraction branch has a 7×7 convolutional kernel, which obtains the spatial feature map using the following formula: ,in, This represents the spatial feature map output by a two-dimensional convolutional layer. The two-dimensional convolution function represents a two-dimensional convolutional layer. The feature tensor of the difference entropy is represented as: ,in, Indicates the number of electrode channels. Indicates the number of discrete frequency bands. Indicates the number of frames in the sliding time window; Spatial feature map pass The activation function is applied and max-pooling is performed through a max-pooling layer to obtain a tensor representation adapted to the 3D convolutional input format, which serves as the input feature tensor for the first-level residual module. .

[0047] The coupling relationship between the spatial structure and spectral structure of the stepwise extraction of frequency domain features by the 3D convolutional residual module includes: , ,in, Indicates the first The output feature tensor of the residual module, Indicates the first The input feature tensor of the residual module, This represents the three-dimensional convolution function corresponding to a 1×1×1 convolution kernel. This represents the three-dimensional convolution function corresponding to a 3×3×3 convolution kernel. Represents a linear correction function. This represents the output feature tensor of the last stage residual module. Assuming a fixed number of channels of 32, the output tensor of the three stacked residual modules is... , Indicates the number of discrete frequency bands. Indicates the number of frames in the sliding time window Figure 2 The three-stage stacked 3D convolutional residual module shown has the output feature tensor of the last stage as follows: .

[0048] The frequency domain feature extraction branch also includes 1×1 convolutional layers and batch normalization layers, which are placed before the average pooling layer. Specifically, the sequence is: 2D convolutional layer, max pooling layer, 3D convolutional residual module, 1×1 convolutional layer, batch normalization layer, average pooling layer, and feature flattening layer are connected sequentially. The 1×1 convolutional layer is configured to process the output feature tensor of the last residual module. A 1×1 convolution operation is performed, and the batch normalization layer is configured to perform batch normalization on the result of the 1×1 convolution operation.

[0049] Next, the average pooling layer performs global average pooling on the result of the batch normalization operation through two dimensions: frequency and time, to obtain the channel-compressed representation vector. : ,in, This represents a global average pooling function applied to both frequency and time dimensions. , represents a 32-dimensional real vector; The feature flattening layer is linearly transformed and mapped using the following formula: ,in, This represents the first output feature of the frequency domain feature extraction branch. Indicates trainable parameters, ,express A real matrix; , This 4-dimensional frequency domain representation maintains an equivalent structure to the output of the time domain branch, providing a structurally unified input basis for subsequent time-frequency fusion. The essence of "feature flattening" is the Flatten operation, which reshapes a multi-dimensional tensor. Its most common scenario is as a transition from convolutional / pooling layers to fully connected layers.

[0050] The working process of the temporal feature extraction branch is described below: The temporal feature extraction branch aims to model the dynamic changes and long-term dependencies of EEG signals over time to help identify potential patterns in attention evolution. The input is a CSP-processed EEG signal. CSP processing of the EEG signal yields a spatially filtered signal and its corresponding spatial pattern that maximizes the discrimination between two (or more) classes of EEG task data. A commonly used CSP feature form is logarithmic variance. ,in, It is the first The CSP component signals are obtained with a length of [number]. The feature vectors. Plotting these features in a two-dimensional graph usually shows that data points of different classes are well separated.

[0051] In the feature embedding layer, the input signal first passes through an embedding module consisting of two layers of two-dimensional convolution operations to extract joint representations across channels and in the local temporal neighborhood. This process treats the CSP-processed EEG signal as a two-dimensional input image, first extracting spatial structural features along the channel dimension, then compressing it to a fixed number of channels, ultimately forming a unified temporal representation. ,in, This indicates the number of channels to be set. Indicates the number of frames in the sliding time window; The temporal feature extraction branch introduces a multi-scale Mamba module: the internal structure of the first attention decoding module and the second attention decoding module is as follows. Figure 3 As shown, to effectively model multi-scale dynamic dependencies in EEG temporal data, the temporal branch introduces an attention decoding module based on state space modeling. Compared to traditional self-attention mechanisms, this module has lower computational complexity (linear) for long sequence modeling, while possessing efficient representation capabilities for local-global temporal dynamics. See [link to documentation]. Figure 3 Taking the first attention decoding module as an example, it first performs layer normalization on the input timing, and then generates a first gated projection branch and a second gated projection branch, where the first gated projection branch is denoted as... The second gated projection branch is denoted as ,in, Indicates the layer normalization processing function, This indicates the timing of the input to the attention decoding module. Represents the projection function; Then, the two gated projection branches are processed differently: the first gated projection branch Local dependencies are extracted using one-dimensional convolution, and activation values ​​are obtained using a non-linear activation function. ,in, Represents a one-dimensional convolution function. Represents a nonlinear activation function; Second gated projection branch The activation value is obtained through a nonlinear activation function. ; Next, a gating mechanism is used to perform weighted fusion output to obtain the weighted fusion result. ,in, This represents element-wise multiplication. And the weighted fusion results After performing layer normalization and projection processing sequentially, the output of the attention decoding module is obtained, denoted as... , .

[0052] To fully exploit time-dependent features at different levels, this embodiment constructs two attention decoding paths containing the first attention decoding module and the second attention decoding module: Unified timing representation As the timing input to the first attention decoding module, the attention decoding result of the first attention decoding module is calculated and denoted as... ; attention decoding results As the timing input to the second attention decoding module, the attention decoding result of the second attention decoding module is calculated and denoted as... ; The stitching module stitches together the data along the channel dimension to obtain a fused representation (stitching result): ,in, This represents the concatenation function.

[0053] Multi-scale Mamba is a multi-scale architecture based on the State-Space Model (SSM), representing an extension of the Mamba model for multi-scale processing. It combines Mamba's advantages in long sequence modeling with its multi-scale feature extraction capabilities, making it suitable for tasks requiring the simultaneous capture of local details and global dependencies. This embodiment employs parallel single-layer and two-layer Mamba modules to model the local transient dynamics and global long-term dependencies of EEG signals, respectively, enhancing the ability to capture long-term dependencies through an efficient state-space modeling mechanism.

[0054] The linear layer then undergoes two linear transformations to further map the result to the final time-domain representation, i.e., the second output feature: ,in, This represents the second output feature of the temporal feature extraction branch. The second output feature represents the trainable parameters and the output of the time-domain branch. The first output feature of the frequency domain branch output Maintaining the same structural dimensions provides a foundation for subsequent time-frequency fusion.

[0055] In another embodiment, the temporal feature extraction branch is further configured with a first average pooling layer, a first linear layer, a second average pooling layer, and a second linear layer, see [link to previous embodiment]. Figure 2 The first average pooling layer and the first linear layer are placed between the first attention decoding module and the second attention decoding module. Then, the attention decoding result from the first attention decoding module is sequentially processed by average pooling and linear transformation, and finally the processed attention decoding result is... Output to the second attention decoding module; The second average pooling layer and the second linear layer are positioned between the second attention decoding module and the concatenation module. The attention decoding result from the second attention decoding module is then subjected to average pooling and linear transformation sequentially. Finally, the processed attention decoding result output by the second linear layer is... The attention decoding result after processing the output of the first linear layer The pieces are then stitched together to obtain the stitched result. Correspondingly, the second output feature .

[0056] Then, the feature fusion module will output the first feature With the second output feature A new fused feature vector is obtained by concatenation. ; The fusion of feature vectors is related to the auditory attention decoding results, including obtaining the final classification result through a fully connected layer. ,in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This represents the predicted probability of the current input by the dual-path time-frequency fusion neural network model. The result can be binary, divided into two categories: auditory attention on the left and right sides. For example, when the predicted probability... A value between 0 and 0.5 indicates that auditory attention is on the left side; when the predicted probability... A value between 0.5 and 1 indicates that auditory attention is on the right side. The result can also be more than binary; accordingly, corresponding training samples are needed to train the model and update the network's weights and bias coefficients.

[0057] The trainable parameters mentioned above include as well as and the weights of the fully connected layer and bias These are all values ​​that the neural network automatically adjusts through optimization algorithms during training. By continuously adjusting these parameters, the neural network learns to extract patterns from the data and make accurate predictions. Specifically, the dual-path time-frequency fusion neural network model is trained in the following way: Multiple learning samples are collected. Each learning sample includes the differential entropy feature tensor corresponding to the same EEG signal and the CSP feature after CSP processing, as well as its corresponding classification label. Based on the collected learning samples, training sample sets and test sample sets are formed respectively; The dual-path time-frequency fusion neural network model was iteratively learned using the training sample set, and the network model was supervised training using the binary cross-entropy loss function. The prediction accuracy of the dual-path time-frequency fusion neural network model is calculated using the test sample set. Iterative learning stops when the accuracy reaches a preset threshold and / or a preset termination condition is met (e.g., when the test sample set loss no longer decreases after N consecutive epochs).

[0058] This embodiment leverages the advantages of multi-scale Mamba in modeling long-term dependencies and the strong representational capabilities of convolutional neural networks in spectral space modeling to achieve efficient and robust auditory attention decoding within a short time window, addressing the limitations of traditional AAD methods in terms of feature extraction dimensionality and temporal modeling capabilities. The temporal branch effectively models local transients and global long-term dependencies through parallel shallow and deep Mamba modules, while the frequency domain branch extracts rich spectral space features using 3D convolution and residual connections. The dual-path fusion strategy of the temporal and frequency domain branches further enhances the model's robustness and decoding accuracy.

[0059] The above embodiments provide a new technical framework for auditory attention decoding, which not only deepens the understanding of human auditory attention mechanisms but also lays the foundation for the development of hearing aids and intelligent hearing assistance systems. It allows for further optimization of model performance within shorter time windows, exploration of real-time decoding application scenarios, and expansion into more complex multi-sound-source environments to enhance its practicality. For example, in one embodiment of the invention, an auditory enhancement method is provided, such as... Figure 4 As shown, the hearing enhancement method includes the following steps: Using the auditory attention decoding method described above, the classification results of auditory attention decoding are determined, which represent the location where auditory attention is more concentrated; Based on the classification results of auditory attention decoding, the collected sound signals are enhanced: the sound signals from the directions where auditory attention is more concentrated are enhanced to a higher degree than the sound signals from other directions (sound signals from other directions may not be enhanced, or may even be weakened).

[0060] In one embodiment of the present invention, a hearing aid is provided, including a left hearing aid and / or a right hearing aid, wherein the gain and / or volume of the hearing aid are controlled using the above-described hearing enhancement method, such as... Figure 5 As shown: The classification results of auditory attention decoding are divided into left and right categories; If the classification result of auditory attention decoding is a left classification result, then increase the gain and / or volume of the left hearing aid; And / or, if the classification result of auditory attention decoding is a right classification result, then increase the gain and / or volume of the right hearing aid.

[0061] In one embodiment of the present invention, a multi-channel audio system is provided, which utilizes the above-described auditory enhancement method to perform sound tuning on the audio system, including: The audio system includes N channels, and the classification results of auditory attention decoding are divided into N classification results corresponding to the channels of the audio system. Based on the classification results of auditory attention decoding, increase the channel level of the corresponding channel.

[0062] In one embodiment of the present invention, a method for constructing a dual-path time-frequency fusion neural network model is provided, such as... Figure 6 As shown, it includes the following steps: A frequency domain feature extraction branch is constructed: a two-dimensional convolutional layer, a max pooling layer, a three-dimensional convolutional residual module, an average pooling layer, and a feature flattening layer are configured on this branch. The two-dimensional convolutional layer is configured to extract spatial texture features from the two-dimensional spatial image obtained by unfolding the differential entropy feature tensor extracted from the EEG signal along the electrode channel dimension, thereby obtaining a spatial feature map. The feature map is then sequentially processed by the max pooling layer, the three-dimensional convolutional residual module, the average pooling layer, and the linear transformation mapping (i.e., the Flatten operation) of the feature flattening layer to obtain the first output feature representing the frequency domain features. A temporal feature extraction branch is constructed: This branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a concatenation module, and a linear layer. The feature embedding layer is configured to extract spatial structural features along the channel dimension from the CSP-processed EEG signal and compress them to a set number of channels to form a unified temporal representation. The first attention decoding module receives the unified temporal representation and outputs the attention decoding result to the second attention decoding module. The concatenation module concatenates the attention decoding result of the second attention decoding module with the attention decoding result of the first attention decoding module along the channel dimension to obtain a concatenated result. The linear layer linearly transforms the concatenated result into a second output feature representing the temporal domain features. A feature fusion module is constructed, which is configured to fuse the first output feature and the second output feature to obtain a fused feature vector.

[0063] Furthermore, following any one or a combination of the aforementioned technical solutions, the first attention decoding module is constructed to include: The first-level normalization unit is configured to perform layer normalization on the unified time-series representation: ,in, Indicates the layer normalization processing function, This represents the temporal representation formed by the feature embedding layer; The first gated projection unit is configured to generate a first gated projection branch from the output of the first layer normalization unit, denoted as... ,in, Represents the projection function; The second gated projection unit is configured to generate a second gated projection branch from the output of the first layer normalization unit, denoted as... ; The first convolutional unit is configured to project a first gated branch. Perform one-dimensional convolution processing: ,in, This represents a one-dimensional convolution function; The first nonlinear activation unit is configured to perform nonlinear activation processing on the output of the first convolutional unit to obtain activation values. ,in, Represents a nonlinear activation function The second nonlinear activation unit is configured to perform nonlinear activation processing on the output of the second gated projection unit to obtain an activation value. ; The first summation unit is configured to perform weighted fusion through the following gating mechanism to obtain the weighted fusion result. ,in, This represents element-wise multiplication. The second-level normalization unit is configured to normalize the weighted fusion results. Perform layer normalization; The mapping unit is configured to project the output of the second-layer normalization unit to obtain the attention decoding result, denoted as... .

[0064] Furthermore, following any one or a combination of the aforementioned technical solutions, the second attention decoding module is constructed to include: The third-layer normalization unit is configured to process the attention decoding result output by the first attention decoding module. Perform layer normalization: ; The third gating projection unit is configured to generate a third gating projection branch from the output of the third-layer normalization unit, denoted as... ; The fourth gated projection unit, configured to generate a fourth gated projection branch from the output of the third-layer normalization unit, is denoted as... ; The second convolutional unit is configured to project the third gated branch. Perform one-dimensional convolution processing: ,in, This represents a one-dimensional convolution function; The third nonlinear activation unit is configured to perform nonlinear activation processing on the output of the second convolutional unit to obtain activation values. ,in, Represents a nonlinear activation function; The fourth nonlinear activation unit is configured to perform nonlinear activation processing on the output of the fourth gated projection unit to obtain the activation value. ; The second summing unit is configured to perform weighted fusion through the following gating mechanism to obtain the weighted fusion result. ,in, This represents element-wise multiplication. The fourth normalization unit is configured to normalize the weighted fusion results. Perform layer normalization; The mapping unit is configured to project the output of the fourth-layer normalization unit to obtain the attention decoding result, denoted as... .

[0065] Furthermore, following any one or a combination of the aforementioned technical solutions, the temporal representation formed by the feature embedding layer is defined as follows: ,in, This indicates the number of channels to be set. Indicates the number of frames in the sliding time window; The splicing module of the temporal feature extraction branch obtains the splicing result using the following formula. ,in, This represents the concatenation function.

[0066] The first attention decoding module and the second attention decoding module have the same construction; see details below. Figure 3 The only difference between the two is in the input data: the unified temporal representation formed by the feature embedding layer serves as the input to the first attention decoding module, and the attention decoding result of the first attention decoding module... As the timing of the input to the second attention decoding module.

[0067] Furthermore, following any or a combination of the aforementioned technical solutions, the temporal feature extraction branch is further configured with a first average pooling layer, a first linear layer, a second average pooling layer, and a second linear layer. The first average pooling layer and the first linear layer are positioned between the first attention decoding module and the second attention decoding module. The attention decoding result of the first attention decoding module is then subjected to average pooling and linear transformation processing in sequence, and the processed attention decoding result is then output to the second attention decoding module. The second average pooling layer and the second linear layer are set between the second attention decoding module and the splicing module. Then, the attention decoding result of the second attention decoding module is processed by average pooling and linear transformation in sequence. Finally, the processed attention decoding result output by the second linear layer is spliced ​​with the processed attention decoding result output by the first linear layer to obtain the spliced ​​result.

[0068] Furthermore, following any one or a combination of the aforementioned technical solutions, the linear layer of the temporal feature extraction branch processes the splicing result in the following manner. Perform a linear transformation mapping to obtain the second output feature: ,in, This represents the second output feature of the time-domain feature extraction branch. This represents the trainable parameters.

[0069] Furthermore, following any or a combination of the aforementioned technical solutions, the max pooling layer of the frequency domain feature extraction branch is configured to embed the original channel-dimensional features into the spatial feature map to obtain a tensor representation adapted to the 3D convolution input format; the three-dimensional convolutional residual module is stacked in multiple consecutive levels and is configured to extract the coupling relationship between the frequency domain features in the spatial structure and the spectral domain structure level by level to obtain the output feature tensor of the last residual module; the average pooling layer is configured to perform global average pooling through two dimensions of frequency and time to obtain the channel-compressed representation vector; the feature flattening layer is configured to linearly transform and map the channel-compressed representation vector to the first output feature of the frequency domain feature extraction branch.

[0070] Furthermore, following any one or a combination of the aforementioned technical solutions, the three-dimensional convolutional residual modules are stacked continuously at N levels. The coupling relationship between the frequency domain features extracted by the three-dimensional convolutional residual modules in the spatial and spectral domain structures includes: , ,in, Indicates the first The output feature tensor of the residual module, Indicates the first The input feature tensor of the residual module, This represents the three-dimensional convolution function corresponding to a 1×1×1 convolution kernel. This represents the three-dimensional convolution function corresponding to a 3×3×3 convolution kernel. Represents a linear correction function. This represents the output feature tensor of the last-stage residual module.

[0071] Furthermore, following any one or a combination of the aforementioned technical solutions, the two-dimensional convolutional layer has a 7×7 convolutional kernel, which yields its spatial feature map using the following formula: ,in, This represents the spatial feature map output by the two-dimensional convolutional layer. The two-dimensional convolution function represents a two-dimensional convolutional layer. The feature tensor of the difference entropy is represented as: ,in, Indicates the number of electrode channels. Indicates the number of discrete frequency bands. The variable represents the number of frames in the sliding time window; the differential entropy (DE) feature tensor represents the stability of EEG in the logarithmic power domain, which can effectively amplify the spectral differences between adjacent channels, suppress noise interference, and enhance separability, thus providing a good input representation basis for spectral modeling. It is very suitable for processing EEG rhythm signals that approximately follow a Gaussian distribution. In the process of extracting differential entropy (DE) features, the continuous EEG signal needs to be segmented into time windows (such as 2-second segments, which can overlap), and the differential entropy feature is calculated on each time window; for the signal of each channel and each time window, it is decomposed into the target rhythm frequency band through a bandpass filter to obtain several sub-signals of different frequency bands; for the signal of each frequency band, it is regarded as a random variable, and the variance of the signal is calculated. Substitute into the formula to calculate the difference entropy feature: .

[0072] The spatial feature map pass The activation function is then max-pooled through a max-pooling layer to compress the spatial dimensions and enhance nonlinear expressiveness. After this step, the original channel-dimensional features are embedded into the spatial feature map, forming a tensor representation adapted to the 3D convolutional input format, which serves as the input feature tensor for the first-level residual module. .

[0073] Furthermore, following any one or a combination of the aforementioned technical solutions, the average pooling layer obtains the channel-compressed representation vector using the following formula. : ,in, This represents a global average pooling function that performs pooling on both frequency and time dimensions. The feature tiling layer is linearly transformed and mapped using the following formula: ,in, This represents the first output feature of the frequency domain feature extraction branch. This represents the trainable parameters.

[0074] Furthermore, following any one or a combination of the aforementioned technical solutions, the frequency domain feature extraction branch is further configured with a 1×1 convolutional layer and a batch normalization layer. The two-dimensional convolutional layer, the max pooling layer, the three-dimensional convolutional residual module, the 1×1 convolutional layer, the batch normalization layer, the average pooling layer, and the feature flattening layer are sequentially connected. The 1×1 convolutional layer is configured to perform a 1×1 convolution operation on the output feature tensor of the last-stage residual module, and the batch normalization layer is configured to perform a batch normalization operation on the result of the 1×1 convolution operation.

[0075] Furthermore, following any one or a combination of the aforementioned technical solutions, the feature fusion module will convert the first output feature... With the second output feature The fused feature vector is obtained by concatenation. ; The final classification result is obtained through a fully connected layer. ,in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This represents the predicted probability of the dual-path time-frequency fusion neural network model for the current input.

[0076] Accordingly, the training method for the dual-path time-frequency fusion neural network model is as follows: Figure 7 This includes the following steps: A dual-path time-frequency fusion neural network model is constructed using the method described above; Multiple learning samples are collected. Each learning sample includes the differential entropy feature tensor corresponding to the same EEG signal and the CSP feature after CSP processing, as well as its corresponding classification label. Based on the collected learning samples, training sample sets and test sample sets are formed respectively; The dual-path time-frequency fusion neural network model is iteratively learned using the training sample set, and the network model is supervised training using the binary cross-entropy loss function; The prediction accuracy of the dual-path time-frequency fusion neural network model is calculated using the test sample set. When the accuracy reaches a preset threshold and / or a preset termination condition is met, the iterative learning stops.

[0077] This embodiment provides a dual-path time-frequency fusion neural network model obtained from the construction and training methods, which can be applied to auditory attention decoding. The model outputs the final classification result to predict which direction the current auditory attention is focused on. Here, the entire content of the auditory attention decoding method provided in the above embodiment is incorporated into this construction method embodiment and training method embodiment by reference.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An auditory attention decoding method, characterized in that, Includes the following steps: Acquire real-time EEG signals; Extract the differential entropy feature tensor from the frequency band of the real-time EEG signal, and perform CSP processing on the EEG signal to obtain CSP features; The differential entropy feature tensor and CSP feature are input into a pre-constructed dual-path time-frequency fusion neural network model; The dual-path time-frequency fusion neural network model outputs the auditory attention decoding result; The dual-path time-frequency fusion neural network model is constructed in the following way: by constructing a time-domain feature extraction branch, a frequency-domain feature extraction branch, and a feature fusion module, wherein: The frequency domain feature extraction branch is configured with a two-dimensional convolutional layer, a max pooling layer, a three-dimensional convolutional residual module, an average pooling layer, and a feature flattening layer to finally output the first output feature; The temporal feature extraction branch is configured with a feature embedding layer, a first attention decoding module, a second attention decoding module, a concatenation module, and a linear layer to finally output the second output feature; The feature fusion module is configured to fuse the first output feature and the second output feature to obtain a fused feature vector, which is related to the auditory attention decoding result.

2. The auditory attention decoding method according to claim 1, characterized in that, The two-dimensional convolutional layer is configured to extract spatial texture features from the two-dimensional spatial image obtained by unfolding the differential entropy feature tensor along the electrode channel dimension, thereby obtaining a spatial feature map; the max pooling layer is configured to embed the original channel dimension features into the spatial feature map, thereby obtaining a tensor representation adapted to the 3D convolutional input format; the three-dimensional convolutional residual module is stacked in multiple consecutive levels, and is configured to extract the coupling relationship between frequency domain features and spatial and spectral domain structures level by level, thereby obtaining the output feature tensor of the last residual module; the average pooling layer is configured to perform global average pooling through two dimensions of frequency and time, thereby obtaining the channel compressed representation vector; the feature flattening layer is configured to linearly transform and map the channel compressed representation vector to the first output feature of the frequency domain feature extraction branch.

3. The auditory attention decoding method according to claim 1, characterized in that, The feature embedding layer is configured to extract spatial structure features along the channel dimension of the CSP features and compress them to a set number of channels to form a unified temporal representation; the first attention decoding module receives the unified temporal representation and outputs the attention decoding result to the second attention decoding module; the concatenation module concatenates the attention decoding result of the second attention decoding module with the attention decoding result of the first attention decoding module along the channel dimension to obtain the concatenation result; the linear layer linearly transforms the concatenation result and maps it to the second output feature of the temporal feature extraction branch.

4. The auditory attention decoding method according to claim 2, characterized in that, The three-dimensional convolutional residual modules are stacked continuously in N levels. The coupling relationship between the frequency domain features extracted by the three-dimensional convolutional residual modules in the spatial and spectral domain structures includes: ， ,in, Indicates the first The output feature tensor of the residual module, Indicates the first The input feature tensor of the residual module, This represents the three-dimensional convolution function corresponding to a 1×1×1 convolution kernel. This represents the three-dimensional convolution function corresponding to a 3×3×3 convolution kernel. Represents a linear correction function. This represents the output feature tensor of the last-stage residual module.

5. The auditory attention decoding method according to claim 4, characterized in that, The two-dimensional convolutional layer has a 7×7 convolutional kernel, which obtains the spatial feature map using the following formula: ,in, This represents the spatial feature map output by the two-dimensional convolutional layer. The two-dimensional convolution function represents a two-dimensional convolutional layer. The feature tensor of the difference entropy is represented as: ,in, Indicates the number of electrode channels. Indicates the number of discrete frequency bands. Indicates the number of frames in the sliding time window; The spatial feature map pass The activation function is then max-pooled through a max-pooling layer to obtain the tensor representation of the adapted 3D convolutional input format, which serves as the input feature tensor of the first-level residual module. .

6. The auditory attention decoding method according to claim 4, characterized in that, The average pooling layer obtains the channel-compressed representation vector using the following formula. : ,in, This represents a global average pooling function that performs pooling on both frequency and time dimensions. The feature tiling layer is linearly transformed and mapped using the following formula: ,in, This represents the first output feature of the frequency domain feature extraction branch. This represents the trainable parameters.

7. The auditory attention decoding method according to any one of claims 1 to 6, characterized in that, The frequency domain feature extraction branch is further configured with a 1×1 convolutional layer and a batch normalization layer. The two-dimensional convolutional layer, the max pooling layer, the three-dimensional convolutional residual module, the 1×1 convolutional layer, the batch normalization layer, the average pooling layer, and the feature flattening layer are sequentially connected. The 1×1 convolutional layer is configured to perform a 1×1 convolution operation on the output feature tensor of the last residual module, and the batch normalization layer is configured to perform a batch normalization operation on the result of the 1×1 convolution operation.

8. The auditory attention decoding method according to claim 1, characterized in that, Both the first attention decoding module and the second attention decoding module obtain the corresponding attention decoding results in the following ways: The input timing sequence is subjected to layer normalization and a first gated projection branch and a second gated projection branch are generated, wherein the first gated projection branch is denoted as... The second gated projection branch is denoted as ,in, Indicates the layer normalization processing function, This indicates the timing of the input to the attention decoding module. Represents the projection function; First gated projection branch Local dependencies are extracted using one-dimensional convolution, and activation values ​​are obtained using a non-linear activation function. ,in, Represents a one-dimensional convolution function. Represents a nonlinear activation function; Second gated projection branch The activation value is obtained through a nonlinear activation function. ; The weighted fusion result is obtained by performing weighted fusion using the following gating mechanism. ,in, This represents element-wise multiplication. The weighted fusion result is then subjected to layer normalization and projection processing in sequence to obtain the attention decoding result.

9. The auditory attention decoding method according to claim 8, characterized in that, In the feature embedding layer of the temporal feature extraction branch, a unified temporal representation is defined as... ,in, This indicates the number of channels to be set. Indicates the number of frames in the sliding time window; The unified timing representation As the timing input to the first attention decoding module, the attention decoding result of the first attention decoding module is calculated and denoted as... ; attention decoding results As the timing input to the second attention decoding module, the attention decoding result of the second attention decoding module is calculated and denoted as... ; The splicing module obtains the splicing result using the following formula. ,in, This represents the concatenation function.

10. The auditory attention decoding method according to claim 9, characterized in that, The linear layer obtains the second output feature in the following manner: ,in, This represents the second output feature of the time-domain feature extraction branch. This represents the trainable parameters.

11. The auditory attention decoding method according to any one of claims 1 to 6 or any one of claims 8 to 10, characterized in that, The feature fusion module will use the first output feature With the second output feature The fused feature vector is obtained by concatenation. ; The fused feature vector is related to the auditory attention decoding result, including obtaining the final classification result through a fully connected layer. ,in, Indicates the weights of the fully connected layer. Indicates the bias of the fully connected layer. This represents the predicted probability of the dual-path time-frequency fusion neural network model for the current input.

12. The auditory attention decoding method according to claim 11, characterized in that, The dual-path time-frequency fusion neural network model is trained in the following way: Multiple learning samples are collected. Each learning sample includes the differential entropy feature tensor corresponding to the same EEG signal and the CSP feature after CSP processing, as well as its corresponding classification label. Based on the collected learning samples, training sample sets and test sample sets are formed respectively; The dual-path time-frequency fusion neural network model is iteratively learned using the training sample set, and the network model is supervised training using the binary cross-entropy loss function; The prediction accuracy of the dual-path time-frequency fusion neural network model is calculated using the test sample set. When the accuracy reaches a preset threshold and / or a preset termination condition is met, the iterative learning stops.

13. A method for enhancing hearing, characterized in that, Includes the following steps: Using the auditory attention decoding method as described in any one of claims 1 to 12, the classification result of auditory attention decoding is determined, which characterizes the location where auditory attention is more concentrated; Based on the classification results of the auditory attention decoding, the collected sound signals are enhanced: the enhancement level of sound signals from directions where auditory attention is more concentrated is higher than that of sound signals from other directions.

14. A hearing aid, comprising a left hearing aid and / or a right hearing aid, characterized in that, Controlling the gain and / or volume of the hearing aid using the hearing enhancement method as described in claim 13, comprising: The classification results of auditory attention decoding are divided into left and right categories; If the classification result of the auditory attention decoding is a left classification result, then increase the gain and / or volume of the left hearing aid; And / or, if the classification result of the auditory attention decoding is a right classification result, then increase the gain and / or volume of the right hearing aid.

15. A multi-channel audio system, characterized in that, The sound system is tuned using the hearing enhancement method as described in claim 13, comprising: The audio system includes N channels, and the classification results of auditory attention decoding are divided into N classification results corresponding to the channels of the audio system. Based on the classification results of the auditory attention decoding, the channel level of the corresponding channel is increased.

Images