Auditory space attention detection method, device and equipment and readable storage medium

Abstract

The invention relates to an auditory space attention detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring brain-computer signal characteristics; the brain-computer signal features represent brain waves, changing along with time, of the target object; performing binarization convolution processing on the brain-computer signal features through a binarization convolution layer to obtain target features; detecting auditory space attention information based on the target features; the auditory space attention information represents sound source position information perceived by the target object through auditory sense. The method can give consideration to both processing efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of brain-computer signal recognition, and in particular to an auditory spatial attention detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Neuroscience research shows that brain signals can decode auditory attention information. Existing brain signals include invasive signals and non-invasive signals. Non-invasive signals have the advantages of high temporal resolution, non-invasiveness, cost-effectiveness and equipment portability.

[0003] In traditional technology, in order to achieve auditory attention decoding of non-intrusive signals, a full-precision convolutional neural network model is used for detection. However, it requires a lot of computing resources, which limits its application in mobile or wearable device scenarios. Summary of the invention

[0004] Based on this, it is necessary to provide an auditory spatial attention detection method, device, computer equipment, computer-readable storage medium and computer program product that can reduce the amount of calculation and ensure accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides an auditory spatial attention detection method, comprising:

[0006] Acquire brain-computer signal features; brain-computer signal features represent brain waves of the target object that change over time;

[0007] Through the binary convolution layer, the brain-computer signal feature is subjected to binary convolution processing to obtain the target feature;

[0008] Auditory spatial attention information is detected based on the target features; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

[0009] In one embodiment, the obtaining of brain-computer signal features includes:

[0010] Acquire brain-computer signals; the brain-computer signals are brain wave signals of the target object that change over time;

[0011] Through a full-precision convolution layer, feature extraction is performed on the feature data to obtain full-precision brain-computer signal features;

[0012] Among them, the activation function of the full-precision convolution layer is PRelu, and / or the full-precision brain-computer signal features are the input features of the first-layer binary convolution layer.

[0013] In one embodiment, the brain-computer signal feature is output by a convolution layer before a binarization convolution layer, and the brain-computer signal feature is binarized;

[0014] The binarization convolution layer is used to perform binarization convolution processing on the brain-computer signal feature to obtain the target feature, including:

[0015] Performing an XOR operation on each bit in the brain-computer signal feature through a binary convolution kernel to obtain an XOR operation result for each bit;

[0016] Among the XOR operation results of each bit, the number of bits whose XOR operation results are preset values ​​is counted to obtain a binary convolution feature;

[0017] Performing activation processing on the binary convolution feature to obtain an activation value;

[0018] According to the activation value, a target feature is determined.

[0019] In one embodiment, the performing activation processing on the binary convolution feature to obtain an activation value includes:

[0020] If the pre-activation data corresponding to the binary convolution feature is less than a threshold, determining an activation value based on the pre-activation data;

[0021] If the pre-activation data corresponding to the binary convolution feature is greater than or equal to the threshold, adjusting the pre-activation data based on a preset weight, and determining an activation value based on the adjusted pre-activation data;

[0022] The preset weights are obtained by optimizing the training process of the binary convolution layer.

[0023] In one embodiment, determining the target feature according to the activation value includes:

[0024] Performing pooling processing according to the activation value to obtain a pooling feature;

[0025] Among them, the pooled feature is the feature output by the binary convolution layer, and the last pooled feature is the target feature.

[0026] In one embodiment, the auditory spatial attention information includes a focus direction and a sound source position angle, wherein the focus direction indicates that the sound source is on the left side or the right side of a preset direction, and the sound source position angle indicates a deviation angle of the sound source relative to the preset direction;

[0027] The detecting of auditory spatial attention information based on the target feature comprises:

[0028] By focusing on the side of the head, detecting the direction of the sound source for the target feature, and obtaining the focus direction;

[0029] The sound source position of the target feature is detected by the spatial position head to obtain the sound source position angle.

[0030] In one embodiment, the method further comprises:

[0031] Selecting the minimum value between the parameter gradient to be optimized of the binary convolution layer and the preset growth gradient as the candidate gradient;

[0032] Selecting the maximum value between the preset reduction gradient and the candidate gradient as the target gradient;

[0033] According to the target gradient, the gradient of the parameter to be optimized is optimized.

[0034] In a second aspect, the present application also provides an auditory spatial attention detection device, comprising:

[0035] A feature acquisition module is used to acquire brain-computer signal features; brain-computer signal features represent the brain waves of the target object that change over time;

[0036] A binarization module, used for performing binarization convolution processing on the brain-computer signal features through a binarization convolution layer to obtain target features;

[0037] A detection module is used to detect auditory spatial attention information based on the target feature; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

[0038] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of auditory spatial attention detection in any of the above embodiments are implemented.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of auditory spatial attention detection in any of the above embodiments are implemented.

[0040] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of auditory spatial attention detection in any of the above embodiments are implemented.

[0041] The above-mentioned auditory spatial attention detection method, device, computer equipment, computer-readable storage medium and computer program product obtain brain-computer signal features. Since the brain-computer signal features reflect the brain waves of the target object that change over time, they can reflect the auditory spatial attention pattern of the target object from the perspective of time-frequency analysis, laying the foundation for the classification of auditory spatial attention. On this basis, since the brain-computer signal features of brain waves that change over time have the characteristic of large data volume, the brain-computer signal features are binarized and convoluted through a binarized convolution layer to obtain target features; through binarized convolution processing, the amount of data can be reduced, and important information can still be extracted from the brain-computer signal features while reducing the computational overhead and memory usage; and then the auditory spatial attention information is detected based on the target feature; through the auditory spatial attention information, the sound source position information perceived by the target object through hearing is accurately reflected. Therefore, the amount of calculation is reduced through the binarized convolution layer, and the feature extraction is guaranteed through convolution processing, taking into account both processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] FIG1 is a diagram of an application environment of an auditory spatial attention detection method according to an embodiment;

[0044] Figure 2 is a flow chart of an auditory spatial attention detection method in one embodiment;

[0045] Figure 3 A schematic diagram of a process for acquiring brain-computer signal features in one embodiment;

[0046] Figure 4 A schematic diagram of a process of binarization convolution layer processing in one embodiment;

[0047] Figure 5 A schematic diagram of a specific process of an auditory spatial attention detection method in one embodiment;

[0048] Figure 6 is a schematic diagram of a brain wave signal in one embodiment;

[0049] Figure 7 is a schematic diagram of gradient estimation in one embodiment;

[0050] Figure 8A diagram of an application environment of an auditory spatial attention detection method in one embodiment;

[0051] Fig. 9 It is a schematic diagram of the effect of an auditory spatial attention detection method in one embodiment;

[0052] Fig.10 is a structural block diagram of an auditory spatial attention detection device in one embodiment;

[0053] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The auditory spatial attention detection method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​In which, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0056] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0057] In an exemplary embodiment, Figure 2 As shown, a method for detecting auditory spatial attention is provided, which is described by taking the method applied to the terminal 102 in FIG. 1 as an example, and includes the following steps 202 to 206. Among them:

[0058] Step 202, obtaining brain-computer signal characteristics; the brain-computer signal characteristics represent the brain waves of the target object that change over time.

[0059] The brain-computer signal feature is the input feature of the binary convolution layer. The brain-computer signal feature is the feature of the brain wave changing over time, that is, the feature of the brain-computer signal in the time dimension. The brain-computer signal feature can reflect the change in the amplitude value caused by the sound stimulation to reflect the location of the sound source perceived by the target object. Optionally, the brain-computer signal feature can be the amplitude value feature of the brain wave at different times, or the amplitude change value feature of the brain wave at different times.

[0060] Brain waves are signals generated by the brain of the target object. When the method is applied to a mobile device, the brain waves are non-invasive signals. Optionally, the brain waves are in the form of an electroencephalogram (EEG). Optionally, the terminal obtains the brain waves through a brain-computer interface (BCI).

[0061] In an optional embodiment, for the first binary convolution layer, the steps for acquiring brain-computer signal features include: acquiring brain waves of the target object that change over time; high-pass filtering the brain waves to obtain a high-pass filtered signal; band-pass filtering the high-pass filtered signal to obtain a band-pass filtered signal; resampling the band-pass filtered signal to obtain a resampled signal; segmenting the resampled signal using a sliding window to obtain the brain-computer signal to be extracted; and extracting features from the brain-computer signal through a neural network to obtain brain-computer signal features.

[0062] Among them, high-pass filtering is to remove signal components below the frequency by setting a cutoff frequency, which helps to eliminate the low-frequency trend caused by poor electrode contact or electrode drift; band-pass filtering is to retain the signal within the 1-50 Hz frequency band, while removing the signal outside the frequency band, which helps to focus on the EEG signal frequency band of interest and reduce the impact of electrode drift. The purpose of using resampling is to reduce the total amount of data by reducing the sampling rate, which helps to increase the calculation speed of subsequent processing steps; and the sliding window technology extracts data fragments by continuously moving a fixed-length window on the data stream. By sliding the window on the data and extracting the signal fragments in each window, multiple data fragments can be created to control the amount of data processed, which is consistent with the focus of this study on low-latency settings.

[0063] In an optional embodiment, for the second binary convolution layer, the step of acquiring the brain-computer signal features includes: determining the brain-computer signal features for input to the second binary convolution layer based on the features output by the first binary convolution layer. The second binary convolution layer can be replaced by the i-th binary convolution layer. In this case, the brain-computer signal features for input to the i-th binary convolution layer are the features output by the i-1-th binary convolution layer, thereby forming a stack of multiple binary convolution layers.

[0064] Step 204, performing binarization convolution processing on the brain-computer signal features through a binarization convolution layer to obtain target features.

[0065] The binary convolution layer is a neural network structure used for binary convolution processing. In the binary convolution kernel of the binary convolution layer, the weights are designed to contain only two values, so the storage and calculation of the weights only require 1 bit of data, forming a bit-by-bit calculation. This design can reduce the complexity and amount of calculation of the model. Correspondingly, before the brain-computer signal is convolved, it is necessary to perform convolution processing on the brain-computer signal so that the feature extraction process is based on the convolution layer.

[0066] Optionally, binary neural networks (BNNs) significantly reduce memory usage and speed up inference during deployment by limiting the model's weights and activation values ​​to binary. Unlike traditional matrix multiplication, binary neural networks perform bit-by-bit XNOR operations and count the values ​​of 1 (PopCount), which can ensure high efficiency.

[0067] Optionally, both the weight and activation functions are symbolically doubled; optionally, XNOR-Net can be used to further integrate the scaling factors of weights and activations to reduce the memory usage and accelerate. In some embodiments, a 58-fold acceleration can be achieved, and the memory is reduced to one thirty-second.

[0068] Optionally, there are multiple variations of binary neural networks. Among them, to solve the quantization error caused by binarization, XNOR-Net can be enhanced by incorporating scaling factors and learning them through back-propagation through XNOR-Net++. In addition, IR-Net improves performance by standardizing weights before convolution and gradually reshaping gradients. Bi-Real Net improves top-1 accuracy by 10% by increasing memory savings and reducing computational costs. In addition, BinaryDenseNet introduces a dedicated architecture for binary convolution. ReCU improves performance by allowing large weights to retain their sign-flipping potential. FABNet introduces a frequency-aware binarization network for single image super-resolution, optimizing quantization errors by considering spatial frequency components. BiSRNet introduces a redesigned compact base model combined with binarized spectral redistribution convolution to effectively and practically restore hyperspectral images from compressed measurements.

[0069] Optionally, in the case of training a binary convolution layer, the weights are designed to contain only two values, +1 and -1. The brain-computer signal features serving as samples are multiplied (MULT) by the weights to obtain the multiplication result, and then the multiplication result is summed (SUM) to achieve the binarized convolution processing. In the case of deploying the binary convolution layer to an edge device, the weights are designed to contain only two values, +1 and 0. The binarized convolution processing is achieved by performing an exclusive OR operation (XNOR) and a statistical (PopCount) processing with a value of 1 on the brain-computer signal features obtained by the edge device.

[0070] The target feature is a feature that can be used to reflect auditory spatial attention information. The target feature is extracted based on a binary convolution layer; optionally, the target feature is extracted layer by layer based on stacked binary convolution layers, that is, the feature extracted by the last binary convolution layer. Optionally, a pooling layer is provided between the stacked binary convolution layers to perform pooling processing on the features output by the binary convolution layers, thereby controlling the number of dimensions of the binary convolution processing and achieving effects such as adaptive adjustment of channel dimensions and dimensionality reduction.

[0071] Step 206, detecting auditory spatial attention information based on the target features; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

[0072] Auditory spatial attention information is sound source position information reflected by the auditory dimension, and the sound source position information is the position information of the sound source emitting the sound relative to the target object. Optionally, the auditory spatial attention information is information in Auditory Spatial Attention Detection (ASAD), which mainly focuses on how to determine where a person focuses his or her auditory attention by analyzing auditory signals.

[0073] Auditory spatial attention information at least involves direction in binary classification tasks, especially in scenarios where the target object is focused on the left or right sound. Optionally, in real-life communication scenarios, multiple sound sources from different directions often interact at the same time. In this case, auditory spatial attention information may also include direction or position angle to avoid oversimplification of the scenario by the binary classification model and achieve accurate recognition in complex scenarios.

[0074] In an optional embodiment, auditory spatial attention information is detected based on target features, including: inputting the target features after global mean pooling processing into a classification head for detecting auditory spatial attention information for classification and recognition, so as to obtain the auditory spatial attention information.

[0075] In an optional embodiment, the target features after global mean pooling processing are input into a classification head for detecting auditory spatial attention information for classification and identification, including: weighting the target features after global mean pooling processing by the weights contained in the fully connected layer for detecting auditory spatial attention information to obtain auditory spatial attention information. Among them, the weights contained in the fully connected layer can be set for the attention side (AS) head or for the spatial position (SL) head: the attention side head is used to determine the attention side of the stimulus and classify it as the left or right side. The spatial position head is used to predict the spatial position of the auditory stimulus, and the spatial position is one of multiple degrees that deviate from a preset direction. For example, the multiple degrees are five, namely: 5°, 30°, 45°, 60° and 90°.

[0076] In the above-mentioned auditory spatial attention detection method, the brain-computer signal features are obtained. Since the brain-computer signal features reflect the brain waves of the target object that change over time, they can reflect the auditory spatial attention pattern of the target object from the perspective of time-frequency analysis, laying the foundation for the classification of auditory spatial attention. On this basis, since the brain-computer signal features of brain waves that change over time have the characteristic of large data volume, the brain-computer signal features are binarized and convolved through a binary convolution layer to obtain the target features; the binary convolution process can reduce the amount of data, and while reducing the computational overhead and memory usage, important information can still be extracted from the brain-computer signal features; then the auditory spatial attention information is detected based on the target features; the auditory spatial attention information accurately reflects the sound source location information perceived by the target object through hearing. Therefore, the amount of calculation is reduced through the binary convolution layer, and the feature extraction is guaranteed through the convolution process, taking into account both processing efficiency and accuracy.

[0077] In one embodiment, Figure 3 As shown, obtaining brain-computer signal features includes steps 302 to 304. Among them:

[0078] Step 302, obtaining brain-computer signals; brain-computer signals are brain wave signals of the target object that change over time.

[0079] Brain-computer signals are preprocessed signals.

[0080] The brain-computer signal is the signal of the brain-computer signal in the time dimension, that is, the brain-computer signal is the signal of the brain wave changing over time. The brain-computer signal can reflect the change of the amplitude value caused by sound stimulation, laying the foundation for feature extraction in the time dimension. Optionally, the brain-computer signal can be the amplitude value of the brain wave at different times, or the amplitude change value of the brain wave at different times.

[0081] In an optional embodiment, obtaining a brain-computer signal includes: obtaining brain waves of a target object that change over time; high-pass filtering the brain waves to obtain a high-pass filtered signal; band-pass filtering the high-pass filtered signal to obtain a band-pass filtered signal; resampling the band-pass filtered signal to obtain a resampled signal; and segmenting the resampled signal using a sliding window to obtain a brain-computer signal.

[0082] Step 304, extracting features from the feature data through a full-precision convolutional layer to obtain full-precision brain-computer signal features.

[0083] Among them, the activation function of the full-precision convolution layer is PRelu, and / or the full-precision brain-computer signal features are the input features of the first binary convolution layer.

[0084] The full-precision convolution layer is the maximum number of bits under the corresponding standard. The full-precision convolution layer is based on the precision floating point number under the corresponding standard. For example: under the IEEE 754 standard, the full-precision convolution layer can be a 32-bit single-precision floating point number, including 1 sign bit, 8 exponent bits, and 23 mantissa bits; the full-precision convolution layer can also be a 64-bit double-precision floating point number that follows the IEEE 754 standard, which provides more exponent bits and mantissa bits, including 1 sign bit, 11 exponent bits, and 52 mantissa bits.

[0085] In a full-precision convolutional layer, both weights (convolution kernels) and activation values ​​are expressed in full precision. Optionally, each convolutional layer consists of multiple convolution kernels that slide over the input image to produce feature maps, and after the convolution operation, an activation function such as ReLU, PReLU, etc. is passed to introduce nonlinearity so that the network can determine complex features.

[0086] In full-precision representation, weights and activation values ​​are stored using more bits, which can reduce the error of numerical representation and improve the performance of the model. However, this high-precision representation also means that the storage and computational costs of the model are high, so the full-precision convolution layer is used for feature extraction only before the first layer of binarization network. Optionally, the number of full-precision convolution layers is one, that is, the first convolution layer with greater accuracy impact and smaller computational complexity is the full-precision convolution layer, and the other convolution layers are binarization convolution layers.

[0087] Optionally, the activation value is obtained using the PReLU activation function. Even if there is a large deviation in the brain-computer signal, it can be adjusted by the corresponding preset weights. Moreover, during the training process of the corresponding model, matrix multiplication can be used to extract features from weights and brain-computer signals in the form of +1 and -1 to ensure efficiency. Moreover, since the PReLU activation function does not cause the loss of information transmitted by brain-computer signals in the form of negative numbers, it can ensure that subsequent feature extraction is more accurate. Therefore, when computing resources are sufficient, the training efficiency can be guaranteed and the accuracy of the preset weights can be guaranteed.

[0088] In this embodiment, because the accuracy is greatly affected when feature extraction is performed first and the amount of calculation is small, the full-precision convolution layer is used for feature extraction to greatly increase the accuracy with a smaller amount of parameters, so as to avoid the problem of feature extraction performance degradation that may be caused in the initial binarization stage.

[0089] In one embodiment, Figure 4 As shown, the brain-computer signal features are output by the previous convolution layer of the binary convolution layer. For example, for the first binary convolution layer, the brain-computer signal features are features output by other convolution networks, such as the full-precision convolution layer mentioned above; and for the second binary convolution layer, the brain-computer signal features are features output by the first binary convolution layer, and so on, forming a stack of binary convolution layers.

[0090] The brain-computer signal features are binarized, which means that the brain-computer signal features with an accuracy greater than 2 are binarized to obtain brain-computer signal features for subsequent processing. Optionally, the input features of the first binarized convolution layer can be binarized by a sign function to perform an XOR operation. Optionally, in the case of stacking binarized convolution layers, the input features of the second binarized convolution layer are the features output by the first binarized convolution layer. In this case, the binarization process can be omitted as required.

[0091] Through the binary convolution layer, the brain-computer signal features are binarized and convolved to obtain the target features, including:

[0092] Step 402, using a binary convolution kernel, performs an XOR operation on each bit in the brain-computer signal feature to obtain an XOR operation result for each bit.

[0093] The binary convolution kernel is a convolution kernel that represents weights in a binary form. Since the binary convolution kernel is used for XOR operations, its weight value exists in the form of 0 and 1, or in the form of -1 and 1. Optionally, the binary convolution kernel is a weight in the form of a matrix, vector, etc., which is used to perform weighted processing on brain-computer signal features at different locations.

[0094] Bit is the unit of data. Each bit can store one bit of data. One bit of data can be one of 0 and 1, or one of -1 and 1. Bit is the basic unit that constitutes the binary convolution kernel and the binary brain-computer signal feature. The binary brain-computer signal feature is located in multiple feature positions. The presence of one bit of data in each feature position can reflect the accurate characterization of the brain-computer signal feature. The result of the XOR operation is the weighted operation result of each bit in the brain-computer signal feature.

[0095] In an optional embodiment, a binary convolution kernel is used to perform an XOR operation on each bit in the brain-computer signal feature to obtain the XOR operation result of each bit, including: determining the bit to be processed in the brain-computer signal feature at intervals of a preset step size; according to the correspondence between the feature position and the weight position, the weight of the binary convolution kernel at each position is used to perform an XOR operation on the bit to be processed to obtain the XOR operation result of each bit.

[0096] In an optional embodiment, a binary convolution kernel is used to perform an XOR operation on each bit in the brain-computer signal feature to obtain the XOR operation result of each bit, including: determining whether each bit in the binary convolution kernel and each bit corresponding to the brain-computer signal feature have the same data according to the correspondence between the feature position and the weight position; assigning a value of 1 to bits with the same data, and assigning a value of 0 to bits with different data to obtain the XOR operation result of each bit.

[0097] Exemplarily, the preset step size is 1 pixel position, the binary convolution kernel is a 2*2 convolution kernel matrix, and there are N feature positions of the brain-computer signal feature. Then, among the N feature positions, every 1 pixel position, a 2*2 feature position is selected as each 2*2 bit to be processed; since each 2*2 bit to be processed has 4 feature positions, and the 2*2 convolution kernel matrix has 4 weight positions, and there is a corresponding relationship between the feature position and the weight position at the position with the same coordinate, it is necessary to perform an XOR operation;

[0098] Among them, for each 2*2 bit to be processed, the XOR operation is performed with the 2*2 convolution kernel matrix, with the A position of the 2*2 as the data of the A1 bit, and the A position in the 2*2 convolution kernel matrix as the data of the A2 bit, the XOR operation result is assigned to the data of the A3 bit at the A position. Correspondingly, if the data of the A1 bit is 1, and the data of the A2 bit is 1, then the A3 bit is assigned to 1; if the data of the A1 bit is 0, and the data of the A2 bit is 0, then the A3 bit is assigned to 1; if the data of the A1 bit is 0, and the data of the A2 bit is 1, then the A3 bit is assigned to 0; if the data of the A1 bit is 1, and the data of the A2 bit is 0, then the A3 bit is assigned to 0; if the data of the A1 bit is 1, and the data of the A2 bit is 0, then the A3 bit is assigned to 0; and so on, the XOR operation result of each 2*2 bit to be processed is obtained.

[0099] Step 404: among the XENO operation results of each bit, count the number of bits whose XENO operation results are preset values ​​to obtain a binary convolution feature.

[0100] The preset value is a binary value, which can be one of 0 or 1, or one of -1 and +1. In the inference process, the binary convolution feature is a value obtained by counting the number of bits of the data with the preset value. Optionally, the binary convolution feature can be a Hamming weight or the number of bits of 0.

[0101] In an optional embodiment, among the XENOR operation results of each bit, the number of bits whose XENOR operation results are preset values ​​is counted to obtain a binary convolution feature, including: using a popcount function, the number of bits whose XENOR operation results are 1 is counted to obtain a Hamming weight.

[0102] Step 406, performing activation processing on the binary convolution feature to obtain an activation value of the binary convolution feature.

[0103] The activation value is the binary convolution feature after nonlinear processing, and is the result of processing the binary convolution feature by the activation function. Optionally, the binary convolution feature can be processed by a PReLU activation function, a ReLU activation function, or other activation functions to obtain the activation value.

[0104] In an optional embodiment, the binary convolution feature is activated to obtain the activation value of the binary convolution feature, including: if the binary convolution feature is less than the preset activation value, the activation value is determined based on the preset activation value; if the binary convolution feature is greater than the preset activation value, the activation value is determined based on the adjusted binary convolution feature; wherein the preset activation value may be 0. Thus, the activation value is obtained using an activation function related to the ReLU concept.

[0105] In another optional embodiment, the activation value of the brain-computer signal feature and / or the binary convolution feature is activated based on the PReLU activation function. Thus, at least one of the brain-computer signal feature and the binary convolution feature is an activation value obtained based on the PReLU activation function.

[0106] In an optional embodiment, the binary convolution feature is activated to obtain an activation value, including: if the pre-activation data corresponding to the binary convolution feature is less than a threshold, determining the activation value based on the pre-activation data; if the pre-activation data corresponding to the binary convolution feature is greater than or equal to the threshold, adjusting the pre-activation data based on a preset weight, and determining the activation value based on the adjusted pre-activation data; wherein the preset weight is optimized during the training process of the binary convolution layer.

[0107] The threshold is a preset parameter set for pre-activation data; the threshold matches the type of pre-activation data; when the pre-activation data is a binary convolution feature, the threshold is set for the binary convolution feature; when the pre-activation data is a normalized layer feature, the threshold is set for the normalized layer feature. The pre-activation data can be the binary convolution feature itself or the normalized layer feature output by the batch normalization layer (Batch Normalization, BN).

[0108] The preset weights are optimized during the training process of the binary convolution layer. Optionally, the process of obtaining the preset weights includes: first, during the neural network definition process, the preset weights of the PReLU activation function are initialized to a small positive number, such as 0.25. Then, during the training of the network, the preset weights are gradient updated through the back-propagation algorithm. This process adjusts the value of the preset weights according to the loss function of the network to minimize the prediction error. Then, during the back-propagation process, the gradient of the preset weights is updated by learning the patterns and corresponding loss functions in the data set, and the preset weights are continuously updated through gradient descent or other optimization algorithms, and finally an optimal value is obtained, so that the activation function can better adjust the binary convolution features, taking into account both efficiency and accuracy.

[0109] In an optional embodiment, determining the activation value based on the pre-activation data includes: using the binarized convolution feature as the activation value.

[0110] In an optional embodiment, determining the activation value based on the pre-activation data includes: when the pre-activation data is a batch normalized feature output by a batch normalization layer, using the normalized feature as the activation value.

[0111] In an optional embodiment, the pre-activation data is adjusted based on preset weights, and the activation value is determined based on the adjusted pre-activation data, including: based on the preset weights, weighted processing is performed on the channel values ​​of the binary convolution features to obtain the adjusted binary convolution features; and the adjusted binary convolution features are used as the activation values.

[0112] Therefore, the activation value is obtained using the PReLU activation function. Even if there is a large deviation in the binary convolution feature, it can be adjusted through the corresponding preset weights. Moreover, during the training process of the corresponding model, matrix multiplication can be used to extract features of weights in the form of +1 and -1 and brain-computer signal features to ensure efficiency. Moreover, since the PReLU activation function will not cause the loss of information transmitted by the negative binary convolution feature, it can ensure that the binary convolution feature is more accurate. Therefore, when there are sufficient computing resources, the training efficiency can be guaranteed, and the accuracy of the preset weights can be guaranteed.

[0113] Step 408, determining the target feature according to the activation value.

[0114] In one embodiment, determining the target feature according to the activation value includes: inputting the activation value into a pooling process to obtain the target feature. Thus, the target feature can be determined through a binarization convolution layer to reduce the amount of data required for the corresponding processing.

[0115] In one embodiment, determining the target feature according to the activation value includes: inputting the activation value to the next binarization convolution layer so that the next binarization convolution layer executes steps 402-408, until the activation value of the last binarization convolution layer is obtained, and then determining the target feature according to the next binarization convolution layer.

[0116] In one embodiment, the target feature is determined according to the activation value, including: performing pooling processing according to the activation value to obtain a pooled feature; wherein the pooled feature is a feature output by a binary convolution layer, and the last pooled feature is the target feature.

[0117] In an optional embodiment, performing pooling processing on the activation values ​​to obtain pooling features includes: performing mean pooling processing on the activation values ​​to obtain pooling features.

[0118] In an optional embodiment, performing pooling processing according to the activation value to obtain the pooling feature includes: performing global processing on the activation value to obtain the pooling feature.

[0119] In an optional embodiment, pooling processing is performed according to the activation value to obtain a pooled feature, including: performing global average pooling processing on the last pooled feature to flatten the feature of the last pooled feature to facilitate subsequent classification detection.

[0120] Therefore, by performing pooling processing on the last pooled feature, the feature of the last pooled feature can be expanded into a form other than binary, so that the general classification head can use the target feature for inspection.

[0121] In this embodiment, limitations are imposed on each binary convolution layer. Compared with traditional matrix multiplication, the amount of data required for the XOR operation and the statistics of the binary convolution features is lower, which can reduce computational overhead and memory usage and ensure accuracy. At the same time, the binary convolution layer can extract and enhance brain-computer signal features, ensuring the accuracy of feature extraction.

[0122] In one embodiment, the auditory spatial attention information includes a focus direction and a sound source position angle. The focus direction indicates that the sound source is on one of the left and right sides of a preset direction, and the sound source position angle indicates the deviation angle of the sound source relative to the preset direction.

[0123] Correspondingly, the auditory spatial attention information is detected based on the target feature, including: detecting the sound source direction of the target feature by paying attention to the side head to obtain the attention direction; detecting the sound source position of the target feature by the spatial position head to obtain the sound source position angle.

[0124] The focus direction is the classification result of the binary classification task; the focus direction is the analysis result of the target feature representation. Optionally, when the focus direction represents that the sound source is on the left side of the preset direction, the auditory spatial attention of the target object is focused on the left side of the preset direction; when the focus direction represents that the sound source is on the right side of the preset direction, the auditory spatial attention of the target object is focused on the right side of the preset direction.

[0125] The sound source position angle is the classification result of the multi-classification task. The sound source position angle includes multiple angles, which are used to reflect that the auditory spatial attention of the target object is focused on a specific angle, for example, the auditory spatial attention is focused on a position with an angle of +30 degrees from the preset direction, or the auditory spatial attention is focused on a position with an angle of -5 degrees from the preset direction.

[0126] In an optional embodiment, the direction of the sound source is detected by focusing on the side of the head to obtain the focus direction, including: fully connecting the target feature through the weight matrix of the focus side of the head to obtain a direction classification fully connected feature; inputting the direction classification fully connected feature into the softmax activation function for activation to obtain the focus direction.

[0127] In an optional embodiment, the attention side of the sound source is detected for the target feature through the attention side classification head to obtain the attention side, including: fully connecting the target feature through the weight matrix of the attention side classification head to obtain the attention side classification fully connected feature; inputting the attention side classification fully connected feature into the softmax activation function for activation to obtain the attention side.

[0128] In this embodiment, the left or right focus direction and the sound source position angle can be decoupled to avoid mutual interference between data. Auditory spatial attention information at least involves the direction in the binary classification task, especially in the scene where the target object is identified to focus on the left or right sound. At the same time, in real-life communication scenarios, multiple sound sources from different directions often interact at the same time. At this time, the auditory spatial attention information may also include direction angles or position angles to avoid oversimplification of the scene by the binary classification model and achieve accurate recognition in complex scenes.

[0129] In one embodiment, the method further includes: selecting the minimum value between the gradient of the parameter to be optimized of the binary convolution layer and the preset growth gradient as the candidate gradient; selecting the maximum value between the preset decrease gradient and the candidate gradient as the target gradient; and optimizing the gradient of the parameter to be optimized according to the target gradient.

[0130] The preset growth gradient and the preset reduction gradient represent opposite gradient changes to form opposite gradient change trends. Optionally, since the minimum value of the parameter gradient to be optimized and the preset growth gradient is used as the candidate gradient, and the maximum value of the preset reduction gradient and the candidate gradient is selected as the target gradient, the parameter gradient to be optimized is controlled to change between the preset growth gradient and the preset reduction gradient. Among them, the preset reduction gradient and the preset growth gradient represent opposite gradient changes, and the closer the parameter gradient to be optimized is to the preset growth gradient, the greater the gradient growth; the closer the parameter gradient to be optimized is to the preset reduction gradient, the greater the gradient reduction.

[0131] Optionally, the preset growth gradient may be +1, the parameter gradient to be optimized may be the weight gradient of the binary convolution layer, and the preset decrease gradient may be -1.

[0132] In this embodiment, since the features of the binary convolution layer are all binary, the gradient cannot be calculated directly, so the straight-through estimator (STE) is used to calculate the gradient during the back propagation process. The straight-through estimator is used to approximate the gradient, so that each binary convolution layer is trained using the gradient descent method, which significantly reduces the computation and memory requirements, making it very suitable for hardware with limited resources.

[0133] In an exemplary embodiment, the auditory spatial attention detection method is a multi-classification task that aims to identify the direction and angular position of auditory attention from electroencephalogram (EEG) signals. The ASAD dataset contains EEG records corresponding to auditory stimuli presented at multiple spatial angles: 5°, 30°, 45°, 60°, and 90° from the left and right sides, a total of 10 different directions; its classification problem can be expressed as follows:

[0134] (1)

[0135] in, represents the input EEG signal matrix, represents the number of EEG channels, Represents the time point within the decision window. Output Represents the classification label, corresponding to the auditory attention direction from 90° on the left to 90° on the right.

[0136] This classification problem can be solved by the B-TCNN network, such as Figure 5 As shown, B-TCNN processes the following in each decision window: Figure 6The brain-computer signal shown is mainly composed of three layers of 1D convolutional layers, which are designed to capture temporal patterns from EEG signals. In short, standard full-precision convolutional layers are used in the first layer to avoid the problem of feature extraction performance degradation that may be caused by the initial binarization stage. Next, two layers of binarized convolutional layers are used to further extract discriminative features. The binarization operation for these layers reduces computational overhead and memory usage while maintaining the ability to extract important features from the EEG data. These EEG features are then flattened using a global average pooling method.

[0137] In addition, each convolutional layer is followed by a batch normalization layer (BN) and a PRLUE function, and a pooling layer is provided after the batch normalization layer (BN) and the PRLUE function between two layers of binary convolutional layers.

[0138] In the classification stage, two independent classification heads are used to output the attention side information and spatial position information respectively. It is worth noting that since the ReLU activation function sets negative values ​​to zero, which may have an adverse effect on the feature extraction process in the binarization convolution, the PReLU activation function is used instead of the ReLU function.

[0139] Specifically, in a binary convolutional network, the calculation of weights and activation values ​​is simplified to binary operations, which is particularly suitable for deployment on edge devices. This embodiment relates to specific calculations of weights and activation functions during training, and approximate processing of gradient derivatives during back propagation.

[0140] In binarized convolution, the inference process is based on the following expression:

[0141]

[0142] in, Indicates output, ◎ indicates bit-by-bit operations, including XNOR and popcount; represents weight; Represents the activation value of the previous convolutional layer; Indicates the input channel, represents the kernel size, Indicates the output channel, Indicates the input length.

[0143] The symbolic function sign() is defined as:

[0144]

[0145] Since the sign function is not differentiable, a straight-through estimator (STE) is used to calculate the gradient during back-propagation. STE is used to approximate the gradient and train BNNs using the gradient descent method.

[0146] The approximate gradient of the STE function is expressed as follows:

[0147]

[0148] During forward propagation, both activations and weights are binarized using a sign function. During backpropagation, STE is used to approximate the gradient. This approach significantly reduces computational and memory requirements, making it well suited for hardware with limited resources. Figure 7 As shown, Figure 7 (a) is a schematic diagram of the parameters of back propagation. Figure 7 (b) in the figure is a schematic diagram after parameter derivation of back propagation.

[0149] By using binarized convolutions, the proposed B-TCNN is expected to achieve significant reductions in computational complexity and memory usage.

[0150] The B-TCNN model has two different classification heads to handle different problems in the multi-classification ASAD task; the classification heads are - the attention side (AS) head, which is used to determine the attention side of the stimulus, classifying it as left or right; and the spatial location (SL) head: used to predict the spatial location of the auditory stimulus, distinguishing five exemplary angles: 5°, 30°, 45°, 60°, and 90°.

[0151] This dual-head architecture enables the B-TCNN model to effectively optimize and improve the classification process of the above two problems.

[0152] set up and Represent the real labels of AS and SL headers respectively, and Represent the prediction values ​​of AS and SL heads respectively. Cross entropy loss of AS head The calculation is as follows:

[0153]

[0154] in, Indicates the total number of decision windows.

[0155] Similarly, the cross entropy loss of the SL head is The calculation is as follows:

[0156]

[0157] in, is the number of categories (e.g. ).

[0158] Combined loss function is the sum of the individual losses:

[0159]

[0160] By optimizing this combined loss function, the B-TCNN model improves the classification accuracy in the attention side and spatial location tasks, thereby improving the overall performance of the model in the ASAD task.

[0161] In addition, during the preprocessing of the EEG signals, a high-pass FIR filter was used to eliminate electrode drift, the data were re-referenced to the average of each channel, and the data were band-pass filtered from 1 to 50 Hz using a FIR low-pass filter. Then, the data were resampled to 128 Hz. In order to keep the focus on low-latency settings consistent, the EEG data were segmented using a sliding window with a window overlap of 50% and a decision window length of 1 second.

[0162] To evaluate the performance of the B-TCNN model, a five-fold cross validation method was used in the subject-dependent condition. During model training, the Adam optimizer was used, and the cosine learning rate schedule was used to start training from an initial learning rate of 0.001. The model was trained 100 times in total. The PyTorch framework was used and run on an Nvidia RTX 4090 GPU.

[0163] To demonstrate this, we can use the publicly available ASA dataset, which contains data from 20 participants with normal hearing and native Mandarin speakers (9 males and 11 females). The dataset contains 40 Mandarin stories, each about 1 to 1.5 minutes long, told by professional males or females. To ensure the participants' involvement and motivation in the task, each story is followed by three multiple-choice questions.

[0164] Participants performed a total of 20 trials, each involving a different combination of attention side and spatial location. In each trial, they were instructed to focus on an auditory stimulus in a specific direction and listen to two different stories simultaneously. The direction angle between the stimuli was adjusted every four trials in the order of ±90°, ±60°, ±45°, ±30°, and ±5°. Negative angles indicated left-hand directions and positive angles indicated right-hand directions, e.g. Figure 8 The EEG data were recorded by the BrainAmp system with a sampling rate of 500 Hz and 64 channels. The total length of EEG recording data was 12 hours to conduct a comprehensive analysis of multi-classification ASA perception.

[0165] Finally, the ASA dataset provides a channel attention convolutional neural network (CA-CNN) model as a reference model for evaluating future model competition benchmarks. 10 classification tasks are performed by the B-TCNN model and its performance is compared with the proposed B-TCNN model.

[0166] like Fig. 9 As shown in the figure, the classification accuracy of the B-TCNN model on each subject's data is significantly better than that of the CA-CNN model (paired t-test, p < 0.001). Specifically, B-TCNN achieved a higher average accuracy of 93.8%, while CA-CNN's accuracy was only 86.1%. In addition, the accuracy of the B-TCNN model across all subject data showed a smaller standard deviation (SD), highlighting its higher robustness relative to the CA-CNN model.

[0167] The B-TCNN model proposed in this study is compared with several advanced multi-classification techniques on the ASA dataset, mainly comparing their accuracy, number of parameters, and computational efficiency.

[0168] Among them, EEGNet is a model widely used in BCI that combines efficient depth-wise separable convolutions to effectively handle the temporal and spatial complexity in EEG data.

[0169] STAnet is an end-to-end model designed for EEG-driven ASAD tasks. The model dynamically assigns differentiated weights to EEG channels through a spatial attention mechanism and captures temporal patterns in EEG signals through a temporal attention mechanism.

[0170] ST-GCN considers the graph-based model of electrode placement and spatial relationships, which applies graph convolution operations to extract spatial patterns from multi-channel EEG signals. Its integrated temporal attention mechanism further enhances the processing performance of dynamic EEG data.

[0171] As shown in Table 1, B-TCNN performs significantly better than other existing technologies in the multi-classification ASAD task. Specifically, the accuracy of B-TCNN is 62.4% higher than that of ST-GCN, 8.7% higher than that of EEGNet, and 7.3% higher than that of STAnet, highlighting the high level of the model in dealing with ASAD tasks. Table 1 is as follows:

[0172] Table 1

[0173]

[0174] In terms of model efficiency, B-TCNN uses only 33K parameters, which is particularly outstanding compared to EEGNet's 163K, CA-CNN's 105K, and STAnet's 50K parameters. In addition, B-TCNN uses the multiply-accumulate (MACs) metric, so it only requires 1.4M MACs, making it the least computationally dependent model. These results highlight the efficiency and lightweight nature of B-TCNN, making it a practical solution for ASAD tasks.

[0175] It can be seen that EEG-based ASAD shows great potential in integrating user attention data to improve hearing aids and brain-computer interfaces (BCIs). In this study, a B-TCNN model was proposed, which combined with a binarization module to effectively capture the temporal features in EEG signals. B-TCNN showed relatively high accuracy in the 10-classification ASAD task with only 33K parameters, making it a more streamlined and efficient alternative to existing methods. These findings suggest that B-TCNN can significantly promote the practical application of ASAD and improve its effectiveness in real scenarios.

[0176] In addition, B-TCNN covers a wider range of auditory tasks and environments while incorporating real-time data from different sources to enhance its adaptability.

[0177] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0178] Based on the same inventive concept, the embodiment of the present application also provides an auditory spatial attention detection device for implementing the auditory spatial attention detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more auditory spatial attention detection device embodiments provided below can refer to the limitations of the auditory spatial attention detection method above, and will not be repeated here.

[0179] In an exemplary embodiment, Fig.10 As shown, a device for detecting auditory spatial attention is provided, comprising:

[0180] The feature acquisition module 1002 is used to acquire brain-computer signal features; the brain-computer signal features represent the brain waves of the target object that change over time;

[0181] A binarization module 1004 is used to perform binarization convolution processing on the brain-computer signal feature through a binarization convolution layer to obtain a target feature;

[0182] The detection module 1006 is used to detect auditory spatial attention information based on the target feature; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

[0183] In one embodiment, the feature acquisition module 1002 is used to:

[0184] Acquire brain-computer signals; the brain-computer signals are brain wave signals of the target object that change over time;

[0185] Through a full-precision convolution layer, feature extraction is performed on the feature data to obtain full-precision brain-computer signal features;

[0186] Among them, the activation function of the full-precision convolution layer is PRelu, and / or the full-precision brain-computer signal features are the input features of the first-layer binary convolution layer.

[0187] In one embodiment, the brain-computer signal feature is output by a convolution layer before a binarization convolution layer, and the brain-computer signal feature is binarized;

[0188] The binarization module 1004 is used for:

[0189] Performing an XOR operation on each bit in the brain-computer signal feature through a binary convolution kernel to obtain an XOR operation result for each bit;

[0190] Among the XOR operation results of each bit, the number of bits whose XOR operation results are preset values ​​is counted to obtain a binary convolution feature;

[0191] Performing activation processing on the binary convolution feature to obtain an activation value;

[0192] According to the activation value, a target feature is determined.

[0193] In one embodiment, the binarization module 1004 is used to:

[0194] If the pre-activation data corresponding to the binary convolution feature is less than a threshold, determining an activation value based on the pre-activation data;

[0195] If the pre-activation data corresponding to the binary convolution feature is greater than or equal to the threshold, adjusting the pre-activation data based on a preset weight, and determining an activation value based on the pre-activation data;

[0196] The preset weights are obtained by optimizing the training process of the binary convolution layer.

[0197] In one embodiment, the binarization module 1004 is used to:

[0198] Performing pooling processing according to the activation value to obtain a pooling feature;

[0199] Among them, the pooled feature is the feature output by the binary convolution layer, and the last pooled feature is the target feature.

[0200] In one embodiment, the auditory spatial attention information includes a focus direction and a sound source position angle, wherein the focus direction indicates that the sound source is on the left side or the right side of a preset direction, and the sound source position angle indicates a deviation angle of the sound source relative to the preset direction;

[0201] The detection module 1006 is used to:

[0202] By focusing on the side of the head, detecting the direction of the sound source for the target feature, and obtaining the focus direction;

[0203] The sound source position of the target feature is detected by the spatial position head to obtain the sound source position angle.

[0204] In one embodiment, the device further comprises a training module, wherein the training module is configured to:

[0205] Selecting the minimum value between the parameter gradient to be optimized of the binary convolution layer and the preset growth gradient as the candidate gradient;

[0206] Selecting the maximum value between the preset reduction gradient and the candidate gradient as the target gradient;

[0207] According to the target gradient, the gradient of the parameter to be optimized is optimized.

[0208] Each module in the above-mentioned auditory spatial attention detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0209] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an auditory spatial attention detection method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0210] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0211] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0212] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0213] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0214] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0215] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0216] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0217] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting auditory spatial attention, It is characterized in that The method comprises: Acquire brain-computer signal features; brain-computer signal features represent brain waves of the target object that change over time; Through the binary convolution layer, the brain-computer signal feature is subjected to binary convolution processing to obtain the target feature; Auditory spatial attention information is detected based on the target features; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

2. The method according to claim 1, It is characterized in that The obtaining of brain-computer signal features includes: Acquire brain-computer signals; the brain-computer signals are brain wave signals of the target object that change over time; Through a full-precision convolution layer, feature extraction is performed on the feature data to obtain full-precision brain-computer signal features; Among them, the activation function of the full-precision convolution layer is PRelu, and / or the full-precision brain-computer signal features are the input features of the first-layer binary convolution layer.

3. The method according to claim 1, It is characterized in that The brain-computer signal feature is output by a convolution layer before the binarization convolution layer, and the brain-computer signal feature is binarized; The binarization convolution layer is used to perform binarization convolution processing on the brain-computer signal feature to obtain the target feature, including: Performing an XOR operation on each bit in the brain-computer signal feature through a binary convolution kernel to obtain an XOR operation result for each bit; Among the XOR operation results of each bit, the number of bits whose XOR operation results are preset values ​​is counted to obtain a binary convolution feature; Performing activation processing on the binary convolution feature to obtain an activation value; According to the activation value, a target feature is determined.

4. The method according to claim 3, It is characterized in that The step of performing activation processing on the binary convolution feature to obtain an activation value includes: If the pre-activation data corresponding to the binary convolution feature is less than a threshold, determining an activation value based on the pre-activation data; If the pre-activation data corresponding to the binary convolution feature is greater than or equal to the threshold, adjusting the pre-activation data based on a preset weight, and determining an activation value based on the adjusted pre-activation data; The preset weights are obtained by optimizing the training process of the binary convolution layer.

5. The method according to claim 3, It is characterized in that Determining the target feature according to the activation value includes: Performing pooling processing according to the activation value to obtain a pooling feature; Among them, the pooled feature is the feature output by the binary convolution layer, and the last pooled feature is the target feature.

6. The method according to claim 1, It is characterized in that The auditory spatial attention information includes a focus direction and a sound source position angle, wherein the focus direction indicates that the sound source is on the left side or the right side of a preset direction, and the sound source position angle indicates the deviation angle of the sound source relative to the preset direction; The detecting of auditory spatial attention information based on the target feature comprises: By focusing on the side of the head, detecting the direction of the sound source for the target feature, and obtaining the focus direction; The sound source position of the target feature is detected by the spatial position head to obtain the sound source position angle.

7. The method according to claim 1, It is characterized in that The method further comprises: Selecting the minimum value between the parameter gradient to be optimized of the binary convolution layer and the preset growth gradient as the candidate gradient; Selecting the maximum value between the preset reduction gradient and the candidate gradient as the target gradient; According to the target gradient, the gradient of the parameter to be optimized is optimized.

8. An auditory spatial attention detection device, It is characterized in that The device comprises: A feature acquisition module is used to acquire brain-computer signal features; brain-computer signal features represent the brain waves of the target object that change over time; A binarization module, used for performing binarization convolution processing on the brain-computer signal features through a binarization convolution layer to obtain target features; A detection module is used to detect auditory spatial attention information based on the target feature; the auditory spatial attention information represents the sound source position information perceived by the target object through hearing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images