Method and system for decoding free-associative semantics using stereotactic electroencephalogram signals

Abstract

The present invention discloses a method and system for decoding free - associative semantics using stereotactic electroencephalogram signals. The method of the present invention includes obtaining free - associative stereotactic electroencephalogram signals when a user makes free associations to a target image; pre - processing the free - associative stereotactic electroencephalogram signals and then inputting them into a feature extraction module and a classifier in a pre - trained free - associative electroencephalogram signal decoding network, extracting electroencephalogram signal semantic features through the feature extraction module, and classifying the electroencephalogram signal semantic features through the classifier to obtain the image category of the target image. The free - associative electroencephalogram signal decoding network also includes an auxiliary feature extraction module, a domain discriminator, a pre - trained image feature extractor, and a global adaptation module. The present invention aims to solve the problems in existing research, such as the small amount of free - associative brain signal data, the high prediction difficulty, and the fact that existing deep - learning models cannot be well applied to the semantic decoding of full - channel stereotactic electroencephalogram signals.

Description

Technical Field

[0001] The present invention relates to the field of visual and memory decoding of brain signals, and particularly to a method and system for decoding free - association semantics using stereotactic electroencephalogram signals. Background Art

[0002] Decoding the semantic information of brain signals during free association in the human brain has always been one of the core goals of neuroscience research. By decoding the brain signals during free association to achieve the prediction of associated semantic information, it is expected to promote the understanding of the mechanisms of human brain association and memory, and is of great significance for the development of general brain - computer interfaces that are not restricted by experimental paradigms. Experiments for collecting brain signals during free association in the human brain are time - consuming and the amount of data is small. Therefore, it is difficult to use brain signals during free association to achieve the prediction of associated content. In recent years, significant progress has been made in the research on the physiological mechanisms of the brain during free association and recall. However, the research on using these physiological mechanisms to achieve semantic decoding of free - association brain signals is still very lacking. In addition, there are problems such as different numbers of signal channels and implanted brain regions among subjects for stereotactic electroencephalogram signals, and traditional deep - learning models for cortical electroencephalogram (EEG) are not well - suited for semantic decoding of full - channel stereotactic electroencephalogram signals. Summary of the Invention

[0003] The technical problem to be solved by the present invention: Aiming at the above - mentioned problems of the prior art, the present invention provides a method and system for decoding free - association semantics using stereotactic electroencephalogram signals. The present invention aims to solve the problems of small amount of free - association brain - signal data, high prediction difficulty, and the fact that existing deep - learning models are not well - suited for semantic decoding of full - channel stereotactic electroencephalogram signals in existing research.

[0004] To solve the above - mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A method for decoding free - association semantics using stereotactic electroencephalogram signals, comprising the following steps: obtaining free - association stereotactic electroencephalogram signals when a user makes free associations for a target image; preprocessing the free - association stereotactic electroencephalogram signals; inputting the preprocessed free - association stereotactic electroencephalogram signals into a feature extraction module and a classifier in a pre - trained free - association electroencephalogram signal decoding network, extracting electroencephalogram signal semantic features through the feature extraction module, and classifying the electroencephalogram signal semantic features through the classifier to obtain the image category of the target image. Besides the feature extraction module and the classifier, the free - association electroencephalogram signal decoding network further includes an auxiliary feature extraction module, a domain discriminator, a pre - trained image feature extractor, and a global adaptation module. The auxiliary feature extraction module has the same structure as the feature extraction module and shares parameters. The auxiliary feature extraction module is used to extract electroencephalogram signal semantic features for image - viewing stereotactic electroencephalogram signals when the user views an image. The domain discriminator is used to distinguish whether the electroencephalogram signal semantic features are obtained from free association or image viewing. The pre - trained image feature extractor is used to extract image features for the target image. The global adaptation module is used to map the electroencephalogram signal semantic features of the free - association stereotactic electroencephalogram signals and the image - viewing stereotactic electroencephalogram signals into the same semantic space.

[0006] Optionally, the training of the free - association electroencephalogram signal decoding network includes:

[0007] S101, for different types of target images, respectively obtaining two types of stereotactic electroencephalogram signal samples, namely free - association stereotactic electroencephalogram signal samples when the user makes free associations for the target image and image - viewing stereotactic electroencephalogram signal samples when the user views the image;

[0008] S102, preprocessing the two types of stereotactic electroencephalogram signal samples in the data set, and dividing the data set for the target image and its pre - processed two types of stereotactic electroencephalogram signal samples;

[0009] S103. Extract the semantic features of the electroencephalogram (EEG) signals from the preprocessed free association stereotactic EEG signal samples in the dataset using the feature extraction module, and extract the semantic features of the EEG signals from the preprocessed image viewing stereotactic EEG signal samples using the auxiliary feature extraction module. Input the semantic features of the two types of EEG signals into the global adaptation module to map them to the same semantic space. Then, use the domain discriminator to distinguish whether the semantic features of the EEG signals are obtained from free association or image viewing, and use the classifier for classification. Moreover, use the adversarial training method to continuously update the parameters of the feature extraction module, global adaptation module, classifier, and domain discriminator until the specified number of training rounds is completed or the classification accuracy of the classifier reaches the preset threshold. When updating the parameters of the feature extraction module, global adaptation module, and classifier, the loss function adopted for the feature extraction module, global adaptation module, and classifier is the weighted sum of the spatial aggregation loss and the feature alignment loss. The spatial aggregation loss is the distance between the class centers of the two tasks in the feature space, and the feature alignment loss is the distance between the features extracted from the stereotactic EEG signals and the image features extracted using the pre-trained model. The loss function adopted for the domain discriminator is the cross-entropy loss.

[0010] Optionally, the calculation function expression of the spatial aggregation loss is:

[0011] ,

[0012] where is the spatial aggregation loss, and respectively represent the semantic features of the EEG signals in the source domain and target domain of category c , where the source domain is the image viewing stereotactic EEG signal samples and the target domain is the free association stereotactic EEG signal samples. , represents the sample feature center of the target domain category c , represents the expected value of any semantic feature of the EEG signals in the sample set with respect to the source domain, represents the expected value of any semantic feature of the EEG signals in the sample set with respect to the target domain, is the sample set of the source domain, is the sample set of the target domain; the calculation function expression of the feature alignment loss is:

[0013] ,

[0014] where is the feature alignment loss, represents the number of samples. Indicates i The semantic features of EEG signals of samples, Indicates i The image corresponding to the sample is the image feature output by the pre-trained image feature extractor.

[0015] Optionally, the global adaptation module includes a self-attention mechanism within the domain and a cross-attention mechanism across domains, and mapping the EEG signal semantic features of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to the same semantic space includes: first, applying the self-attention mechanism to extract features for the EEG signal semantic features of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to achieve alignment within the domain, and then using the cross-attention mechanism to extract features for the aligned EEG signal semantic features within the domain to achieve mapping to the same semantic space.

[0016] Optionally, the preprocessing includes: using the label information of the brain area where each electrode in the input stereotactic EEG signal is located to eliminate the data of the electrodes located in the white matter area; for the electrodes on the same probe in the input stereotactic EEG signal, subtracting the data of the electrodes in the shallow brain area from the data of the adjacent electrodes in the deep brain area to achieve re-referencing of the stereotactic EEG signal; and performing a 50Hz notch filter on the re-referenced stereotactic EEG signal to remove power frequency interference.

[0017] Optionally, the feature extraction module includes a channel sampling module connected in sequence and N2 cascade-connected feature extraction units, the channel sampling module is used to randomly select channel slices to generate synthetic data for the input original data, and then expand it to a specified number of channels by splicing it to the original data; the feature extraction unit is composed of a time domain convolution of size 1×T and N1 cascade-connected information fusion modules, the information fusion module includes a layer of depth-separable convolution connected in sequence, a layer of convolution module with a convolution kernel size of 1×n for fusing the features of each channel, and a layer of convolution kernel size of The convolution module for fusing all channels and the residual connection, the residual connection combines the input features of the information fusion module with the convolution kernel size of The output features of the convolution module used to fuse all channels are fused and used as the output features of the information fusion module, where N1, N2 and T are natural numbers, n is the number of features, is the number of channels of stereotactic EEG signals.

[0018] Optionally, the image categories of the target image include buildings, natural scenes and faces.

[0019] In addition, the present invention also provides a system for decoding free - association semantics using stereotactic electroencephalogram (EEG) signals, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the method for decoding free - association semantics using stereotactic EEG signals.

[0020] In addition, the present invention also provides a computer - readable storage medium in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the method for decoding free - association semantics using stereotactic EEG signals through a processor.

[0021] In addition, the present invention also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the method for decoding free - association semantics using stereotactic EEG signals through a processor.

[0022] Compared with the prior art, the present invention mainly has the following advantages: The method of the present invention includes obtaining free - association stereotactic EEG signals when a user makes free associations to a target image; pre - processing the free - association stereotactic EEG signals and then inputting them into a feature extraction module and a classifier in a pre - trained free - association EEG signal decoding network, extracting semantic features of the EEG signals through the feature extraction module, and classifying the semantic features of the EEG signals through the classifier to obtain the image category of the target image. The free - association EEG signal decoding network also includes an auxiliary feature extraction module, a domain discriminator, a pre - trained image feature extractor, and a global adaptation module. The present invention can solve the problems in existing research, such as the small amount of free - association brain signal data, the high prediction difficulty, and the fact that existing deep - learning models are not well - suited for semantic decoding of full - channel stereotactic EEG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of the network structure of the free - association EEG signal decoding network in an embodiment of the present invention.

[0025] Figure 3 It is a schematic diagram of the training process of the free - association EEG signal decoding network in an embodiment of the present invention.

[0026] Figure 4 It is a schematic diagram of the network structure of the feature extraction module in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] As Figure 1 shown, the method for decoding free - association semantics using stereotactic EEG signals in this embodiment includes the following steps: obtaining the free - association stereotactic EEG signals when the user makes free associations for a target image; pre - processing the free - association stereotactic EEG signals; inputting the pre - processed free - association stereotactic EEG signals into the feature extraction module and classifier in a pre - trained free - association EEG signal decoding network, extracting the semantic features of the EEG signals through the feature extraction module, and classifying the semantic features of the EEG signals through the classifier to obtain the image category of the target image. In this embodiment, the image categories of the target images respectively include buildings, natural scenes, and faces. The free - association stereotactic EEG signals when the user makes free associations for the target image are the free - association stereotactic EEG signals when the user makes free associations for one of the three categories of target images of buildings, natural scenes, and faces. Free association is to make free associations for the images that have been viewed. By classifying the semantic features of the EEG signals through the classifier, the image categories of the target images obtained are buildings, natural scenes, or faces.

[0029] As Figure 2 shown, in this embodiment, in addition to the feature extraction module and classifier, the free - association EEG signal decoding network also includes an auxiliary feature extraction module, a domain discriminator, a pre - trained image feature extractor, and a global adaptation module. The auxiliary feature extraction module has the same structure as the feature extraction module and shares parameters. The auxiliary feature extraction module is used to extract the semantic features of the EEG signals for the image - viewing stereotactic EEG signals when the user views the image. The domain discriminator is used to distinguish whether the semantic features of the EEG signals are obtained from free association or image viewing. The pre - trained image feature extractor is used to extract image features for the target image; the global adaptation module is used to map the semantic features of the free - association stereotactic EEG signals and the image - viewing stereotactic EEG signals to the same semantic space.

[0030] As Figure 3 shown, the training of the free - association EEG signal decoding network in this embodiment includes:

[0031] S101, for different types of target images, respectively obtaining two types of stereotactic EEG signal samples, namely, the free - association stereotactic EEG signal samples when the user makes free associations for the target image and the image - viewing stereotactic EEG signal samples when the user views the image;

[0032] S102, pre - processing the two types of stereotactic EEG signal samples in the data set, and dividing the data set for the target image and its two types of pre - processed stereotactic EEG signal samples;

[0033] S103. Extract the semantic features of the electroencephalogram (EEG) signals from the preprocessed free association stereotactic EEG signal samples in the dataset using the feature extraction module, extract the semantic features of the EEG signals from the preprocessed image viewing stereotactic EEG signal samples using the auxiliary feature extraction module, input the semantic features of the two types of EEG signals into the global adaptation module to map them to the same semantic space, then distinguish whether the semantic features of the EEG signals are obtained from free association or image viewing through the domain discriminator, and classify them through the classifier. Use the method of adversarial training to continuously update the parameters of the feature extraction module, global adaptation module, classifier, and domain discriminator until the specified number of training rounds is completed or the classification accuracy of the classifier reaches the preset threshold. When updating the parameters of the feature extraction module, global adaptation module, and classifier, the loss function used is the weighted sum of the spatial aggregation loss and the feature alignment loss. The spatial aggregation loss is the distance between the class centers of the two tasks in the feature space, and the feature alignment loss is the distance between the features extracted from the stereotactic EEG signals and the image features extracted using the pre-trained model. The loss function used for the domain discriminator is the cross-entropy loss. It should be noted that the method of adversarial training combined with the domain discriminator is an existing method, and its implementation details will not be elaborated here. In the training method of the free association EEG signal decoding network (cross-task domain adaptation method) in this embodiment, the cross-task domain adaptation training that uses the brain signals when the subject views relevant content to assist in the free association semantic decoding significantly improves the accuracy of the free association EEG signal decoding network for free association semantic decoding.

[0034] Among them, the calculation function expression of the spatial aggregation loss is:

[0035] ,

[0036] Among them, is the spatial aggregation loss, and respectively represent the semantic features of the EEG signals in the source domain and target domain of category c , where the source domain is the image viewing stereotactic EEG signal sample, and the target domain is the free association stereotactic EEG signal sample. , represents the sample feature center of the target domain category c , represents the sample set regarding the source domain in which any semantic feature of the EEG signal is the expected value, represents the sample set regarding the target domain in which any semantic feature of the EEG signal is the expected value, is the sample set of the source domain, is the sample set of the target domain.

[0037] Among them, the calculation function expression of feature alignment loss is:

[0038] ,

[0039] in, is the feature alignment loss, represents the number of samples, Indicates i The semantic features of EEG signals of samples, Indicates i The image corresponding to the sample is the image feature output by the pre-trained image feature extractor.

[0040] In this embodiment, the global adaptation module includes a self-attention mechanism within the domain and a cross-attention mechanism across domains, and mapping the semantic features of the EEG signals of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to the same semantic space includes: first, applying the self-attention mechanism to extract features for the semantic features of the EEG signals of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to achieve alignment within the domain, and then using the cross-attention mechanism to extract features for the semantic features of the EEG signals after alignment within the domain to achieve mapping to the same semantic space.

[0041] In this embodiment, preprocessing includes: using the label information of the brain area where each electrode in the input stereotactic EEG signal is located to eliminate the data of the electrode located in the white matter area; for the electrodes on the same probe in the input stereotactic EEG signal, subtracting the data of the electrode in the shallow brain area from the data of the adjacent electrode in the deep brain area to achieve re-reference of the stereotactic EEG signal; and performing a 50Hz notch filter on the stereotactic EEG signal after re-reference to remove power frequency interference.

[0042] In this embodiment, in step S102, the target image and its two types of preprocessed stereoscopic EEG signal samples are divided into data sets: all stereoscopic EEG signals when viewing the image and part of the stereoscopic EEG signals during free association are used for training, and the remaining stereoscopic EEG signals during free association are used for testing, and multi-fold cross-validation is performed.

[0043] like Figure 4As shown, in this embodiment, the feature extraction module includes a channel sampling module and N2 cascaded feature extraction units connected in sequence. The channel sampling module is used to expand the input raw data to a specified number of channels by randomly selecting channel slices to generate synthetic data and splicing it to the raw data; the feature extraction unit is composed of a time-domain convolution of size 1×T and N1 cascaded information fusion modules connected. The information fusion module includes a depthwise separable convolution in sequence, a convolution module with a convolution kernel size of 1×n for fusing features of each channel, and a convolution module with a convolution kernel size of for fusing all channels, as well as a residual connection. The residual connection fuses the input features of the information fusion module with the output features of the convolution module with a convolution kernel size of for fusing all channels as the output features of the information fusion module, where N1, N2, and T are natural numbers, n is the number of features, is the number of channels of the stereotactic electroencephalogram signal.

[0044] In this embodiment, the feature extraction module has the following advantages: First, by expanding to the specified number of channels by randomly selecting channel slices and splicing, it can effectively adapt to data inputs with different numbers of channels; Second, the feature extraction unit composed of a time-domain convolution of size 1×T and N1 cascaded information fusion modules connected, and the network structure of the information fusion module including a depthwise separable convolution in sequence, a convolution module with a convolution kernel size of 1×n for fusing features of each channel, a convolution module with a convolution kernel size of for fusing all channels, as well as a residual connection, can effectively capture cross-time and cross-channel dependencies at multiple levels.

[0045] To verify the effectiveness of the training method (cross-task domain adaptation method) of the free association electroencephalogram signal decoding network in this embodiment in improving the accuracy of free association semantic decoding, in this embodiment, the stereotactic electroencephalogram signal data of three self-collected experimental subjects is used, and the accuracy and variance obtained by the five-fold cross-validation method are shown in Table 1. In Table 1, the accuracy and variance of semantic decoding using only free association electroencephalogram signals (the deep learning model used is the free association electroencephalogram signal decoding network in this embodiment) are also compared.

[0046] Table 1 Comparison of classification accuracies of using the cross-task domain adaptation method in this embodiment and training only with free association stereotactic electroencephalogram data (classification of free association content categories, i.e., buildings, natural scenes, and faces)

[0047]

[0048] As can be seen from Table 1, compared with training and testing using only the stereotactic EEG signals during free association, the training method (cross-task domain adaptation method) of the free association EEG signal decoding network in the method of this embodiment uses the stereotactic EEG signals during image viewing and free association for joint training, and can achieve higher accuracy and more stable prediction of the free association semantic types.

[0049] To verify the effect of the feature extraction module in the method of this embodiment on decoding semantic features, this embodiment uses the stereotactic EEG signal data of three self-collected experimental subjects, and the accuracy and variance obtained by using the five-fold cross-validation method are shown in Table 2. At the same time, Table 2 also compares the accuracy and variance of using a convolutional neural network for semantic feature extraction.

[0050] Table 2 Comparison of classification accuracies of using the feature extraction module in the method of this embodiment and using a convolutional neural network (classification of free association content categories, i.e., buildings, natural scenes, and faces)

[0051]

[0052] As can be seen from Table 2, compared with using a convolutional neural network to extract features from stereotactic EEG signals, using the feature extraction module in the method of this embodiment can better achieve the extraction of free association semantics encoded by stereotactic EEG signals, and achieve higher accuracy and more stable prediction.

[0053] In summary, to solve the problems of small amount of free association data and high prediction difficulty in existing research, this embodiment utilizes the "mental time retrospection" theory in the episodic memory theory, that is, when recalling, the brain will reactivate a series of representations at the time of the event occurrence, map the semantic features of the stereotactic EEG signals when the subject views the image and the stereotactic EEG signals when freely associating with the image to the same semantic space, and align with the image semantics. At the same time, aiming at the characteristics of large differences in the number of stereotactic EEG signal channels and insufficient modeling of the dependence relationship between the stereotactic EEG signal channels and time by existing models, a multi-level feature extraction module that can effectively capture cross-time and cross-channel dependence relationships is designed, which effectively improves free association semantic decoding and improves prediction accuracy and robustness.

[0054] In addition, this embodiment also provides a system for decoding free association semantics using stereotactic EEG signals, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the method for decoding free association semantics using stereotactic EEG signals.

[0055] In addition, this embodiment also provides a computer-readable storage medium storing a computer program or instructions, which are programmed or configured to execute the method for decoding free association semantics using stereotactic electroencephalogram signals through a processor.

[0056] In addition, this embodiment also provides a computer program product including a computer program or instructions, which are programmed or configured to execute the method for decoding free association semantics using stereotactic electroencephalogram signals through a processor.

[0057] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present invention can be in the form of a method, a system, or a computer program product. Therefore, the present invention can be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can be in the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes. The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks. These computer program instructions can also be stored in a computer-readable memory capable of guiding the computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks. These computer program instructions can also be loaded onto the computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0058] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A method for decoding free - association semantics using stereotactic electroencephalogram signals, characterized in that, It includes the following steps: obtaining the free association stereotactic electroencephalogram signals when the user makes free associations for the target image; preprocessing the free association stereotactic electroencephalogram signals; Inputting the preprocessed free association stereotactic electroencephalogram signals into the feature extraction module and classifier in the pre-trained free association electroencephalogram signal decoding network, extracting the electroencephalogram signal semantic features through the feature extraction module, and classifying the electroencephalogram signal semantic features through the classifier to obtain the image category of the target image. In addition to the feature extraction module and classifier, the free association electroencephalogram signal decoding network also includes an auxiliary feature extraction module, a domain discriminator, a pre-trained image feature extractor, and a global adaptation module. The auxiliary feature extraction module has the same structure as the feature extraction module and shares parameters. The auxiliary feature extraction module is used to extract the electroencephalogram signal semantic features for the image viewing stereotactic electroencephalogram signals when the user views the image. The domain discriminator is used to distinguish whether the electroencephalogram signal semantic features are obtained from free association or image viewing. The pre-trained image feature extractor is used to extract the image features for the target image. The global adaptation module is used to map the electroencephalogram signal semantic features of the free association stereotactic electroencephalogram signals and the image viewing stereotactic electroencephalogram signals to the same semantic space; The training of the free association electroencephalogram signal decoding network includes: S101, for different types of target images, respectively obtaining two types of stereotactic electroencephalogram signal samples, namely, the free association stereotactic electroencephalogram signal samples when the user makes free associations for the target image and the image viewing stereotactic electroencephalogram signal samples when the user views the image; S102, preprocessing the two types of stereotactic electroencephalogram signal samples in the dataset, and dividing the target image and its preprocessed two types of stereotactic electroencephalogram signal samples into the dataset; S103, using the feature extraction module to extract the electroencephalogram signal semantic features from the preprocessed free association stereotactic electroencephalogram signal samples in the dataset, using the auxiliary feature extraction module to extract the electroencephalogram signal semantic features from the preprocessed image viewing stereotactic electroencephalogram signal samples, inputting the two types of electroencephalogram signal semantic features into the global adaptation module to map them to the same semantic space, then distinguishing whether the electroencephalogram signal semantic features are obtained from free association or image viewing through the domain discriminator and classifying them through the classifier, and continuously updating the parameters of the feature extraction module, the global adaptation module, the classifier, and the domain discriminator in an adversarial training manner until the specified number of training rounds is completed or the classification accuracy of the classifier reaches the preset threshold. When updating the parameters of the feature extraction module, the global adaptation module, and the classifier, the loss function adopted is the weighted sum of the spatial aggregation loss and the feature alignment loss. The spatial aggregation loss is the distance between the class centers of the two tasks in the feature space. The feature alignment loss is the distance between the features extracted from the stereotactic electroencephalogram signals and the image features extracted using the pre-trained model. The loss function adopted for the domain discriminator is the cross-entropy loss.

2. The method for decoding free-association semantics using stereotactic electroencephalogram signals according to claim 1, wherein The calculation function expression of the spatial aggregation loss is: ， Among them, is the spatial aggregation loss, and respectively represent the EEG signal semantic features of the source domain and the target domain of class c , where the source domain is the image viewing stereotactic EEG signal sample, and the target domain is the free association stereotactic EEG signal sample. , where the source domain is the image viewing stereotactic EEG signal sample, and the target domain is the free association stereotactic EEG signal sample. represents the sample feature center of the target domain class c , represents the expected value of any EEG signal semantic feature in the sample set about the source domain, represents the expected value of any EEG signal semantic feature in the sample set about the target domain, is the sample set of the source domain, is the sample set of the target domain; the calculation function expression of the feature alignment loss is: ， Among them, is the feature alignment loss, represents the number of samples, represents the i semantic features of the electroencephalogram signal of the th sample, i and represents the image features output by the pre-trained image feature extractor for the image corresponding to the i th sample.

3. The method for decoding free - association semantics using stereotactic electroencephalogram signals according to claim 1, wherein The global adaptation module includes a self-attention mechanism within the domain and a cross-attention mechanism across domains. The mapping of the EEG signal semantic features of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to the same semantic space includes: first, applying the self-attention mechanism to extract features for the EEG signal semantic features of the free association stereotactic EEG signals and the image viewing stereotactic EEG signals to achieve alignment within the domain, and then using the cross-attention mechanism to extract features for the aligned EEG signal semantic features within the domain to achieve mapping to the same semantic space.

4. The method for decoding free-association semantics using stereotactic electroencephalogram signals according to claim 1, wherein The preprocessing includes: using the label information of the brain area where each electrode in the input stereotactic EEG signal is located to eliminate the data of the electrode located in the white matter area; for the electrodes on the same probe in the input stereotactic EEG signal, subtracting the data of the adjacent electrode in the deep brain area from the data of the electrode in the shallow brain area to achieve re-reference of the stereotactic EEG signal; and performing 50Hz notch filtering on the stereotactic EEG signal after re-reference to remove power frequency interference.

5. The method for decoding free - association semantics using stereotactic electroencephalogram signals according to claim 1, wherein The feature extraction module includes a channel sampling module and N2 cascaded feature extraction units connected in sequence. The channel sampling module is used to expand the input raw data to a specified number of channels by randomly selecting channel slices to generate synthetic data and splicing it to the raw data; the feature extraction unit is composed of a time-domain convolution of size 1×T and N1 cascaded information fusion modules connected. The information fusion module includes a depthwise separable convolution layer, a convolution module with a convolution kernel size of 1×n for fusing features of each channel, and a convolution module with a convolution kernel size of for fusing all channels, as well as a residual connection. The residual connection fuses the input features of the information fusion module with the output features of the convolution module with a convolution kernel size of for fusing all channels as the output features of the information fusion module, where N1, N2, and T are natural numbers, n is the number of features, is the number of channels of the stereotactic electroencephalogram signal.

6. The method for decoding free - associative semantics using stereotactic electroencephalogram signals according to claim 1, wherein The image categories of the target images include buildings, natural scenes and faces.

7. A system for decoding free - association semantics using stereotactic electroencephalogram signals, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the method for decoding free association semantics using stereotactic EEG signals as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the method for decoding free association semantics using stereotactic EEG signals as described in any one of claims 1 to 6 through a processor.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the method for decoding free association semantics using stereotactic EEG signals as described in any one of claims 1 to 6 through a processor.

Images