Image three-classification decoding method and device based on brain hole and eye movement features

Abstract

The present disclosure relates to a kind of image three classification decoding method, device, electronic equipment and storage medium based on brain hole and eye movement characteristics.The method includes: based on the electroencephalogram sample data of decoding model training constructs electroencephalogram space-time feature extraction model, electroencephalogram frequency domain feature extraction model and extracts electroencephalogram space-time feature, electroencephalogram frequency domain feature;Based on the eye movement signal sample data constructs eye movement time domain feature extraction model and extracts time domain feature, based on electroencephalogram space-time feature, electroencephalogram frequency domain feature, time domain feature constructs the classification model of electroencephalogram and eye movement characteristics, realizes the image three classification decoding based on brain hole and eye movement characteristics.The present disclosure makes full use of the synergistic advantage of electroencephalogram and eye movement dual-mode brain information data, extracts the space-time-frequency joint feature of dual-mode brain information, improves the image three classification decoding efficiency of brain-computer interface based on fast sequence visual presentation, promotes the fusion of brain-computer interface in small sample image target fast retrieval and brain-computer intelligence.

Description

Technical Field

[0001] This disclosure relates to the fields of brain-computer interfaces and image classification. Specifically, it relates to an image three-classification decoding method, device, electronic device, and computer-readable storage medium based on brain hole and eye movement features. Background Technology

[0002] Rapid serial visual presentation (RSVP) is a crucial paradigm in brain-computer interface (BCI) applications. It continuously presents stimuli in the same space at a high rate per second using a visual stimulator. When a target image of interest to the subject appears in the presented image sequence, it induces corresponding event-related potentials (EPPs). Through brain signal decoding algorithms, rapid retrieval of massive amounts of images can be achieved. RSVP-BCI systems leverage the image processing and understanding capabilities of the human visual system, offering unique advantages over machine vision and manual methods in image retrieval. Compared to machine vision, RSVP-BCI does not require large sample sizes and possesses stronger adaptability and generalization capabilities, especially for newly emerging or variant target types. Compared to manual recognition, RSVP-BCI is faster, automatically, quickly, and accurately recognizing up to 15 images per second. RSVP-BCI has broad application prospects in the field of quickly searching, locating, and classifying sparse target objects of interest in massive images, especially in scenarios where the samples of target objects of interest are scarce or highly variable. For example, it can be applied in fields such as target detection in complex backgrounds, targeted medical image screening, image recognition, and brainprint authentication.

[0003] However, while most existing RSVP-BCI technologies and methods focus on binary classification problems—distinguishing between targets and non-targets—methods for multi-class classification of images are still lacking. Furthermore, with the increasing demands of RSVP-BCI system applications, RSVP-BCI-based binary image classification methods can no longer meet practical application requirements. These methods not only need to identify whether something is a target but also differentiate the target type, i.e., multi-class classification problems. Current RSVP-BCI-based three-class image classification methods exhibit low decoding and recognition accuracy and cannot effectively distinguish the brain response patterns and key components induced by dual targets.

[0004] Therefore, one or more methods are needed to solve the above problems.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide an image three-classification decoding method, apparatus, electronic device, and computer-readable storage medium based on brainwave and eye-tracking features, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0007] According to one aspect of this disclosure, an image three-classification decoding method based on brainwave and eye-tracking features is provided, comprising:

[0008] The first sample data used for decoding model training is used to extract the spatiotemporal features of each sample in the first sample data based on hierarchical discriminant analysis. The spatiotemporal features of each sample in the first sample data are then correlated with the category labels of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model.

[0009] The second sample data used for decoding model training is used to extract the EEG frequency domain features of each sample in the second sample data based on the co-space pattern method. The EEG frequency domain features of each sample in the second sample data are then correlated with the category labels of each sample in the second sample data to construct an EEG frequency domain feature extraction model.

[0010] The third sample data used for decoding model training is filtered by a time filter to extract the temporal features of the index data of each sample in the third sample data. The temporal features of the index data of each sample in the third sample data are then correlated with the category labels of each sample in the third sample data to construct an eye-tracking temporal feature extraction model.

[0011] The feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data are added together to generate EEG and eye movement feature vectors. Based on the feedforward convolutional neural network model, the EEG and eye movement feature vectors are correlated with the corresponding category labels in the first, second, and third samples to complete the establishment of the classification model of EEG and eye movement features.

[0012] In one exemplary embodiment of this disclosure, the method further includes:

[0013] The first and second sample data used for decoding model training are EEG sample data.

[0014] The third sample data used for decoding model training is eye movement signal sample data.

[0015] In one exemplary embodiment of this disclosure, the method further includes:

[0016] Before extracting the spatiotemporal features of each sample in the first sample data and the EEG frequency domain features of each sample in the second sample data, the EEG data is filtered based on a bandpass filter with a preset frequency.

[0017] In one exemplary embodiment of this disclosure, the method further includes:

[0018] The first sample data used for decoding model training is decomposed into four time windows based on a preset number of data points. Based on hierarchical discriminant analysis, the spatiotemporal features of the four time windows of each sample in the first sample data are extracted. The spatiotemporal features of the four time windows of each sample in the first sample data are added together to generate a 1*4 spatiotemporal feature vector. The spatiotemporal feature vector of each sample in the first sample data is then correlated with the category label of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model.

[0019] In one exemplary embodiment of this disclosure, the method further includes:

[0020] The sample data in the second sample data used for decoding model training is decomposed into 3 frequency bands based on a preset frequency range. Based on the common spatial pattern method, the EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are extracted. The EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are added together to generate a 1*3 EEG frequency domain feature vector. The correspondence between the EEG frequency domain feature vector of each sample in the second sample data and the category label of each sample in the second sample data is established to construct the EEG frequency domain feature extraction model.

[0021] In one exemplary embodiment of this disclosure, the method further includes:

[0022] The sample data in the third sample data used for decoding model training is decomposed into three indicator data groups based on preset category labels. The temporal features of the three indicator data of each sample in the third sample data are extracted by a temporal filter. The temporal features of the three indicator data of each sample in the third sample data are added together to generate a 1*3 temporal feature vector. The temporal feature vector of the indicator data of each sample in the third sample data is correlated with the category label of each sample in the third sample data to construct an eye-tracking temporal feature extraction model.

[0023] In one exemplary embodiment of this disclosure, the method further includes:

[0024] The feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data are added together to generate a 1*10 EEG and eye movement feature vector. Based on the feedforward convolutional neural network model, the EEG and eye movement feature vectors are correlated with the corresponding category labels in the first, second, and third samples to complete the establishment of the classification model of EEG and eye movement features.

[0025] In one exemplary embodiment of this disclosure, the method further includes:

[0026] The fourth sample data is used to generate the spatiotemporal features, frequency domain features, and temporal features of the index data based on the EEG spatiotemporal feature extraction model, the EEG frequency domain feature extraction model, and the eye movement temporal feature extraction model, respectively. The feature vectors corresponding to the spatiotemporal features, EEG frequency domain features, and temporal features of the index data are added together to generate the EEG and eye movement feature vectors of the fourth sample data.

[0027] Using the EEG and eye movement feature vectors of the fourth sample data as input, a classification model based on EEG and eye movement features is used to complete the image three-class decoding based on brainwave and eye movement features.

[0028] In one exemplary embodiment of this disclosure, the method further includes:

[0029] The fourth sample data is either test data or image data to be decoded in three categories.

[0030] In one aspect of this disclosure, an image three-classification decoding device based on brainwave and eye-tracking features is provided, comprising:

[0031] The EEG spatiotemporal feature extraction model construction module is used to extract the spatiotemporal features of each sample in the first sample data used for decoding model training based on hierarchical discriminant analysis, and to establish a correspondence between the spatiotemporal features of each sample in the first sample data and the category labels of each sample in the first sample data, thereby constructing the EEG spatiotemporal feature extraction model.

[0032] The EEG frequency domain feature extraction model construction module is used to extract the EEG frequency domain features of each sample in the second sample data used for decoding model training based on the co-space pattern method, and establish a correspondence between the EEG frequency domain features of each sample in the second sample data and the category label of each sample in the second sample data to construct the EEG frequency domain feature extraction model.

[0033] The eye-tracking temporal feature extraction model construction module is used to extract the temporal features of the index data of each sample in the third sample data through a temporal filter, and establish a correspondence between the temporal features of the index data of each sample in the third sample data and the category label of each sample in the third sample data to construct the eye-tracking temporal feature extraction model.

[0034] The EEG and eye-tracking feature classification model construction module is used to add the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data to generate EEG and eye-tracking feature vectors. Based on the feedforward convolutional neural network model, the module establishes a correspondence between the EEG and eye-tracking feature vectors and the corresponding category labels of the EEG and eye-tracking feature vectors in the first, second, and third samples to construct a classification model for EEG and eye-tracking features.

[0035] In one aspect of this disclosure, an electronic device is provided, comprising:

[0036] Processor; and

[0037] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.

[0038] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to any one of the preceding claims.

[0039] An exemplary embodiment of this disclosure discloses an image tri-classification decoding method based on brainwave and eye-tracking features. The method includes: constructing an EEG spatiotemporal feature extraction model and an EEG frequency domain feature extraction model based on EEG sample data trained with a decoding model, and extracting EEG spatiotemporal and frequency domain features; constructing an eye-tracking time domain feature extraction model based on eye-tracking signal sample data and extracting time domain features; and constructing a classification model of EEG and eye-tracking features based on EEG spatiotemporal, frequency, and time domain features to achieve image tri-classification decoding based on brainwave and eye-tracking features. This disclosure fully utilizes the synergistic advantages of bimodal brain information data (EEG and eye-tracking) to extract spatiotemporal and frequency joint features of bimodal brain information, improving the efficiency of image tri-classification decoding based on a fast sequence visual presentation brain-computer interface, and promoting the rapid retrieval of targets in small sample images and the integration of brain-computer intelligence.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0041] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0042] Figure 1 A flowchart of an image three-classification decoding method based on brain hole and eye movement features according to an exemplary embodiment of the present disclosure is shown;

[0043] Figure 2 A schematic block diagram of an image three-classification decoding apparatus based on brain hole and eye movement features according to an exemplary embodiment of the present disclosure is shown;

[0044] Figure 3 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown; and

[0045] Figure 4 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0046] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0047] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0048] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0049] In this example embodiment, a three-class image decoding method based on brainwave and eye-tracking features is first provided; see reference Figure 1As shown, this image tri-classification decoding method based on brainwave and eye movement features may include the following steps:

[0050] Step S110: Using the first sample data used for decoding model training, extract the spatiotemporal features of each sample in the first sample data based on hierarchical discriminant analysis, and establish a correspondence between the spatiotemporal features of each sample in the first sample data and the category labels of each sample in the first sample data to construct an EEG spatiotemporal feature extraction model.

[0051] Step S120: Using the second sample data used for decoding model training, extract the EEG frequency domain features of each sample in the second sample data based on the co-space pattern method, and establish a correspondence between the EEG frequency domain features of each sample in the second sample data and the category labels of each sample in the second sample data to construct an EEG frequency domain feature extraction model.

[0052] Step S130: The third sample data used for decoding model training is processed by a time filter to extract the temporal features of the index data of each sample in the third sample data, and a correspondence is established between the temporal features of the index data of each sample in the third sample data and the category label of each sample in the third sample data to construct an eye-tracking temporal feature extraction model.

[0053] Step S140: Add the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data to generate EEG and eye movement feature vectors. Based on the feedforward convolutional neural network model, establish a correspondence between the EEG and eye movement feature vectors and the corresponding category labels of the EEG and eye movement feature vectors in the first, second, and third samples to complete the establishment of the classification model of EEG and eye movement features.

[0054] An exemplary embodiment of this disclosure discloses an image tri-classification decoding method based on brainwave and eye-tracking features. The method includes: constructing an EEG spatiotemporal feature extraction model and an EEG frequency domain feature extraction model based on EEG sample data trained with a decoding model, and extracting EEG spatiotemporal and frequency domain features; constructing an eye-tracking time domain feature extraction model based on eye-tracking signal sample data and extracting time domain features; and constructing a classification model of EEG and eye-tracking features based on EEG spatiotemporal, frequency, and time domain features to achieve image tri-classification decoding based on brainwave and eye-tracking features. This disclosure fully utilizes the synergistic advantages of bimodal brain information data (EEG and eye-tracking) to extract spatiotemporal and frequency joint features of bimodal brain information, improving the efficiency of image tri-classification decoding based on a fast sequence visual presentation brain-computer interface, and promoting the rapid retrieval of targets in small sample images and the integration of brain-computer intelligence.

[0055] The following will further explain an image three-classification decoding method based on brain hole and eye movement features in this example embodiment.

[0056] In step S110, the first sample data used for decoding model training can be used to extract the spatiotemporal features of each sample in the first sample data based on hierarchical discriminant analysis, and a correspondence can be established between the spatiotemporal features of each sample in the first sample data and the category labels of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model.

[0057] In this example embodiment, the method further includes:

[0058] The first and second sample data used for decoding model training are EEG sample data.

[0059] The third sample data used for decoding model training is eye movement signal sample data.

[0060] In this example embodiment, the method further includes:

[0061] Before extracting the spatiotemporal features of each sample in the first sample data and the EEG frequency domain features of each sample in the second sample data, the EEG data is filtered based on a bandpass filter with a preset frequency.

[0062] In this example embodiment, the EEG data is filtered using a 1-30Hz bandpass filter to remove interfering noise signals.

[0063] In this example embodiment, the method further includes:

[0064] The first sample data used for decoding model training is decomposed into four time windows based on a preset number of data points. Based on hierarchical discriminant analysis, the spatiotemporal features of the four time windows of each sample in the first sample data are extracted. The spatiotemporal features of the four time windows of each sample in the first sample data are added together to generate a 1*4 spatiotemporal feature vector. The spatiotemporal feature vector of each sample in the first sample data is then correlated with the category label of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model.

[0065] In this example embodiment, the spatiotemporal features of each EEG sample are extracted using hierarchical discriminant analysis (HDCA) on the EEG sample data used for decoding model training. Combined with the category label corresponding to each EEG sample, the spatiotemporal feature extraction model based on hierarchical discriminant analysis is trained to establish the spatiotemporal feature extraction model of EEG.

[0066] In this example embodiment, hierarchical discriminant analysis (HDCA) is used to extract the spatiotemporal features of each EEG sample used for training the decoding model. The training dataset contains 5000 EEG samples, each with 1000 data points. Each sample corresponds to a 0, 1, or 2 category label, where 0 represents non-target EEG, 1 represents target type 1 EEG, and 2 represents target type 2 EEG. HDCA is used to extract the spatiotemporal features of each EEG sample. During feature extraction, each EEG sample is decomposed into four time windows, each with 250 data points, yielding the spatiotemporal feature value for each window. The feature values ​​from the four windows are then concatenated into a 1*4 feature vector. Combined with the category label corresponding to each EEG sample, the spatiotemporal feature extraction model based on HDCA is trained to establish the spatiotemporal feature extraction model.

[0067] In step S120, the second sample data used for decoding model training can be used to extract the EEG frequency domain features of each sample in the second sample data based on the co-space pattern method, and establish a correspondence between the EEG frequency domain features of each sample in the second sample data and the category labels of each sample in the second sample data to construct an EEG frequency domain feature extraction model.

[0068] In this example embodiment, the method further includes:

[0069] The sample data in the second sample data used for decoding model training is decomposed into 3 frequency bands based on a preset frequency range. Based on the common spatial pattern method, the EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are extracted. The EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are added together to generate a 1*3 EEG frequency domain feature vector. The correspondence between the EEG frequency domain feature vector of each sample in the second sample data and the category label of each sample in the second sample data is established to construct the EEG frequency domain feature extraction model.

[0070] In this example embodiment, the EEG sample data used for decoding model training is divided into several frequency bands using a frequency filter, and the EEG frequency domain features of each frequency band are extracted using the Common Spatial Pattern (CSP) method. Combined with the category label corresponding to each EEG sample, the EEG frequency domain feature extraction model based on the Common Spatial Pattern (CSP) method is trained to establish the EEG frequency domain feature extraction model.

[0071] In this example embodiment, the EEG sample data used for decoding model training is divided into several frequency bands using a frequency filter. There are 5000 EEG sample data points used for decoding model training, each with 1000 data points. Each sample data point corresponds to a 0, 1, or 2 category label, where 0 represents EEG signals not corresponding to the target, 1 represents EEG signals corresponding to target type 1, and 2 represents EEG signals corresponding to target type 2. The filter divides each EEG sample into three frequency bands: 1-5Hz, 1-10Hz, and 1-32Hz. The frequency domain features of each frequency band are extracted using the Common Spatial Pattern (CSP) method, resulting in a 1*3 feature vector. This vector, combined with the category label corresponding to each EEG sample, is used to train the EEG frequency domain feature extraction model based on the CSP method, thus establishing the EEG frequency domain feature extraction model.

[0072] In step S130, the third sample data used for decoding model training can be filtered by a time filter to extract the temporal features of the index data of each sample in the third sample data, and a correspondence can be established between the temporal features of the index data of each sample in the third sample data and the category labels of each sample in the third sample data to construct an eye-tracking temporal feature extraction model.

[0073] In this example embodiment, the sample data in the third sample data used for decoding model training is decomposed into three indicator data groups based on preset category labels. The temporal features of the three indicator data of each sample in the third sample data are extracted by a temporal filter. The temporal features of the three indicator data of each sample in the third sample data are added together to generate a 1*3 temporal feature vector. The temporal feature vector of the indicator data of each sample in the third sample data is then correlated with the category label of each sample in the third sample data to construct an eye-tracking temporal feature extraction model.

[0074] In this example embodiment, for the eye movement signal sample data used for decoding model training, the temporal features of pupil area, pupil area change rate, and pupil area change acceleration of each eye movement data sample are extracted by a temporal filter. Combined with the category label corresponding to each eye movement sample, the temporal feature extraction model of eye movement based on the temporal filter is trained to establish the temporal feature extraction model of eye movement.

[0075] In this example embodiment, for the eye movement signal sample data used for decoding model training, temporal features of pupil area, pupil area change rate, and pupil area change acceleration for each eye movement data sample are extracted using a temporal filter. The decoding model training uses 5000 eye movement sample data, each with 1000 data points. Each sample data corresponds to a 0, 1, or 2 category label, where 0 represents eye movements not corresponding to the target, 1 represents eye movements corresponding to target type 1, and 2 represents eye movements corresponding to target type 2. A differential filter is used for the temporal filter to extract eye movement features, resulting in 1*3 temporal features of the eye movements. Combined with the category label corresponding to each eye movement sample, the eye movement temporal feature extraction model based on the differential filter is trained to establish the eye movement temporal feature extraction model.

[0076] In step S140, the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data can be added together to generate EEG and eye movement feature vectors. Based on the feedforward convolutional neural network model, the EEG and eye movement feature vectors are associated with the corresponding category labels in the first, second, and third samples to establish a correspondence, thereby completing the establishment of the classification model for EEG and eye movement features.

[0077] In this example embodiment, the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data are added together to generate a 1*10 EEG and eye movement feature vector. Based on the feedforward convolutional neural network model, the EEG and eye movement feature vectors are correlated with the corresponding category labels in the first, second, and third samples to complete the establishment of the classification model for EEG and eye movement features.

[0078] In this example embodiment, the features of each sample extracted in steps S110-S130 are concatenated into a one-dimensional feature vector, which is then input into a feedforward convolutional neural network (FF-CNN) classifier. The FF-CNN classifier is trained by combining the category label corresponding to each sample to establish a classification model for EEG and eye movement features.

[0079] In this example embodiment, the method further includes:

[0080] The fourth sample data is used to generate the spatiotemporal features, frequency domain features, and temporal features of the index data based on the EEG spatiotemporal feature extraction model, the EEG frequency domain feature extraction model, and the eye movement temporal feature extraction model, respectively. The feature vectors corresponding to the spatiotemporal features, EEG frequency domain features, and temporal features of the index data are added together to generate the EEG and eye movement feature vectors of the fourth sample data.

[0081] Using the EEG and eye movement feature vectors of the fourth sample data as input, a classification model based on EEG and eye movement features is used to complete the image three-class decoding based on brainwave and eye movement features.

[0082] In this example embodiment, the method further includes:

[0083] The fourth sample data is either test data or image data to be decoded in three categories.

[0084] In this example embodiment, 1250 EEG and eye-tracking sample data used for decoding model testing are input into the feature extraction model trained in steps S110-S130 to obtain the EEG and eye-tracking features of each test sample. Following the method in step 5, the extracted features are concatenated into a one-dimensional feature vector, which is then input into the classifier trained in step S140 to obtain the decoding result of the test sample, thus completing the image three-classification decoding based on EEG and eye-tracking fusion.

[0085] In the embodiment of this example, the image three-classification decoding method based on EEG and eye-tracking fusion fully utilizes the synergistic advantages of EEG and eye-tracking bimodal brain information data, extracts the spatiotemporal-frequency joint features of bimodal brain information, and improves the efficiency of RSVP-BCI-based image three-classification decoding. This is of great significance for promoting the development of RSVP-BCI in rapid target retrieval of small sample images and brain-computer intelligence fusion technology.

[0086] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0087] Furthermore, in this example embodiment, an image three-classification decoding device based on brainwave and eye-tracking features is also provided. (See reference...) Figure 4As shown, the image three-classification decoding device 400 based on brainwave and eye movement features may include: an EEG spatiotemporal feature extraction model construction module 210, an EEG frequency domain feature extraction model construction module 220, an eye movement temporal domain feature extraction model construction module 230, and an EEG and eye movement feature classification model construction module 240. Wherein:

[0088] The EEG spatiotemporal feature extraction model construction module 210 is used to extract the spatiotemporal features of each sample in the first sample data used for decoding model training based on hierarchical discriminant analysis, and establish a correspondence between the spatiotemporal features of each sample in the first sample data and the category labels of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model.

[0089] The EEG frequency domain feature extraction model construction module 220 is used to extract the EEG frequency domain features of each sample in the second sample data for decoding model training based on the co-space pattern method, and establish a correspondence between the EEG frequency domain features of each sample in the second sample data and the category label of each sample in the second sample data to construct the EEG frequency domain feature extraction model.

[0090] The eye-tracking temporal feature extraction model construction module 230 is used to extract the temporal features of the index data of each sample in the third sample data through a temporal filter, and establish a correspondence between the temporal features of the index data of each sample in the third sample data and the category label of each sample in the third sample data to construct the eye-tracking temporal feature extraction model.

[0091] The EEG and eye movement feature classification model construction module 240 is used to add the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data to generate EEG and eye movement feature vectors. Based on the feedforward convolutional neural network model, the module establishes a correspondence between the EEG and eye movement feature vectors and the corresponding category labels of the EEG and eye movement feature vectors in the first, second, and third samples to construct a classification model for EEG and eye movement features.

[0092] The specific details of each of the above-mentioned image three-classification decoding device modules based on brain hole and eye movement features have been described in detail in the corresponding image three-classification decoding method based on brain hole and eye movement features, so they will not be repeated here.

[0093] It should be noted that although several modules or units of an image three-classification decoding device 400 based on brainwave and eye-tracking features have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0094] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0095] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”

[0096] The following reference Figure 3 To describe an electronic device 300 according to such an embodiment of the present invention. Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0097] like Figure 3 As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including storage unit 320 and processing unit 310), and a display unit 340.

[0098] The storage unit stores program code that can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 310 can perform actions such as... Figure 1 Steps S110 to S140 are shown in the diagram.

[0099] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 3201 and / or cache memory 3202, and may further include read-only memory (ROM) 3203.

[0100] Storage unit 320 may also include a program / utility 3204 having a set (at least one) program module 3203, such program module 3203 including but not limited to: operating system, one or more application programs, other program modules and program data, each of these examples or some combination of these may include an implementation of a network environment.

[0101] Bus 350 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0102] Electronic device 300 can also communicate with one or more external devices 370 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 350. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0103] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0104] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0105] refer to Figure 4 As shown, a program product 400 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0109] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0111] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0112] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image tri-classification decoding method based on brainwave and eye-tracking features, characterized in that, The method includes: The sample data in the first sample data used for decoding model training is decomposed into 4 time windows based on a preset number of data points. Based on hierarchical discriminant analysis, the spatiotemporal features of the 4 time windows of each sample in the first sample data are extracted. The spatiotemporal features of the 4 time windows of each sample in the first sample data are added together to generate a 1*4 spatiotemporal feature vector. The spatiotemporal feature vector of each sample in the first sample data is then correlated with the category label of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model. The sample data in the second sample data used for decoding model training is decomposed into 3 frequency bands based on a preset frequency range. Based on the common space pattern method, the EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are extracted. The EEG frequency domain features of the 3 frequency bands of each sample in the second sample data are added together to generate a 1*3 EEG frequency domain feature vector. The correspondence between the EEG frequency domain feature vector of each sample in the second sample data and the category label of each sample in the second sample data is established to construct the EEG frequency domain feature extraction model. The sample data in the third sample data used for decoding model training is decomposed into 3 indicator data groups based on preset category labels. The temporal features of the 3 indicator data of each sample in the third sample data are extracted by a temporal filter. The temporal features of the 3 indicator data of each sample in the third sample data are added together to generate a 1*3 temporal feature vector. The temporal feature vector of the indicator data of each sample in the third sample data is correlated with the category label of each sample in the third sample data to construct an eye-tracking temporal feature extraction model. The feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data are added together to generate a 1*10 EEG and eye movement feature vector. Based on the feedforward convolutional neural network model, the EEG and eye movement feature vectors are correlated with the corresponding category labels in the first, second, and third samples to complete the establishment of the classification model of EEG and eye movement features.

2. The method as described in claim 1, characterized in that, The method further includes: The first and second sample data used for decoding model training are EEG sample data. The third sample data used for decoding model training is eye movement signal sample data.

3. The method as described in claim 1, characterized in that, The method further includes: Before extracting the spatiotemporal features of each sample in the first sample data and the EEG frequency domain features of each sample in the second sample data, the EEG data is filtered based on a bandpass filter with a preset frequency.

4. The method as described in claim 1, characterized in that, The method further includes: The fourth sample data is used to generate the spatiotemporal features, frequency domain features, and temporal features of the index data based on the EEG spatiotemporal feature extraction model, the EEG frequency domain feature extraction model, and the eye movement temporal feature extraction model, respectively. The feature vectors corresponding to the spatiotemporal features, EEG frequency domain features, and temporal features of the index data are added together to generate the EEG and eye movement feature vectors of the fourth sample data. Using the EEG and eye movement feature vectors of the fourth sample data as input, a classification model based on EEG and eye movement features is used to complete the image three-class decoding based on brainwave and eye movement features.

5. The method as described in claim 4, characterized in that, The method further includes: The fourth sample data is either test data or image data to be decoded in three categories.

6. An image three-classification decoding device based on brainwave and eye-tracking features, characterized in that, The device includes: The EEG spatiotemporal feature extraction model construction module is used to decompose each sample data in the first sample data used for decoding model training into 4 time windows based on a preset number of data points. Based on hierarchical discriminant analysis, it extracts the spatiotemporal features of each sample in the first sample data into 4 time windows. It adds the spatiotemporal features of each sample in the first sample data into 4 time windows to generate a 1*4 spatiotemporal feature vector. It establishes a correspondence between the spatiotemporal feature vector of each sample in the first sample data and the category label of each sample in the first sample data to construct the EEG spatiotemporal feature extraction model. The EEG frequency domain feature extraction model construction module is used to decompose each sample data in the second sample data used for decoding model training into 3 frequency bands based on a preset frequency interval. Based on the common space mode method, it extracts the EEG frequency domain features of each sample in the second sample data in the 3 frequency bands. It adds the EEG frequency domain features of each sample in the second sample data in the 3 frequency bands to generate a 1*3 EEG frequency domain feature vector. It establishes a correspondence between the EEG frequency domain feature vector of each sample in the second sample data and the category label of each sample in the second sample data, and constructs the EEG frequency domain feature extraction model. The eye-tracking temporal feature extraction model construction module is used to decompose each sample data in the third sample data used for decoding model training into three indicator data groups based on preset category labels. Through a temporal filter, the temporal features of the three indicator data of each sample in the third sample data are extracted. The temporal features of the three indicator data of each sample in the third sample data are added together to generate a 1*3 temporal feature vector. The temporal feature vector of the indicator data of each sample in the third sample data is associated with the category label of each sample in the third sample data to construct the eye-tracking temporal feature extraction model. The EEG and eye-tracking feature classification model construction module is used to add the feature vectors corresponding to the spatiotemporal features of each sample in the first sample data, the EEG frequency domain features of each sample in the second sample data, and the temporal domain features of the index data of each sample in the third sample data to generate a 1*10 EEG and eye-tracking feature vector. Based on the feedforward convolutional neural network model, the module establishes a correspondence between the EEG and eye-tracking feature vectors and the corresponding category labels of the EEG and eye-tracking feature vectors in the first, second, and third samples to construct a classification model for EEG and eye-tracking features.

7. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images