An adaptive brain-computer information fusion classification method and system

Abstract

The present invention belongs to the technical field of the application of brain-computer interface technology, and discloses an adaptive brain-computer information fusion classification method and system. The adaptive brain-computer information fusion classification method includes: respectively extracting brain response features and stimulus image features; respectively training linear SVMs, calculating the classification sensitivity index of each feature as the feature reliability label; using the feature reliability label to respectively establish feature reliability prediction models for brain responses and stimulus images; weighting and cascading the brain response and image features according to the reliability label to construct a fused feature set; establishing a linear SVM classification model on the fused feature set, and using the linear SVM classification model to classify the input fused features. The present invention can effectively evaluate the reliability of brain response features and image features, adaptively fuse brain responses and image features, reduce the risk of fusion negative gain caused by feature reliability, and effectively improve the performance of image classification tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interface technology applications, and particularly relates to an adaptive brain-computer information fusion and classification method and system. Background Art

[0002] In recent years, artificial intelligence technologies represented by deep learning have developed rapidly, and their performance in image classification tasks has exceeded that of humans. Although the development of deep learning technology is deeply influenced by the visual processing mechanisms of the human and other primate brains and there is a certain degree of similarity between the two, the existing technologies only simulate the feedforward mechanism of the ventral visual pathway of the brain, resulting in the current deep learning systems being easily affected by factors such as target pose, external environment, and adversarial samples and sharply degrading, and still far from achieving human-like strong generalization ability. In many open military application environments, manual interpretation by visual recognition experts is still the main method of image analysis and decision-making, but manual interpretation is difficult to meet the working requirements of long hours, high intensity, and real-time. The brain-computer hybrid intelligent computing method based on brain-computer interface technology collaboratively processes brain response information and image representation information, realizes information interaction from different sources, combines their respective advantages, and provides a new processing paradigm for image classification tasks in complex open environments.

[0003] Currently, there are mainly three types of technologies used for brain-computer information fusion classification in the industry. Existing technology one: Image classification technology based on the brain response representation space; Existing technology two: Image classification technology based on information fusion; Existing technology three: Image classification technology based on shared subspace learning. Existing technology one, "Bridging the Semantic Gap via Functional Brain Imaging" (IEEE Transactions on Multimedia: 2012, 14(2): 314-325), established an association model between brain responses and low-level features of stimulus videos. By mapping the low-level video features to the brain response representation space, video classification was then achieved. Since it is difficult for traditional computer vision methods to extract efficient image features, and brain responses have strong category selectivity, mapping the low-level image features to the brain response representation space with high-level semantic information can effectively improve the semantic expression ability of the low-level image features, thereby enhancing the classification performance. However, with the development of deep learning technology, the classification ability of image features extracted by deep learning technology has, to some extent, reached or even exceeded the brain response information that can be captured currently. Therefore, the application of related technologies in the field of deep learning has gradually decreased. Existing technology two, "Combining brain computer interfaces with vision for object categorization" (IEEE Conference on Computer Vision and Pattern Recognition: 2008, 1-8), used the method of kernel alignment to select the brain responses most relevant to the low-level image features for data fusion and classified the fused features. This technology did not effectively utilize the complementary information between brain responses and image features, making it difficult to obtain a complete information joint representation. Therefore, its classification performance in complex classification tasks is low. Existing technology three, "Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features" (IEEE Transactions on Pattern Analysis and Machine Intelligence: 2020), used deep learning technology to constrain the image feature space to approximate the spatial distribution of brain responses to obtain the strong generalization ability similar to the brain. However, this technology requires a large amount of paired data of brain responses and stimulus images to train the shared subspace, and the acquisition of brain responses is very expensive and difficult. Existing public datasets are difficult to support the training of this model.At the same time, none of the above three existing technologies consider the reliability of brain responses and image features. In the process of information fusion, poor representation information will produce negative gain to the fused features, thereby affecting the classification performance of the fused features. How to effectively evaluate the reliability of representation information is also a defect of the existing technologies.

[0004] Through the above analysis, the problems and defects existing in the existing technologies are as follows:

[0005] (1) Currently, the classification ability of the image features extracted by using deep learning technologies has reached or even exceeded the brain response information that can be captured currently to a certain extent. Therefore, the application of the existing image classification technologies based on the brain response representation space in the field of deep learning is gradually decreasing.

[0006] (2) It is very difficult for the existing image classification technologies based on information fusion to obtain a joint representation with complete information, so their classification performance in complex classification tasks is low.

[0007] (3) The existing image classification technologies based on shared subspace learning require a large amount of paired data of brain responses and stimulus images to train the shared subspace. However, the acquisition of brain responses is very expensive and difficult, and the existing publicly available datasets are difficult to support the training of this model.

[0008] (4) None of the existing technologies consider the reliability of brain responses and image features. In the process of information fusion, poor representation information will produce negative gain to the fused features, thereby affecting the classification performance of the fused features. How to effectively evaluate the reliability of representation information is also a defect of the existing technologies.

[0009] The difficulty and significance of solving the above problems and defects are as follows: How to propose an effective method for evaluating feature reliability under a small amount of paired datasets of brain responses and stimulus images, and adaptively fuse brain responses and image features according to the reliability of brain responses and image features. The significance of the present invention lies in proposing an adaptive brain-computer information fusion classification method and system based on the reliability of brain responses and image features. This method can effectively reduce the negative gain problem generated in the process of information fusion. Summary of the Invention

[0010] Aiming at the problems existing in the existing technologies, the present invention provides an adaptive brain-computer information fusion classification method and system, in particular an adaptive brain-computer information fusion classification method and system based on feature reliability.

[0011] The present invention is implemented as follows. An adaptive brain-computer information fusion classification method, the adaptive brain-computer information fusion classification method includes two stages: training and inference. In the training stage, class supervision information is used to constrain the model to learn the sample space distribution of the training data set. In the inference stage, the brain response and image data are directly input into the model entrance for forward inference to obtain the classification result.

[0012] Among them, in the training stage, it includes: on the paired brain response and stimulus image data sets, the brain response and stimulus image feature sets are respectively extracted; for the brain response and stimulus image feature sets, linear SVMs are respectively trained, and the classification sensitivity index of each feature is calculated as the feature reliability label; according to the feature reliability label, the feature reliability prediction models of the brain response and stimulus image are respectively trained; according to the feature reliability label, the brain response features and image features are weighted and cascaded to form a fusion feature set, and a linear SVM model is trained on the fusion feature set for classification.

[0013] In the inference stage, it includes: select paired brain responses and stimulus images, and extract the corresponding brain response features and stimulus image features respectively; input the brain response features and image features into the corresponding feature reliability prediction models respectively to estimate the reliability values of the features; according to the predicted feature reliability values, the brain response features and image features are weighted and cascaded, and the fusion features are input into the linear SVM model to output the classification result.

[0014] Furthermore, the adaptive brain-computer information fusion classification method includes the following steps:

[0015] Step 1, training stage:

[0016] (1) On the paired brain response and stimulus image data sets, the brain response feature set and the stimulus image feature set are respectively extracted;

[0017] (2) On the extracted brain response and stimulus image feature sets, linear SVMs are respectively trained, and the classification sensitivity index of each feature is calculated as the reliability label of the feature;

[0018] (3) Using the feature reliability label as the supervision information, a feature reliability prediction model is trained on each of the brain response and image feature sets;

[0019] (4) Using the feature reliability label as the weight, the brain response features and image features are weighted and cascaded to form a fusion feature set;

[0020] (5) Using the fusion feature set as the input, a linear SVM model is trained to achieve classification.

[0021] Step 2, inference stage:

[0022] (1) On the paired brain response and stimulus image test datasets, extract the brain response feature set and the stimulus image feature set respectively;

[0023] (2) Input the extracted brain response features and image features into the corresponding feature reliability prediction models respectively to estimate the reliability of the brain response features and the image features;

[0024] (3) According to the predicted feature reliability values, weight and cascade the brain response features and the image features to obtain the fused features;

[0025] (4) Input the fused features into a linear SVM model and output the classification result of the fused features.

[0026] Furthermore, in step one, in the above step (1) on the paired brain response and stimulus image datasets, extracting the brain response feature set and the stimulus image feature set respectively includes:

[0027] (1) Extraction of the brain response feature set:

[0028] 1) Load the brain response dataset and average the brain responses captured when the same stimulus image is presented multiple times;

[0029] 2) Select the electrodes placed in the inferior temporal lobe region IT and extract the brain response signals corresponding to the electrodes;

[0030] 3) On the brain response signal of each electrode, calculate the mean along the time dimension to remove the influence of the time dimension;

[0031] 4) Flip the processed brain response to a 1*168-dimensional feature, which is used as the average brain response feature of the stimulus image on each electrode in the IT region.

[0032] (2) Extraction of the image feature set:

[0033] 1) Use the PyTorch deep learning framework to load the ResNet34 model, remove the fully connected layer of the network, and set the model parameter "pretrained = True" to load the ImageNet pre-trained model parameters;

[0034] 2) Load the stimulus image dataset, input the stimulus images into the pre-trained ResNet34 model, and obtain the output features of the convolutional layer. The dimension of the output image features is 512;

[0035] 3) Input the image feature set extracted in step 2) into the principal component analysis model, set the model output parameter "n_components = 168", and reduce the dimension of the image features to the same 168 dimensions as the brain response features.

[0036] Further, in step one, on the brain response and stimulus image feature set extracted in step (2), a linear SVM is trained respectively and a feature sensitivity index is calculated, including:

[0037] (1) Randomly divide the extracted brain response feature set into a training set and a test set at a ratio of 1:1.

[0038] (2) Combine the categories of brain responses in pairs, and sequentially extract the brain response features corresponding to the two categories in the training set, and input them into the linear SVM to train the binary classifiers of these two categories, and save the model parameters until all binary classifiers of all category combinations are trained.

[0039] (3) For each brain response feature f in the test set, sequentially load the binary classifier model parameters w(i, j) related to its category i, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification result of f on the binary classifier w(i, j) for other categories j The classification is correct as 1 and incorrect as 0; for each binary classifier w(i, j), statistically calculate the decision distance d of the brain response features f of categories i and j in the test set, and take the maximum value as d max and the minimum value as d min , calculate the classification confidence C of f, where C = (d - d min ) / (d max - d min ).

[0040] (4) Exchange the training set and the test set, repeat steps (2) and (3), complete a process of cross-validation, and obtain all binary classification results of all features f in the brain response feature set and the classification confidence C.

[0041] (5) Calculate the true positive rate of classification of each brain response feature f according to the following formula

[0042]

[0043] where N represents the total number of categories of features.

[0044] Calculate the average false positive rate FPR of the brain response features of the i-th category i :

[0045]

[0046] where mean represents taking the mean.

[0047] (6) Repeat steps (2) to (6) 10 times to complete ten two-fold cross-validations, and calculate the average of each brain response feature f and the average false positive rate of the i-th type of brain response feature

[0048] (7) According to the average of the brain response feature f calculated in step (6) and the average false positive rate of the i-th type of brain response feature Calculate the classification sensitivity of f according to the following formula

[0049]

[0050] where Z(.) represents the inverse of the Gaussian cumulative distribution.

[0051] Furthermore, in step one, in step (3), using the feature reliability label as the supervision information, train a feature reliability prediction model on each of the brain response and the image feature set, including:[[]]

[0052] (1) Use the PyTorch deep learning framework to build feature reliability prediction models with the same structure respectively; among them, the prediction model adopts a fully connected neural network structure, and the number of neurons in the input layer, hidden layer, and output layer are 168, 32, and 1 respectively. The hidden layer uses the ELU activation function, and the output value is the predicted value of the feature reliability.

[0053] (2) Use the reliability value calculated in step one (2) as the reliability label of the brain response feature, and divide the brain response feature set into a training set and a test set at a ratio of 4:1; on the training set, use the mean square error loss function L MSE Supervise the training process of the prediction model:[[]]

[0054]

[0055] where y represents the predicted value of the feature reliability, y' represents the value of the feature reliability label, and n represents the batch size; the prediction model training uses the mean square error loss function and the Adam optimizer, with an initial learning rate of 1e-3, a learning rate decay of 0.1, and a total of 100 epochs of training. The learning rate decays once every 40 epochs.

[0056] Furthermore, in step one, in step (4), using the feature reliability label as the weight, weighted cascade the brain response feature and the image feature to form a fusion feature set, including:[[]]

[0057] (1) Use the reliability value calculated in step one (2) as the reliability label of the brain response and the image feature. In the training set divided in step one (3), extract the paired brain response and image features respectively, and obtain the brain response feature reliability of each pair of stimulus images i and the reliability of image features The fusion weights of the brain response and image features are calculated respectively according to the following formulas and

[0058]

[0059]

[0060] The fusion feature training set F is obtained by adaptively weighted cascading the fusion features according to the fusion weights of the brain response and image features i :

[0061]

[0062] where B i , I i respectively represent the brain response feature and the image feature of the stimulus image i

[0063] (2) According to the fusion feature training set F obtained in step (1) i Train a linear SVM classification model

[0064] Furthermore, in step two, the adaptive fusion classification of the brain response and the image feature set in the inference stage includes:

[0065] (1) Load the stimulus images in the test set, and at the same time extract the brain response features and image features

[0066] (2) Input the paired brain response features and image features extracted from the test set into the trained brain response feature reliability prediction model and image feature reliability prediction model in step one (3) respectively, and output the reliability values of the brain response and image features

[0067] (3) According to the feature reliability values output in step (2), cascade the brain response features and image features in the way of step one (4) with weights to obtain the fusion features

[0068] (4) Input the fusion features obtained in step (3) into the trained linear SVM model in step one (5), and output the classification result

[0069] Another object of the present invention is to provide an adaptive brain-computer information fusion classification system applying the above-mentioned adaptive brain-computer information fusion classification method. The adaptive brain-computer information fusion classification system includes a feature extraction module, a prediction model construction module, a feature reliability prediction module, an adaptive brain-computer information fusion module, and a fusion feature classification module

[0070] A feature extraction module, which is used to load paired brain response data and stimulus image data, and extract brain response features and stimulus image features respectively according to the methods mentioned above;

[0071] A prediction model construction module, which is used to train linear SVMs respectively for the brain response and stimulus image feature sets, calculate the classification sensitivity index of each feature as the feature reliability label; and use the feature reliability label to establish the feature reliability prediction models for the brain response and stimulus image respectively;

[0072] A feature reliability evaluation module, which is used to load the parameters of the brain response feature reliability prediction model and the image feature reliability prediction model, input the extracted brain response features and image features into the corresponding reliability prediction models respectively, and output their feature reliability values;

[0073] An adaptive brain-computer information fusion module, which is used to weighted cascade the brain response and image features according to the output feature reliability values to obtain the fused features;

[0074] A fused feature classification module, which is used to load the parameters of the linear SVM classification model, input the obtained fused features into the linear SVM model, and output the classification category and classification probability.

[0075] Another object of the present invention is to provide a computer device, which includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

[0076] (1) Training stage: On the paired brain response and stimulus image data sets, extract the brain response and stimulus image feature sets respectively; train linear SVMs respectively for the brain response and stimulus image feature sets, calculate the classification sensitivity index of each feature as the feature reliability label; according to the feature reliability label, train the feature reliability prediction models for the brain response and stimulus image respectively; weighted cascade the brain response features and image features according to the feature reliability label to form a fused feature set, and train a linear SVM model on the fused feature set for classification.

[0077] (2) Inference stage: Select paired brain response and stimulus images, extract the corresponding brain response features and stimulus image features respectively; input the brain response features and image features into the corresponding feature reliability prediction models respectively to estimate the reliability values of the features; weighted cascade the brain response features and image features according to the predicted feature reliability values, and input the fused features into the linear SVM model to output the classification result.

[0078] Another object of the present invention is to provide an information data processing terminal, which is used to implement the adaptive brain-computer information fusion classification system.

[0079] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: The adaptive brain-computer information fusion classification method provided by the present invention can effectively evaluate the reliability of brain response features and image features, and adaptively fuse brain responses and image features. The feature reliability evaluation method of the present invention is simple and does not require a large amount of paired brain response and stimulus image data sets. It can effectively utilize the respective advantages of brain responses and image features to construct a complete information joint representation, reduce the risk of fusion negative gain caused by feature reliability, and effectively improve the performance of image classification tasks. The related method can also be quickly migrated to other multi-modal information fusion tasks, can solve the problem of generating negative gain in the multi-modal information fusion process, and has high practicability.

[0080] Based on the paired brain response and stimulus image data sets, the adaptive brain-computer information fusion classification method based on feature reliability proposed by the present invention comprehensively considers the data requirements of the model and the reliability of features, can obtain complete information fusion features, effectively improve the performance of image classification tasks, and can be used for image classification tasks of joint modeling of visual expert brain responses and image features in complex environments. At the same time, the present invention uses the feature classification sensitivity index to evaluate the reliability of features, and adaptively fuses information according to the reliability of brain responses and image features on the paired brain response and stimulus image sets, effectively reducing the negative gain in the information fusion process and improving the performance of image classification.

[0081] The present invention uses the absolute error to describe the prediction performance. The average absolute error on the IT brain response and image features is 0.4462 (the range of feature reliability values is 0-5), and the variance is 0.002, indicating that the special certificate reliability evaluation model proposed by the present invention can stably and accurately predict the reliability of IT brain responses and image features. Table 1 shows the comparison results of the classification accuracies of the adaptive brain-computer information fusion features based on feature reliability, the fusion features of direct concatenation of brain responses and image features, and the brain response (IT) and image single-modal features of the present invention on the test set. The results show that the classification accuracy of the fusion features obtained by the present invention is much higher than that of the image single-modal features. Among them, the classification accuracy of the fusion features obtained by direct feature concatenation is on average 4.45% higher than that of the image single-modal features, and the classification accuracy of the fusion features obtained by the present invention is on average 5.92% higher than that of the image single-modal features. The adaptive feature fusion method proposed by the present invention has a 1.47% higher classification accuracy than the fusion features obtained by direct feature concatenation, indicating that the adaptive brain-computer information fusion classification method based on feature reliability proposed by the present invention can effectively reduce the risk of generating negative gain in the brain-computer information fusion process and improve the classification accuracy. Therefore, the present invention has more practical application value and has a wide application prospect under the new paradigm of brain-computer information collaborative work.

[0082] Table 1 Experimental Results

[0083] Brief Description of the Drawings

[0084] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments of the present invention. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0085] Figure 1 is the adaptive brain-computer information fusion classification method provided by the embodiments of the present invention.

[0086] Figure 2 is the process schematic diagram of the training stage and the inference stage provided by the embodiments of the present invention.

[0087] Figure 3 is the structural block diagram of the adaptive brain-computer information fusion classification system provided by the embodiments of the present invention;

[0088] In the figure: 1. Feature extraction module; 2. Prediction model construction module; 3. Feature reliability evaluation module; 4. Adaptive brain-computer information fusion module; 5. Fusion feature classification module.

[0089] Figure 4 is the structural schematic diagram of the adaptive brain-computer information fusion classification system provided by the embodiments of the present invention.

[0090] Figure 5 is the network structure schematic diagram of the feature reliability prediction module provided by the embodiments of the present invention.

[0091] Figure 6 is the schematic diagram of some stimulus images for classification tasks provided by the embodiments of the present invention. Detailed Embodiments

[0092] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further details the present invention in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0093] Aiming at the problems existing in the prior art, the present invention provides an adaptive brain-computer information fusion classification method and system, which can be used to efficiently fuse brain responses and image features and perform classification. The following describes the present invention in detail with reference to the drawings.

[0094] Such as Figure 1As shown in the figure, the adaptive brain-computer information fusion classification method provided by the embodiment of the present invention includes the following steps:

[0095] S101, extract the brain response feature set and the stimulus image feature set respectively;

[0096] S102, for the brain response and the stimulus image feature set, train a linear SVM respectively, calculate the classification sensitivity index of each feature, and use it as the feature reliability label;

[0097] S103, use the feature reliability label to establish the feature reliability prediction models of the brain response and the stimulus image respectively;

[0098] S104, weight and cascade the brain response and the image features according to the reliability label to construct a fusion feature set;

[0099] S105, establish a linear SVM classification model on the fusion feature set, and use the linear SVM classification model to classify the input fusion features.

[0100] The process schematic diagrams of the training stage and the inference stage provided by the embodiment of the present invention are as Figure 2 shown.

[0101] As Figure 3 shown, the adaptive brain-computer information fusion classification system provided by the embodiment of the present invention includes:

[0102] Feature extraction module 1, which is used to load the paired brain response data and stimulus image data, and extract the brain response features and stimulus image features respectively according to the methods mentioned above;

[0103] Prediction model construction module 2, which is used to train a linear SVM respectively for the brain response and the stimulus image feature set, calculate the classification sensitivity index of each feature, and use it as the feature reliability label; use the feature reliability label to establish the feature reliability prediction models of the brain response and the stimulus image respectively;

[0104] Feature reliability evaluation module 3, which is used to load the brain response feature reliability prediction model parameters and the image feature reliability prediction model parameters, input the extracted brain response features and image features into the corresponding reliability prediction models respectively, and output their feature reliability values;

[0105] Adaptive brain-computer information fusion module 4, which is used to weight and cascade the brain response and the image features according to the output feature reliability values to obtain the fusion features;

[0106] Fusion feature classification module 5, which is used to load the linear SVM classification model parameters, input the obtained fusion features into the linear SVM model, and output the classification category and classification probability.

[0107] The structural schematic diagram of the adaptive brain-computer information fusion classification system provided by the embodiment of the present invention is as Figure 4 shown.

[0108] The technical solution of the present invention will be further described below in conjunction with specific embodiments.

[0109] Existing brain-computer information fusion classification technologies basically do not consider the reliability issues of brain response features and image features, and it is easy to have the problem of negative gain during the information fusion process, thus reducing the classification performance of the fusion features. Based on the paired brain response and stimulus image data sets, the proposed adaptive brain-computer information fusion classification method based on feature reliability comprehensively considers the data requirements of the model and the reliability of features, can obtain complete information fusion features, effectively improve the performance of the image classification task, and can be used for the image classification task of joint modeling of visual expert brain responses and image features in complex environments.

[0110] As Figure 2 shown, the adaptive brain-computer information fusion classification method based on feature reliability provided by the embodiment of the present invention specifically includes the following steps:

[0111] Step 1. Training stage:

[0112] (1) On the paired brain response and stimulus image data sets, extract the brain response feature set and the stimulus image feature set respectively. The specific steps of the brain response feature set and the stimulus image feature set are as follows:

[0113] Specific steps for brain response feature extraction:

[0114] 1) Load the brain response data set, and average the brain responses captured when the same stimulus image is presented multiple times.

[0115] 2) Select the electrodes placed in the inferior temporal lobe (IT) area, and extract the brain response signals corresponding to the electrodes.

[0116] 3) On the brain response signal of each electrode, average along the time dimension to remove the influence of the time dimension.

[0117] 4) Flip the processed brain response into a 1*168-dimensional feature, which is used as the average brain response feature of the stimulus image on each electrode in the IT area.

[0118] Specific steps for image feature extraction:

[0119] 1) Use the PyTorch deep learning framework to load the ResNet34 model, remove the fully connected layer of the network, and set the model parameter "pretrained=True" to load the ImageNet pre-trained model parameters.

[0120] 2) Load the stimulus image dataset, input the stimulus images into the pre-trained ResNet34 model, and obtain the output features of the convolutional layer. The dimension of the output image features is 512.

[0121] 3) Input the image feature set extracted in 2) into the principal component analysis model, set the model output parameter "n_components = 168", and reduce the dimensionality of the image features to 168 dimensions, which is the same as the brain response features.

[0122] (2) On the extracted brain responses and stimulus image feature sets, train linear SVMs respectively, calculate the classification sensitivity index for each feature as the reliability label of the feature. The specific steps for training the linear SVM and calculating the feature sensitivity index are as follows. For the convenience of description, the calculation steps for the reliability of brain response features are taken as an example here, and the reliability of image features is calculated according to the same steps:

[0123] 1) Randomly divide the extracted brain response feature set into a training set and a test set at a ratio of 1:1.

[0124] 2) Combine the categories of brain responses in pairs, sequentially extract the brain response features corresponding to the two categories in the training set, and input them into the linear SVM to train the binary classifiers for these two categories. Save the model parameters until all binary classifiers for all category combinations are trained.

[0125] 3) For each brain response feature f in the test set, sequentially load the binary classifier model parameters w(i, j) related to its category i, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification result of f for other categories j on the binary classifier w(i, j) The classification is correct as 1 and incorrect as 0. For each binary classifier w(i, j), statistically calculate the decision distance d of the brain response features f of categories i and j in the test set, and take the maximum value as d max , the minimum value as d min , calculate the classification confidence C of f, where C = (d - d min ) / (d max - d min ).

[0126] 4) Exchange the training set and the test set, repeat processes 2) and 3), complete one cross-validation process, and obtain all binary classification results and classification confidence C of all features f in the brain response feature set.

[0127] 5) Calculate the true positive rate of classification for each brain response feature f according to the following formula

[0128]

[0129] Where N represents the total number of categories of features.

[0130] And the average false positive rate FPR of the i-th type of brain response feature i :

[0131]

[0132] Where mean represents taking the mean value.

[0133] 6) Repeat steps 2) - 6) ten times to complete ten two-fold cross-validations, and calculate the average of each brain response feature f And the average false positive rate of the i-th type of brain response feature

[0134] 7) According to the average of the brain response feature f calculated in step 6) And the average false positive rate of the i-th type of brain response feature Calculate the classification sensitivity of f according to the following formula

[0135]

[0136] Where Z(.) represents the inverse of the Gaussian cumulative distribution.

[0137] (3) Using the feature reliability label as the supervision information, train a feature reliability prediction model on each of the brain response and image feature sets respectively. The network structure of the feature reliability prediction model is as Figure 5 shown. The specific training steps of the feature reliability prediction model are as follows. For the convenience of description, here the brain response feature reliability prediction model is taken as an example to illustrate, and the image feature reliability prediction model is trained in the same way:

[0138] 1) Use the PyTorch deep learning framework to build feature reliability prediction models with the same structure respectively. The prediction model adopts a fully connected neural network structure, and the numbers of neurons in the input layer, hidden layer, and output layer are 168, 32, and 1 respectively. The hidden layer uses the ELU activation function, and the output value is the predicted value of the feature reliability.

[0139] 2) Use the reliability value calculated in step 1) 2) as the reliability label of the brain response feature, and divide the brain response feature set into a training set and a test set at a ratio of 4:1. On the training set, use the mean squared error loss function L MSE to supervise the training process of the prediction model:

[0140]

[0141] Among them, y represents the predicted value of feature reliability, y' represents the feature reliability label value, and n represents the batch size. The training of the prediction model uses the mean squared error loss function and the Adam optimizer. The initial learning rate is 1e-3, the learning rate decay is 0.1, and it is trained for 100 epochs in total. The learning rate decays once every 40 epochs.

[0142] (4) Using the feature reliability label as the weight, weighted cascade the brain response features and image features to form a fused feature set. The specific steps of the adaptive fusion classification of the brain response and the image feature set are as follows:

[0143] 1) Use the reliability values calculated in step 1 (2) as the reliability labels of the brain response and image features. In the training set divided in step 1 (3), extract the paired brain response and image features respectively, and obtain the brain response feature reliability of each pair of stimulus images i and the image feature reliability Calculate the fusion weights of the brain response and image features respectively according to the following formulas and

[0144]

[0145]

[0146] Adaptive weighted cascade the fused features according to the fusion weights of the brain response and image features to obtain the fused feature training set F i :

[0147]

[0148] Among them, B i , I i represent the brain response features and image features of the stimulus image i respectively.

[0149] 2) Train a linear SVM classification model according to the fused feature training set F obtained in the above steps i

[0150] (5) Use the fused feature set as the input to train a linear SVM model to achieve classification.

[0151] Step 2, Inference stage:

[0152] (1) On the paired brain response and stimulus image test data sets, extract the brain response feature set and the stimulus image feature set respectively. Among them, the feature extraction process on the test data set is exactly the same as that on the training set.

[0153] ​(2) Input the paired brain response features and image features extracted from the test set into the trained brain response feature reliability prediction model and image feature reliability prediction model in step (3) of step one respectively, and output the reliability values of the brain response and image features respectively.

[0154] (3) According to the feature reliability values output in the previous step, weight and cascade the brain response features and image features in the manner of step one (4) to obtain the fused features.

[0155] (4) Input the fused features obtained in the previous step into the trained linear SVM model in step one (5) to output the classification result.

[0156] According to the inference steps shown in step two, the present invention encapsulates the above method into an adaptive brain-computer information fusion classification system based on feature reliability, as Figure 4 shown, including a feature extraction module, a feature reliability prediction module, an adaptive brain-computer information fusion module, and a fused feature classification module.

[0157] (1) Feature extraction module: Load the paired brain response data and stimulus image data, and extract the brain response features and stimulus image features respectively according to the methods mentioned above.

[0158] (2) Feature reliability evaluation module: Load the brain response reliability evaluation model parameters and image feature reliability evaluation model parameters, input the brain response features and image features extracted in (1) into the corresponding reliability evaluation models respectively, and output their feature reliability values.

[0159] (3) Adaptive brain-computer information fusion module: According to the feature reliability values output in (2), weight and cascade the brain response and image features in the manner of step one (4) to obtain the fused features.

[0160] (4) Fused feature classification module: Load the linear SVM classification model parameters, input the fused features obtained in (3) into the linear SVM model, and output the classification category and classification probability.

[0161] The technical solution of the present invention will be further described below in combination with simulation experiments.

[0162] 1. Experimental conditions:

[0163] The hardware conditions for the experiments of the present invention are as follows: an ordinary computer, an Intel i5 CPU, 8G of memory, and an NVIDIA GeForce GTX 1070 graphics card; software platform: Ubuntu 18.04, the PyTorch deep learning framework, and the Python 3.6 language. The brain response and stimulus image dataset used in the present invention comes from the public data of the Brain-Score platform of the McGovern Institute for Brain Research at the Massachusetts Institute of Technology.

[0164] 2. Training data and test data:

[0165] The dataset used in the present invention includes two parts: stimulus images and brain response data. The stimulus images are synthetic images of 8 types of targets (animals, boats, cars, chairs, faces, fruits, airplanes, tables) and random natural scenes, with a total of 3,200 images, 400 images for each type; each type of target contains 8 sub-types, and the number of images for each sub-type is 50. Each stimulus image contains only one target, and the target images are generated by changing the poses of the 3D models of the target objects, as Figure 6 shown. The brain response data is collected from the ventral stream region of two trained adult rhesus monkeys, and the brain responses of the corresponding brain regions are captured through a 168-channel electrode array in the inferior temporal region (IT). During the EEG acquisition process, every 5 - 10 stimulus images are taken as a group and presented in the center of the display in sequence. Each image is displayed for 100 ms, followed by 100 ms of blank, and the rhesus monkeys are kept staring at the center of the display throughout the process. Each stimulus image is presented multiple times, at least 28 times and on average 50 times. Among them, the publicly available data processing framework of Brain-Score (https: / / brain-score.readthedocs.io / en / latest / index.html) can be used to preprocess the brain responses to obtain the preprocessed brain response features.

[0166] 3. Experimental content:

[0167] According to the above training steps, the present invention trains the models at each processing stage in sequence to completely form an adaptive brain-computer information fusion classification system based on feature reliability.

[0168] According to the above inference steps, the accuracy rate of the adaptive brain-computer information fusion classification based on feature reliability of the present invention is tested on the test set; the classification accuracy rates of single-modal brain responses and image features are also tested on the same test dataset; and the classification accuracy rates of two brain-computer information fusion methods, direct feature concatenation and adaptive feature concatenation, are compared. At the same time, the present invention also compares the influence of different deep learning image feature extraction networks on the classification performance of brain-computer information fusion.

[0169] 4. Analysis of experimental results:

[0170] The present invention uses the absolute error to describe the prediction performance. The average absolute error in IT brain response and image features is 0.4462 (the range of feature reliability values is 0 - 5), and the variance is 0.002, indicating that the special certificate reliability evaluation model proposed by the present invention can stably and accurately predict the reliability of IT brain response and image features. Table 1 shows the comparison results of the classification accuracies of the adaptive brain-computer information fusion features based on feature reliability, the fusion features obtained by directly cascading brain responses and image features, and the brain response (IT) and image unimodal features on the test set. The results show that the classification accuracy of the fusion features obtained by the present invention is much higher than that of the image unimodal features. Among them, the classification accuracy of the fusion features obtained by directly cascading features is on average 4.45% higher than that of the image unimodal features, and the classification accuracy of the fusion features obtained by the present invention is on average 5.92% higher than that of the image unimodal features. The proposed adaptive feature fusion method of the present invention has a 1.47% higher classification accuracy than the fusion features obtained by directly cascading features, indicating that the proposed adaptive brain-computer information fusion classification method based on feature reliability can effectively reduce the risk of negative gain in the brain-computer information fusion process and improve the classification accuracy. Therefore, the present invention has more practical application value and has a wide application prospect under the new paradigm of brain-computer information collaborative work.

[0171] Table 1 Experimental results

[0172]

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state disk (SSD)).

[0174] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be covered by the protection scope of the present invention.

Claims

1. An adaptive brain-computer information fusion classification method, characterized in that, the adaptive brain-computer information fusion classification method includes two stages: training and inference; Among them, the training stage includes: on the paired brain response and stimulus image data sets, extracting the brain response and stimulus image feature sets respectively; for the brain response and stimulus image feature sets, training linear SVMs respectively, calculating the classification sensitivity index of each feature as the feature reliability label; according to the feature reliability label, training the feature reliability prediction models of the brain response and stimulus image respectively; weighting and cascading the brain response features and image features according to the feature reliability label to form a fusion feature set, and training a linear SVM model on the fusion feature set for classification; The inference stage includes: selecting paired brain responses and stimulus images, extracting the corresponding brain response features and stimulus image features respectively; inputting the brain response features and image features into the corresponding feature reliability prediction models respectively to estimate the reliability values of the features; weighting and cascading the brain response features and image features according to the predicted feature reliability values, and inputting the fusion features into the linear SVM model to output the classification result; Using the feature reliability label as the supervision information, training a feature reliability prediction model on each of the brain response and image feature sets, including: (1) Using the PyTorch deep learning framework to build feature reliability prediction models with the same structure respectively; among them, the prediction model adopts a fully connected neural network structure, and the number of neurons in the input layer, hidden layer and output layer are 168, 32 and 1 respectively. The hidden layer adopts the ELU activation function, and the output value is the predicted value of the feature reliability; (2) Use the calculated reliability value as the reliability label of the brain response feature, and divide the brain response feature set into a training set and a test set at a ratio of 4:1; on the training set, use the mean squared error loss function L MSE Supervise the training process of the prediction model: Among them, y represents the predicted value of feature reliability, y′ represents the feature reliability label value, and n represents the batch size; the prediction model is trained using the mean square error loss function and the Adam optimizer, the initial learning rate is 1e-3, the learning rate decay is 0.1, and it is trained for 100 epochs in total, and the learning rate decays once every 40 epochs.

2. The adaptive brain-computer information fusion classification method according to claim 1, characterized in that, the adaptive brain-computer information fusion classification method includes the following steps: Step 1, training stage: (1) On the paired brain response and stimulus image data sets, extracting the brain response feature set and the stimulus image feature set respectively; (2) On the extracted brain response and stimulus image feature sets, training linear SVMs respectively, calculating the classification sensitivity index of each feature as the reliability label of the feature; (3) Using the feature reliability label as the supervision information, training a feature reliability prediction model on each of the brain response and image feature sets; (4) Using the feature reliability label as the weight, weighting and cascading the brain response features and image features to form a fusion feature set; (5) Using the fusion feature set as the input, training a linear SVM model to achieve classification; Step 2, inference stage: (1) On the paired brain response and stimulus image test data sets, extracting the brain response feature set and the stimulus image feature set respectively; (2) Input the extracted brain response features and image features into the corresponding feature reliability prediction models respectively to estimate the reliability of the brain response features and image features; (3) According to the predicted feature reliability values, weight and cascade the brain response features and image features to obtain fused features; (4) Input the fused features into a linear SVM model to output the classification results of the fused features.

3. The adaptive brain-computer information fusion classification method according to claim 2, wherein, in step one, in step (1), on the paired brain response and stimulus image data sets, extract the brain response feature set and the stimulus image feature set respectively, including: (1) Extraction of the brain response feature set: 1) Load the brain response data set and average the brain responses captured when the same stimulus image is presented multiple times; 2) Select the electrodes placed in the inferior temporal lobe region IT and extract the brain response signals corresponding to the electrodes; 3) On the brain response signal of each electrode, calculate the mean along the time dimension to remove the influence of the time dimension; 4) Flip the processed brain response into a 1*168-dimensional feature as the average brain response feature of the stimulus image on each electrode in the IT region; (2) Extraction of the image feature set: 1) Use the PyTorch deep learning framework to load the ResNet34 model, remove the fully connected layer of the network, and set the model parameter "pretrained=True" to load the ImageNet pre-trained model parameters; 2) Load the stimulus image data set, input the stimulus image into the pre-trained ResNet34 model to obtain the output features of the convolutional layer, and the dimension of the output image features is 512; 3) Input the image feature set extracted in step 2) into the principal component analysis model, set the model output parameter "n_components=168", and reduce the dimension of the image features to the same 168 dimensions as the brain response features.

4. The adaptive brain-computer information fusion classification method according to claim 2, wherein, in step one, in step (2), on the extracted brain response and stimulus image feature sets, train a linear SVM and calculate the feature sensitivity index respectively, including: (1) Randomly divide the extracted brain response feature set into a training set and a test set at a ratio of 1:1; (2) Combine the categories of the brain responses in pairs, sequentially extract the brain response features of the corresponding two categories in the training set, and input them into the linear SVM to train the binary classifiers of these two categories, and save the model parameters until all category combinations of the binary classifiers are trained; (3) For each brain response feature f in the test set, sequentially load the binary classifier model parameters w(i, j) related to its category i, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification results of f on the binary classifier w(i, j) for other categories j The classification is correct as 1 and incorrect as 0; for each binary classifier w(i, j), statistically calculate the decision distance d of the brain response features f of the two categories i and j in the test set, and take the maximum value as d max , and the minimum value as d min , calculate the classification confidence C of f, where C = (d - d min ) / (d max - d min ); (4) Swap the training set and the test set, repeat steps (2) and (3) to complete one process of cross-validation, and obtain all binary classification results of all features f in the brain response feature set and the classification confidence C; (5) Calculate the classification true positive rate of each brain response feature f according to the following formula where N represents the total number of categories of the features; Calculate the average false positive rate FPR of the i-th type of brain response feature i : where mean represents calculating the mean; (6) Repeat steps (2) to (6) 10 times to complete ten-fold two-way cross-validation, and calculate the average of each brain response feature f and the average false positive rate of the i-th type of brain response feature (7) Based on the average of the brain response feature f calculated in step (6) and the average false positive rate of the brain response feature of the i-th class calculate the classification sensitivity of f according to the following formula where Z(.) represents the inverse of the Gaussian cumulative distribution.

5. The adaptive brain-computer information fusion classification method according to claim 2, wherein, in step one, in step (4), use the feature reliability label as the weight, weight and cascade the brain response features and image features to form a fused feature set, including: (1) Use the reliability value calculated in Step 1(2) as the reliability label for the brain response and image features, and separately extract the paired brain response and image features from the training set divided in Step 1(3) to obtain the reliability of the brain response features for each pair of stimulus images i and the reliability of the image features Respectively obtain the fusion weights of the brain response and image features according to the following formula and Adaptive weighted cascade fusion features according to the fusion weights of brain responses and image features, and obtain the fusion feature training set F i : Among them, B i , I i respectively represent the brain response characteristics and image characteristics of the stimulus image i; (2) Train a linear SVM classification model using the fused feature training set F obtained in step (1). i Train a linear SVM classification model.

6. The adaptive brain-computer information fusion classification method according to claim 2, wherein, in step two, the adaptive fusion classification of the brain response and the image feature set in the inference stage includes: (1) Load the stimulus images in the test set, and extract brain response features and image features simultaneously; (2) Input the paired brain response features and image features extracted from the test set into the trained brain response feature reliability prediction model and image feature reliability prediction model in step (3) of step one respectively, and output the reliability values of the brain response and image features respectively; (3) According to the feature reliability values output in step (2), cascade the brain response features and image features by weighting in the manner in step (4) of step one to obtain fused features; (4) Input the fused features obtained in step (3) into the trained linear SVM model in step (5) of step one, and output the classification result.

7. An adaptive brain-computer information fusion classification system for implementing the adaptive brain-computer information fusion classification method according to any one of claims 1 to 6, characterized in that, the adaptive brain-computer information fusion classification system includes a feature extraction module, a prediction model construction module, a feature reliability prediction module, an adaptive brain-computer information fusion module, and a fused feature classification module; The feature extraction module is used to load paired brain response data and stimulus image data, and extract brain response features and stimulus image features respectively according to the above method; The prediction model construction module is used to train a linear SVM for the brain response and stimulus image feature sets respectively, calculate the classification sensitivity index of each feature as the feature reliability label; Using the feature reliability label, establish the feature reliability prediction models of the brain response and stimulus image respectively; The feature reliability evaluation module is used to load the brain response feature reliability prediction model parameters and image feature reliability prediction model parameters, input the extracted brain response features and image features into the corresponding reliability prediction models respectively, and output their feature reliability values; The adaptive brain-computer information fusion module is used to cascade the brain response and image features by weighting according to the output feature reliability values to obtain fused features; The fused feature classification module is used to load the linear SVM classification model parameters, input the obtained fused features into the linear SVM model, and output the classification category and classification probability.

8. A computer device, characterized in that, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the adaptive brain-computer information fusion classification method according to any one of claims 1 to 6.

9. An information data processing terminal, characterized in that, the information data processing terminal is used to implement the adaptive brain-computer information fusion classification system according to claim 7.

Images