A brain-computer information fusion classification method and system for brain-out-of-loop applications

Abstract

The present invention belongs to the technical field of brain-computer interface technology application, and discloses a brain-computer information fusion classification method and system for brain out-of-loop applications. The brain-computer information fusion classification system for brain out-of-loop applications consists of a data loading device, an image preprocessing device, a brain response generation device, an image feature reliability prediction device and a brain-computer feature fusion classification device. Compared with the prior art, the brain response generation model of the present invention utilizes deep learning technology to construct a prediction model of image features to brain response features in a feature domain. The overall structure of the prediction model is simple. Compared with the method of sharing representation space, it does not require too many paired brain response and stimulus image data sets to be successfully trained. This brain response generation model can be effectively integrated into the existing brain-computer information fusion classification model to realize the reasoning application of brain out-of-loop, and has high practical significance.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer interface technology application, and in particular, relates to a brain-computer information fusion classification method, system, and medium for brain-out-of-loop application. Background Art

[0002] At present, the rapid development of deep learning technology has provided strong technical support for automatic target recognition and detection, and has achieved performance that exceeds that of humans in fields such as image classification and target detection. However, the current deep learning technology does not fully simulate the human visual recognition process and is heavily dependent on data-driven. Image recognition in complex open environments (such as complex scene changes and complex ground environments) is still difficult. Manual interpretation based on visual experts is still the main image recognition and analysis method, but the interpretation behavior of experts is easily affected by environmental factors such as fatigue and injury, and it is difficult to meet the high-intensity, real-time, and long-term work requirements like machines. The brain-machine hybrid intelligent computing method based on brain-computer interface technology constructs a "brain-in-the-loop" information processing system to coordinately process brain responses and image representation based on machine intelligence, realize information interaction between biological intelligence and machine intelligence, and integrate their respective strengths, providing a new computing paradigm for target recognition in complex open environments. However, due to the limitations of human behavior, the "brain-in-the-loop" information processing system is difficult to meet the high-intensity, real-time, and automated image recognition work of machine intelligence. Through the brain responses of visual experts, a brain-computer information fusion model of "brain in the loop" is constructed, and the brain-computer information hybrid intelligent computing mode of "brain out of the loop" application can remove visual experts from the system when applied, realizing the complementary advantages of biological intelligence and machine intelligence.

[0003] "Brain in the loop" modeling refers to the participation of visual experts' brain responses in the model training stage, and the use of visual experts' brain responses to supervise the learning of machine learning models. "Brain out of the loop" application means that although there is no participation of visual experts' brain responses in the process of model application reasoning, the brain response supervision process in the model training stage stores the prior information of visual experts' brain responses to a certain extent, which can simulate the visual processing process of experts. Even if there is no participation of brain responses in the application, supervised reasoning similar to "brain in the loop" can be achieved. At present, there are two main technologies used in the industry for "brain in the loop" modeling and "brain out of the loop" applications. Existing technology 1: transfer learning; existing technology 2: shared subspace learning. Existing technology 1: transfer learning, based on the method of transfer learning, a prediction model from image features to brain response space is established, hoping to use the rich high-level semantic features of the brain response space to guide the classification of image features with medium and low-level semantics. It is widely used in the traditional field of computer vision. Generally, traditional methods such as PCA-CCA, kernel alignment, and Gaussian process regression are used to establish a prediction model from image features to brain response features. However, with the development of deep learning technology, the performance of image classification has reached the level of humans. Compared with traditional computer vision methods, existing deep learning technology can already obtain high-level semantic information. It only establishes a prediction model from image features to brain responses. The performance of using predicted brain responses to classify "brain out of loop" is far from the performance of existing deep learning methods. Therefore, the "brain out of loop" application method of the existing technology one has gradually declined. Existing technology two: shared subspace learning. In recent years, with the development of deep learning technology, multimodal learning methods based on shared subspaces have also made progress. This method requires a large amount of paired data. Through loss function constraints, multiple modal representations of the same data have the minimum distance in the shared subspace to establish a shared representation space supervised by other modal data. When the model is applied, reasoning can be completed in the absence of modalities. The shared subspace method can realize the supervised learning of brain responses to image features. When the model is reasoning, "brain out of loop, brain response supervision" can be achieved. However, given the cost and difficulty of obtaining brain response data, it is difficult to obtain sufficient brain response data, and the existing public data sets are difficult to support the training of such models. Therefore, there is a need for a brain-computer information fusion classification method that can realize “brain-out-of-the-loop” applications on a small amount of data sets.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) The idea in the existing technology is limited by the development of deep learning technology and is difficult to be directly applied to the field of deep learning.

[0006] (2) The second existing technology is limited by the acquisition of brain response data, and it is difficult to use limited brain response data to establish an effective shared representation space model.

[0007] The difficulty of solving the above problems and defects is: how to propose a brain-computer information fusion classification scheme for "brain out of the loop" applications with only a small amount of brain response and stimulus image pairing data sets, which can meet the needs of "brain in the loop" to establish a brain-computer information fusion model and "brain out of the loop" applications.

[0008] The significance of solving the above problems and defects is: The significance of the present invention lies in proposing a prediction model of image to brain response based on deep learning technology, which can be integrated into the brain-computer information fusion classification model, and use image features to generate missing brain response features when applied, thereby realizing the application requirement of "brain out of the loop". Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a brain-computer information fusion classification method, system and medium for brain-out-of-loop application.

[0010] The present invention is implemented as follows: a brain-computer information fusion classification method for brain-out-of-loop applications, the brain-computer information fusion classification method for brain-out-of-loop applications includes a training phase and a reasoning phase, wherein the training phase uses training data to learn the optimization of various model parameters, and the reasoning phase is a systematic application method:

[0011] Training phase:

[0012] Step 1: On the training data set, use a data loading device to load paired brain response and stimulus image data, send them to a preprocessing device, preprocess the brain response, extract image features from the stimulus image, and form a brain response and image feature set;

[0013] Step 2: training a prediction model from image features to brain responses on the paired feature set, and training feature reliability prediction models on the brain response and image feature sets respectively;

[0014] Step 3, using the feature reliability of brain response and stimulus image to construct a weighted cascade fusion feature set, and training a linear SVM classification model on the fusion feature set;

[0015] Reasoning phase:

[0016] Step 1: On the test data set, only the stimulus image data is loaded by using a data loading device, and sent to a preprocessing device to extract image features;

[0017] Step 2: Input the extracted image features into the brain response generation device, load the model parameters, perform forward reasoning, and reconstruct the brain response corresponding to the stimulus image;

[0018] Step 3, inputting the reconstructed brain response and image features into the feature reliability prediction device, loading the model parameters, performing forward reasoning, and outputting the reliability values ​​of the brain response and image features respectively;

[0019] In step 4, the reconstructed brain response, the extracted image features, and the feature reliability values ​​obtained in step 6 are sent to the brain-computer information fusion classification device, the brain response and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

[0020] Furthermore, the data loading device simultaneously loads the paired brain response and stimulation image data, and sends them to the brain response and stimulation image preprocessing devices respectively, extracts the brain response preprocessing features and image features respectively, and constructs paired brain response and stimulation image feature sets;

[0021] The data loading device is inherited from the DataSet class and DataLoader class in the PyTorch deep learning framework. In the training phase, paired training data are read out from paired brain response and stimulus image data sets in turn, and then processed separately.

[0022] The preprocessing device is divided into two parts, which respectively preprocesses the brain response and extracts the stimulus image features using the ResNet34 pre-trained model to construct a brain response and image feature set.

[0023] Further, the brain response is preprocessed as follows:

[0024] From the original brain response data, along the stimulus presentation dimension, the brain responses collected by multiple presentations of the same stimulus image are obtained, and the brain responses belonging to the same stimulus image are averaged;

[0025] Select brain response data from electrodes placed in the inferior temporal region of the ventral visual pathway and extract the corresponding brain responses;

[0026] For the extracted brain response data, the brain response data of 70-170ms were averaged along the time dimension to obtain the spatial response data of each electrode point;

[0027] The acquired brain response data is standardized and flipped to obtain the preprocessed 1*168-dimensional brain response data to construct a brain response feature set.

[0028] Furthermore, the feature extraction and preprocessing process of the stimulus image preprocessing device is as follows:

[0029] Use the PyTorch deep learning framework to build a ResNet34 model, remove its last fully connected layer, load the ImageNet pre-trained model parameters in the torchvison model library, and set the model parameter "pretrained = True";

[0030] The stimulus image was loaded using a data loading device and input into the ResNet34 model for forward inference, the output features of the convolutional layer were obtained as 512 dimensions, and an image feature set was constructed;

[0031] In order to eliminate the dimensionality difference, the acquired image feature set is reduced to 128 dimensions using principal component analysis to construct the image feature set.

[0032] Furthermore, the prediction model of image features to brain responses is trained on the paired feature set, and the feature reliability prediction model is trained on the brain response and image feature set respectively. The specific process is as follows:

[0033] On the paired brain response and stimulus image feature sets, a feature domain prediction model from image features to brain response is constructed, the brain response feature set is used as the true value, the paired image feature set is used to train the brain response generation model, and the successfully trained prediction model parameters are stored in the brain response prediction device;

[0034] The specific steps for building and training the brain response generation model are as follows:

[0035] Construct a brain response denoising autoencoder structure, which uses a fully connected neural network structure. The encoding module consists of three hidden layers, with the number of neurons being 168, 96, and 32 respectively. The decoding module also consists of three hidden layers, with the number of neurons being 32, 96, and 168. The hidden layer uses the ELU activation function, and a batch normalization layer is added between every two layers. The output of the decoding module is consistent with the dimension of the brain response and is used to reconstruct the brain response;

[0036] Load the brain response feature set, add a random Gaussian noise to each input brain response, the random Gaussian noise is generated by Python's numpy toolkit, with a mean of 0 and a variance of 0.2; output the noisy brain response to the denoising autoencoder structure, and use the mean square error loss function to supervise the loss between the reconstructed brain response and the real brain response:

[0037]

[0038] Where y represents the brain response reconstructed by the decoding module, y' represents the real brain response, and n represents the batch size. The Adam optimizer is used, the initial learning rate is 1e-3, the learning rate decay is 0.1, and a total of 50 epochs are trained, with the learning rate decayed every 20 epochs;

[0039] A feature domain mapping model from image features to brain responses was constructed, using a fully connected neural network structure with only a feature encoding structure consisting of three hidden layers with 168, 96, and 32 neurons, respectively. The decoding structure shared parameters with the decoding module of the denoising autoencoder model of the brain response, and was used to construct an information flow pipeline from the image feature domain to the reconstruction of the brain response.

[0040] Load the parameters of the successfully trained autoencoder model, set the encoder parameters of the autoencoder model to be fixed, and the decoder parameters to be learnable; load the paired brain responses and image features and input them into the encoder of the denoising autoencoder model and the constructed image feature encoder, respectively, to obtain the feature domain features f b and f i , and the decoder output of the denoised autoencoder model simultaneously produces the reconstructed brain response r from two sources b and r i , and the real brain response is recorded as g; when training the model, the main idea is to make the image features and brain responses show similarity in the feature domain after the encoder nonlinear mapping, and the decoder decoding and reconstruction should also be consistent with the real brain response, and use the absolute error to supervise the training process of the feature domain and the reconstruction result:

[0041] L total =L f +L b +L i

[0042] L f =L MAE (f b , f i )

[0043] L b =L MAE (r b , g)

[0044] L i =L MAE (r i , g);

[0045] The Adam optimizer was used to optimize the model. The initial learning rate was 1e-3, the learning rate decay was 0.1, and a total of 50 epochs were trained. The learning rate decayed every 20 epochs.

[0046] Load the successfully trained image encoder model parameters and the decoder parameters of the denoising autoencoder model, build a complete information flow pipeline from image features to brain response reconstruction, and save the model parameters.

[0047] Furthermore, the feature reliability of the brain response and the stimulus image is used to construct a weighted cascade fusion feature set, and a linear SVM classification model is trained on the fusion feature set. The specific process is as follows:

[0048] The linear SVM models are trained on the brain response and image feature sets respectively, and the classification sensitivity values ​​of the brain response and image features are calculated respectively as the reliability labels of the brain response and image features;

[0049] The calculation steps of the classification sensitivity value of the feature are as follows,

[0050] The image feature set is randomly divided into a training set and a test set, with each set accounting for half of the data. Based on the categories in the training set, image features of corresponding categories are selected in pairs to train a binary classification linear SVM model. All possible binary classifier combinations are trained in turn, and the model parameters are retained.

[0051] For any test set image feature f, load the binary classifier model parameters w(i, j) related to its category i in turn, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification results of f on the binary classifier w(i, j) for other categories j The correct classification is 1, and the wrong classification is 0. For each binary classifier w(i, j), the decision distance d between the two types of brain response features f in the test set i and j is statistically analyzed, and the maximum value is d max , the minimum value is d min , calculate the classification confidence C of f, where C = (dd min ) / (d max -d min );

[0052] By exchanging the training dataset and the test dataset and repeating the above process, a two-fold cross validation can be completed to obtain the results of all samples in the entire image feature dataset. and its classification confidence C f ;

[0053] According to all the samples obtained and its classification confidence C f , calculate the true positive rate of each sample f in the entire sample set in multi-classification (category N) And the average false positive rate FPR of each category i , where the calculation formula for the true positive rate of sample f is:

[0054]

[0055] The average false positive rate of the i-th category is FPR i :

[0056]

[0057] Repeat the above process 10 times, complete 10 two-fold cross validations, and calculate the average true positive rate of sample f in 10 repeated tests and the average false positive rate for each category i

[0058] According to the average true positive rate of the image feature f calculated and the average false positive rate of category i Calculate the classification sensitivity index of image feature f

[0059]

[0060] Where Z(.) represents the inverse of Gaussian cumulative distribution; the classification sensitivity has reflected the separability of feature f in the representation space. As a reliability evaluation indicator of its feature, the reliability indicator obtained here is the statistical characteristic of the reliability of the entire feature set, which serves as the label data of the subsequent prediction model.

[0061] Furthermore, the feature reliability prediction device constructs the feature reliability prediction models of the brain response and the stimulus image respectively in the following specific processes:

[0062] The obtained feature reliability values ​​are used as labels, and training is performed on the brain response and image feature sets respectively, and the trained two feature reliability prediction model parameters are stored in the feature reliability prediction device; wherein the steps of constructing and training the feature reliability prediction model are as follows:

[0063] An image feature reliability prediction model was constructed. The model adopted a fully connected neural network structure and consisted of three layers: input layer, hidden layer, and output layer. The number of neurons was 168, 32, and 1 respectively. The activation function adopted ELU, and batch normalization was added between the hidden layer and the output layer. The output value of the network can be considered as the feature reliability value predicted by the network.

[0064] The calculated reliability value is used as the reliability label of the image feature, and the mean square error loss function L is used. MSE Supervised prediction model training process:

[0065]

[0066] Where y represents the feature reliability prediction value, y′ represents the feature reliability label value, and n represents the batch size. The prediction model training uses the mean square error loss function and Adam optimizer, with an initial learning rate of 1e-3 and a learning rate decay of 0.1. A total of 100 epochs are trained, and the learning rate decays every 40 epochs.

[0067] The feature reliability prediction models of image features and brain responses are trained separately, and the respective model parameters are saved after successful training;

[0068] The brain response and image features are weighted by the calculated feature reliability labels to construct a brain-computer information fusion feature set, and a linear SVM classification model is trained on the fusion feature set. The successfully trained model parameters are stored in the brain-computer feature fusion classification device.

[0069] The specific steps of feature weighting method are as follows:

[0070] Load paired data B from the brain response and image feature set i and I i , and the calculated reliability values ​​are used as reliability labels for brain responses and image features and Calculate the respective weighting coefficients according to the reliability values ​​of the two and

[0071]

[0072]

[0073] Obtain the fusion feature F according to the weighted coefficients of the two i , forming a fusion feature set:

[0074]

[0075] A linear SVM classification model is trained on the obtained fusion feature set as the classification output of the fusion feature set, and the parameters of the successfully trained SVM model are saved.

[0076] Furthermore, the specific process of the forward reasoning is as follows:

[0077] Only the stimulus image data is loaded by the data loading device and sent to the preprocessing device to extract the image features and reduce the dimension to 168 dimensions;

[0078] The extracted image features are sent to a brain response prediction device, the brain response prediction model parameters are loaded, and forward reasoning is performed to obtain the reconstructed brain response;

[0079] The extracted image features and the obtained reconstructed brain responses are sent to the feature reliability prediction device in pairs, the brain response feature reliability model parameters and the image feature reliability prediction model parameters are loaded respectively, and forward reasoning is performed to obtain the feature reliability prediction values ​​of the two respectively;

[0080] The acquired image features, the corresponding reconstructed brain responses and the acquired feature reliability values ​​are sent to the brain-computer information fusion classification device, the brain responses and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

[0081] Another object of the present invention is to provide a brain-computer information fusion classification system for brain-out-of-the-loop applications that implements the brain-computer information fusion classification method for brain-out-of-the-loop applications, the brain-computer information fusion classification system for brain-out-of-the-loop applications comprising: a data loading device, an image preprocessing device, a brain response generating device, an image feature reliability prediction device, and a brain-computer feature fusion classification device;

[0082] A data loading device, connected to the input image and the input brain response data, for loading the stimulus image and the original brain response to be input into the system;

[0083] An image preprocessing device, connected to the data loading device, for extracting image features of the stimulus image and preprocessing the brain response;

[0084] A brain response generating device is connected to the preprocessing device and establishes a feature domain reconstruction model from image features to brain responses, so as to reconstruct brain responses using image features;

[0085] The image feature reliability prediction device is connected to the preprocessing device and is composed of a brain response reliability prediction and an image feature reliability prediction, and predicts the reliability of the input brain response and image feature respectively;

[0086] The brain-computer feature fusion classification device is respectively connected to the preprocessing device, the brain response generation device and the feature reliability prediction device, and uses the predicted value of feature reliability to weight the cascade reconstructed brain response and image features to form a fusion feature, and classifies the fusion feature to output the result.

[0087] Another object of the present invention is to provide a storage medium for receiving a user input program, wherein the stored computer program enables an electronic device to execute the brain-computer information fusion classification method for brain-out-of-loop application, comprising the following steps:

[0088] Step 1: On the training data set, use a data loading device to load paired brain response and stimulus image data, send them to a preprocessing device, preprocess the brain response, extract image features from the stimulus image, and form a brain response and image feature set;

[0089] Step 2: training a prediction model from image features to brain responses on the paired feature set, and training feature reliability prediction models on the brain response and image feature sets respectively;

[0090] Step 3, using the feature reliability of brain response and stimulus image to construct a weighted cascade fusion feature set, and training a linear SVM classification model on the fusion feature set;

[0091] Step 4: On the test data set, only the stimulus image data is loaded by using a data loading device, and sent to a preprocessing device to extract image features;

[0092] Step 5, input the extracted image features into the brain response generation device, load the model parameters, perform forward reasoning, and reconstruct the brain response corresponding to the stimulus image;

[0093] Step 6, inputting the reconstructed brain response and image features into the feature reliability prediction device, loading the model parameters, performing forward reasoning, and outputting the reliability values ​​of the brain response and image features respectively;

[0094] In step seven, the reconstructed brain response, the extracted image features, and the feature reliability values ​​obtained in step six are sent to the brain-computer information fusion classification device, the brain response and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

[0095] Combined with all the above technical solutions, the advantages and positive effects of the present invention are as follows: the brain-computer information fusion classification method, system, and medium for brain-out-of-loop applications proposed in the present invention use a feature domain mapping method to establish a generative model of image features to brain responses, which can effectively solve the problem of missing brain responses during reasoning applications. Compared with the prior art, the brain response generation model of the present invention uses deep learning technology to construct a nonlinear association model of brain responses in the image feature domain in the feature domain, and optimizes the decoder in advance through a denoising autoencoder structure. The overall structure of the prediction model is simple. Compared with the shared subspace method, it does not require too many paired brain response and stimulus image data sets to be successfully trained. This brain response prediction model can be effectively integrated into the existing brain-computer information fusion classification model to realize the reasoning application of brain-out-of-loop, which has high practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1It is a flow chart of the brain-computer information fusion classification method for brain-out-of-loop applications provided by an embodiment of the present invention.

[0097] Figure 2 It is a structural schematic diagram of the brain-computer information fusion classification system for brain-out-of-loop applications provided by an embodiment of the present invention.

[0098] Figure 3 It is a schematic diagram of the data processing process in the training phase and the reasoning phase provided by an embodiment of the present invention.

[0099] Figure 4 is a schematic diagram of a brain response generating device provided in an embodiment of the present invention.

[0100] Figure 5 Schematic diagram of the network structure of the feature reliability prediction module provided by the embodiment of the present invention.

[0101] Figure 6 Schematic diagram of some stimulus images for classification tasks provided by an embodiment of the present invention.

[0102] Figure 7 It is a schematic diagram of real brain response and reconstructed brain response provided by an embodiment of the present invention.

[0103] In the figure: 1. Data loading device; 2. Image preprocessing device; 3. Brain response generating device; 4. Image feature reliability prediction device; 5. Brain-computer feature fusion classification device; 6. Brain response preprocessing device; 7. Brain response reliability prediction device. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0105] In view of the problems existing in the prior art, the present invention provides a brain-computer information fusion classification method, system and medium for brain-out-of-loop application. The present invention is described in detail below with reference to the accompanying drawings.

[0106] The brain-computer information fusion classification method for brain-out-of-loop application provided by the present invention can also be implemented by other steps by ordinary technicians in the industry. Figure 1 The brain-computer information fusion classification method for out-of-loop applications provided by the present invention is only a specific embodiment.

[0107] Most of the existing brain-computer information fusion classification technologies are based on the construction of hybrid information processing systems based on the "brain in the loop", but the "brain in the loop" is easily affected by human factors such as pain and fatigue of visual experts, and it is difficult to achieve true brain-computer complementary advantages. At this stage, the technology that can be used for "brain out of the loop" modeling is limited by the quality of brain response data and the difficulty of obtaining massive paired brain response and stimulus image data. It is difficult to use deep learning technology to train large-scale shared subspace learning models, so it is difficult to apply to the reasoning application of "brain out of the loop". The present invention proposes a mapping model from the image feature domain to the brain response feature domain, which can reconstruct the brain response according to the image features when applied, and realize the "brain out of the loop" application. Because the overall model parameter amount is small, it does not require a large amount of paired data to complete the training, and can be effectively integrated into the existing adaptive brain-computer information fusion classification system.

[0108] like Figure 1 As shown, the brain-computer information fusion classification method for brain-out-of-loop applications provided by an embodiment of the present invention includes:

[0109] S101: On the training data set, a data loading device is used to load paired brain response and stimulus image data, and the data is sent to a preprocessing device to preprocess the brain response and extract image features from the stimulus image to form a brain response and image feature set.

[0110] S102: training a prediction model from image features to brain responses on a paired feature set, and training feature reliability prediction models on the brain response and image feature sets, respectively.

[0111] S103: Construct a weighted cascade fusion feature set using the feature reliability of brain response and stimulus image, and train a linear SVM classification model on the fusion feature set.

[0112] S104: On the test data set, only the stimulus image data is loaded by a data loading device, and sent to a pre-processing device to extract image features.

[0113] S105: Input the extracted image features into the brain response generation device, load the model parameters, perform forward reasoning, and reconstruct the brain response corresponding to the stimulus image.

[0114] S106: Input the reconstructed brain response and image features into the feature reliability prediction device respectively, load the model parameters, perform forward reasoning, and output the reliability values ​​of the brain response and image features respectively.

[0115] S107: The reconstructed brain response, the extracted image features, and the feature reliability values ​​obtained in S106 are sent to the brain-computer information fusion classification device, the brain response and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

[0116] like Figure 2 As shown, the brain-computer information fusion classification system for "brain out of loop" application provided by the embodiment of the present invention includes a data loading device 1, an image preprocessing device 2, a brain response generating device 3, an image feature reliability prediction device 4, a brain-computer feature fusion classification device 5, a brain response preprocessing device 6 and a brain response reliability prediction device 7.

[0117] The data loading device 1 is connected to the input image and the input brain response data, and is used to load the stimulation image and the original brain response that need to be input into the system.

[0118] The image preprocessing device 2 is connected to the data loading device and is used to extract image features of the stimulus image for preprocessing.

[0119] The brain response preprocessing device 3 is connected to the data loading device and preprocesses the loaded brain responses.

[0120] The brain response generating device 4 is connected to the preprocessing device, and establishes a feature domain reconstruction model from image features to brain responses, so as to reconstruct brain responses using image features.

[0121] The image feature reliability prediction device 5 is connected to the preprocessing device and predicts the reliability of the input image features.

[0122] The brain response reliability prediction device 6 is connected to the preprocessing device and predicts the reliability of the input brain response characteristics.

[0123] The brain-computer feature fusion classification device 7 is respectively connected to the preprocessing device, the brain response generation device and the feature reliability prediction device, and uses the predicted value of feature reliability to weight the cascade reconstructed brain response and image features to form a fusion feature, and classifies the fusion feature to output the result.

[0124] like Figure 3 As shown, the brain-computer information fusion classification system for the "brain out of the loop" application provided by the embodiment of the present invention is divided into two parts: the training stage and the reasoning stage. The training stage learns the optimal model parameters and saves them, and loads each model parameter into the corresponding device in the system. In the reasoning stage, the brain-computer information fusion classification of the "brain out of the loop" application is realized. There are two steps to build the system:

[0125] Step 1: Training phase:

[0126] (1) The data loading device simultaneously loads paired brain response and stimulus image data and sends them to the brain response and stimulus image preprocessing devices respectively, extracts brain response preprocessing features and image features respectively, and constructs paired brain response and stimulus image feature sets.

[0127] The data loading device inherits from the DataSet class and DataLoader class in the PyTorch deep learning framework. During the training phase, paired training data are read out from paired brain response and stimulus image data sets in turn for subsequent processing.

[0128] The preprocessing device is divided into two parts, which respectively preprocess the brain response and extract the stimulus image features using the ResNet34 pre-training model to construct the brain response and image feature set. The specific steps are as follows:

[0129] Preprocessing steps of the brain response preprocessing device:

[0130] 1) From the original brain response data, along the stimulus presentation dimension, obtain the brain responses collected by multiple presentations of the same stimulus image, and average the brain responses belonging to the same stimulus image.

[0131] 2) Select brain response data of electrodes placed in the inferior temporal lobe region of the ventral visual pathway and extract the corresponding brain responses.

[0132] 3) For the brain response data extracted in 2), the brain response data of 70-170ms is averaged along the time dimension to obtain the spatial response data of each electrode point.

[0133] 4) The acquired brain response data is standardized and flipped to obtain the pre-processed 1*168-dimensional brain response data to construct a brain response feature set.

[0134] Feature extraction and preprocessing steps of stimulus image preprocessing device:

[0135] A. Use the PyTorch deep learning framework to build a ResNet34 model, remove its last fully connected layer, load the ImageNet pre-trained model parameters in the torchvison model library, and set the model parameter "pretrained = True".

[0136] B. Use the data loading device to load the stimulus image, input it into the ResNet34 model for forward reasoning, obtain the output features of the convolution layer as 512 dimensions, and construct the image feature set.

[0137] C. In order to eliminate the dimensionality difference, the image feature set obtained in B is reduced to 128 dimensions using principal component analysis to construct the image feature set.

[0138] (2) Based on the paired brain response and stimulus image feature sets, a feature domain prediction model from image features to brain responses is constructed. The brain response feature set is used as the true value, and the paired image feature set is used to train the brain response generation model. The parameters of the successfully trained prediction model are stored in the brain response prediction device.

[0139] The construction of specific brain response generation models (such as Figure 4 The training steps are as follows:

[0140] The first step is to construct a brain response denoising autoencoder structure, which uses a fully connected neural network structure. The encoding module consists of three hidden layers, with the number of neurons being 168, 96, and 32 respectively. The decoding module also consists of three hidden layers, with the number of neurons being 32, 96, and 168. The hidden layer uses the ELU activation function, and a batch normalization layer is added between every two layers. The output of the decoding module is consistent with the dimension of the brain response and is used to reconstruct the brain response. The specific network diagram is shown below. Figure 4 shown.

[0141] The second step is to load the brain response feature set and add a random Gaussian noise to each input brain response. The random Gaussian noise is generated by Python's numpy toolkit with a mean of 0 and a variance of 0.2. The brain response after the noise processing is output to the denoising autoencoder structure, and the mean square error loss function is used to supervise the loss between the reconstructed brain response and the real brain response:

[0142]

[0143] Where y represents the brain response reconstructed by the decoding module, y′ represents the real brain response, and n represents the batch size. The Adam optimizer is used with an initial learning rate of 1e-3 and a learning rate decay of 0.1. A total of 50 epochs are trained, and the learning rate decays every 20 epochs.

[0144] The third step is to build a feature domain mapping model from image features to brain responses. A fully connected neural network structure is used, with only a feature encoding structure consisting of three hidden layers. The number of neurons is 168, 96, and 32 respectively. The decoding structure shares parameters with the decoding module of the denoising autoencoder model of brain responses in the first step, and is used to build an information flow pipeline from the image feature domain to the reconstruction of brain responses.

[0145] The fourth step is to load the parameters of the autoencoder model successfully trained in the second step, set the encoder parameters of the autoencoder model to be fixed, and the decoder parameters to be learnable. Load the paired brain responses and image features and input them into the encoder of the denoising autoencoder model and the image feature encoder constructed in the third step, respectively, to obtain the feature domain features f b and f i , and the decoder output of the denoised autoencoder model simultaneously produces the reconstructed brain response r from two sources b and r i , and the real brain response is recorded as g. When training the model, the main idea is to make the image features and brain responses show similarity in the feature domain after the encoder nonlinear mapping, and the decoder decoding and reconstruction should also be consistent with the real brain response. The absolute error is used to supervise the training process of the feature domain and the reconstruction results:

[0146] L total =L f +L b +L i

[0147] L f =L MAE (f b , f i )

[0148] L b =L MAE (r b , g)

[0149] L i =L MAE (r i , g);

[0150] The Adam optimizer was used to optimize the model. The initial learning rate was 1e-3, the learning rate decay was 0.1, and a total of 50 epochs were trained. The learning rate decayed every 20 epochs.

[0151] The fifth step is to load the image encoder model parameters successfully trained in the fourth step and the decoder parameters of the denoising autoencoder model, build a complete information flow pipeline from image features to brain response reconstruction, and save the model parameters.

[0152] (3) Linear SVM models are trained on the brain response and image feature sets respectively, and the classification sensitivity values ​​of the brain response and image features are calculated respectively as the reliability labels of the brain response and image features. The calculation steps of the classification sensitivity value of the feature are as follows. Here, the reliability calculation of the image feature is taken as an example. The calculation method of the reliability of the brain response feature is consistent:

[0153] Step a: randomly divide the image feature set into a training set and a test set, with each set accounting for half of the data. Based on the categories in the training set, image features of corresponding categories are selected in pairs to train a binary classification linear SVM model. All possible combinations of binary classifiers are trained in turn, and the model parameters are retained.

[0154] Step b: For any test set image feature f, load the binary classifier model parameters w(i, j) related to its category i in turn, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification results of f on the binary classifier w(i, j) for other categories j The correct classification is 1, and the wrong classification is 0. For each binary classifier w(i, j), the decision distance d between the two types of brain response features f in the test set i and j is statistically analyzed, and the maximum value is d max , the minimum value is d min , calculate the classification confidence C of f, where C = (dd min ) / (d max -d min ).

[0155] Step c: Exchange the training dataset and the test dataset, repeat steps b and c, and complete a two-fold cross validation to obtain the results of all samples in the entire image feature dataset. and its classification confidence C f .

[0156] Step d: all samples obtained according to step c and its classification confidence C f , calculate the true positive rate of each sample f in the entire sample set in multi-classification (category N) And the average false positive rate FPR of each category i , where the calculation formula for the true positive rate of sample f is:

[0157]

[0158] The average false positive rate of the i-th category is FPR i :

[0159]

[0160] Step e: Repeat the above process from step a to step d 10 times, complete 10 two-fold cross validations, and calculate the average true positive rate of sample f in the 10 repeated tests. and the average false positive rate for each category i

[0161] Step f: average true positive rate of image feature f calculated in step e and the average false positive rate of category i Calculate the classification sensitivity index of image feature f

[0162]

[0163] Where Z(.) represents the inverse of Gaussian cumulative distribution. The classification sensitivity has reflected the separability of feature f in the representation space and can be used as a reliability evaluation indicator of its features. The reliability indicator obtained here is the statistical characteristic of the reliability of the entire feature set and can be used as label data for subsequent prediction models.

[0164] (4) Construct feature reliability prediction models for brain response and stimulus image, respectively, use the feature reliability values ​​obtained in (3) as labels, train them on brain response and image feature sets, respectively, and save the trained parameters of the two feature reliability prediction models in a feature reliability prediction device. Figure 5 The training steps are as follows. Here, the reliability prediction model of image features is used as an example to illustrate. The training method of the feature reliability prediction model of brain response is the same:

[0165] The image feature reliability prediction model is constructed. The model adopts a fully connected neural network structure, which consists of three layers: input layer, hidden layer and output layer. The number of neurons is 168, 32 and 1 respectively. The activation function adopts ELU, and batch normalization is added between the hidden layer and the output layer. The output value of the network can be regarded as the feature reliability value predicted by the network. The specific network structure diagram is shown in the figure below. Figure 5 shown.

[0166] The calculated reliability value is used as the reliability label of the image feature, and the mean square error loss function L is used. MSE Supervised prediction model training process:

[0167]

[0168] Where y represents the feature reliability prediction value, y′ represents the feature reliability label value, and n represents the batch size. The prediction model training uses the mean square error loss function and Adam optimizer, with an initial learning rate of 1e-3 and a learning rate decay of 0.1. A total of 100 epochs are trained, and the learning rate decays every 40 epochs.

[0169] The feature reliability prediction models of image features and brain responses are trained separately, and their respective model parameters are saved after successful training.

[0170] (5) Using the feature reliability labels calculated in (3) to weight the cascaded brain responses and image features, a brain-computer information fusion feature set is constructed, and a linear SVM classification model is trained on the fusion feature set. The parameters of the successfully trained model are stored in the brain-computer feature fusion classification device.

[0171] The specific steps of feature weighting method are as follows:

[0172] Load paired data B from the brain response and image feature set i and I i , and the reliability value calculated in step 1) is used as the reliability label of brain response and image features and Calculate the respective weighting coefficients according to the reliability values ​​of the two and

[0173]

[0174]

[0175] Obtain the fusion feature F according to the weighted coefficients of the two i , forming a fusion feature set:

[0176]

[0177] A linear SVM classification model is trained on the obtained fusion feature set as the classification output of the fusion feature set, and the parameters of the successfully trained SVM model are saved.

[0178] Step 2: Reasoning stage:

[0179] Due to the lack of brain response during the reasoning application process, it is necessary to implement the reasoning application of "brain out of loop", which is quite different from the training process of step one, such as Figure 3 As shown in the reasoning stage, the specific reasoning application steps are as follows:

[0180] The data loading device is used to load only the stimulus image data and sent to the preprocessing device to extract the image features and reduce the dimension to 168 dimensions.

[0181] The extracted image features are sent to the brain response prediction device, the brain response prediction model parameters are loaded, and forward reasoning is performed to obtain the reconstructed brain response.

[0182] The extracted image features and the acquired reconstructed brain responses are sent to the feature reliability prediction device in pairs, and the brain response feature reliability model parameters and the image feature reliability prediction model parameters are loaded respectively, and forward reasoning is performed to obtain the feature reliability prediction values ​​of the two.

[0183] The acquired image features, the corresponding reconstructed brain responses and the acquired feature reliability values ​​are sent to the brain-computer information fusion classification device, the brain responses and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

[0184] The application effect of the present invention is described in detail below in conjunction with experiments.

[0185] 1. Experimental conditions

[0186] The hardware conditions of the experiment of the present invention are: an ordinary computer, Intel i5 CPU, 8G memory, and an NVIDIA GeForce GTX 1070 graphics card; the software platform: Ubuntu 18.04, PyTorch deep learning framework, Python 3.6 language; the brain response and stimulus image dataset used in the present invention comes from the public data of the Brain-Score platform of the McGovern Institute for Brain Research at the Massachusetts Institute of Technology.

[0187] 2. Dataset

[0188] The data set used in the present invention includes two parts: stimulus images and brain response data. The stimulus images are composite images of 8 types of targets (animals, ships, cars, chairs, faces, fruits, airplanes, tables) and random natural scenes, with a total of 3200 images, 400 images per type; each type of target contains 8 sub-classes, and the number of images per sub-class is 50. Each stimulus image contains only one target, and the target image is generated by changing the posture of the 3D model of the target object, such as Figure 6 As shown. The brain response data were collected from the ventral stream area of ​​two well-trained adult rhesus monkeys. The brain responses of the corresponding brain areas were captured by a 168-channel electrode array in the inferotemporal area (IT). During the EEG acquisition process, each group of 5-10 stimulus images were presented in the center of the display in turn. Each image was displayed for 100ms, followed by a 100ms blank. The rhesus monkeys kept staring at the center of the display during the whole process. Each stimulus image was presented multiple times, at least 28 times, and an average of 50 times. Among them, the data processing framework disclosed by the Brain-Score (https: / / brain-score.readthedocs.io / en / latest / index.html) platform can be used to preprocess the brain response and obtain the preprocessed brain response characteristics.

[0189] 3. Experimental content

[0190] According to the process of the above-mentioned training phase, the present invention sequentially trains the brain response generation model, the reliability prediction model of brain response and image features, and the linear SVM classification model of fusion features, thereby completely constituting an adaptive brain-computer information fusion classification system that can realize brain-out-of-loop applications.

[0191] According to the process of the above-mentioned reasoning stage, the present invention first verifies the reconstruction effect of the brain response generation model and the prediction performance of the feature reliability prediction model; then tests the accuracy of the adaptive brain-computer information fusion classification of the present invention when the brain response is missing on the test set; and tests the classification accuracy of the single-modal brain response and image features on the same test data set. At the same time, the present invention also compares the effects of different deep learning image feature extraction networks (including AlexNet, VGG, GoogLeNet and ResNet) on the performance of brain-computer information fusion classification.

[0192] 4. Experimental results analysis

[0193] The present invention uses correlation values ​​to measure the similarity between the reconstructed brain response and the true value. The reconstruction results of the brain response reconstructed by image features obtained by different feature extraction networks are shown in Table 1. The specific reconstruction examples are as follows: Figure 7 , where the thick solid line represents the reconstructed brain response and the thin dotted line represents the true value of the brain response. It is found that the feature domain reconstruction method proposed in the present invention can effectively reconstruct the effective information of most brain responses, and there are certain differences between the reconstruction results of different feature extraction networks. The reason for the difference is that the feature extraction capabilities of different feature extraction networks are different and show different correlations with brain responses. The present invention uses absolute error to describe the prediction performance. The average absolute error on IT brain response and image features is 0.4462 (the range of feature reliability values ​​is 0-5), and the variance is 0.002, indicating that the special certificate reliability evaluation model proposed in the present invention can stably and accurately predict the reliability of IT brain response and image features. The present invention compares the difference between the brain-computer information fusion classification performance and the single-modal image feature classification performance under "brain out of loop", as shown in Table 2, compared with the single-modal feature classification performance, the average improvement is 1.31%, indicating that the feature domain reconstruction brain response method proposed in the present invention can effectively solve the problem of brain-computer information fusion classification when brain response is missing. Therefore, the present invention has more practical application value and has broad application prospects under the new paradigm of brain-computer information collaboration.

[0194] Table 1 Reconstruction results of brain responses reconstructed by image features

[0195]

[0196] Table 2 Differences between the classification performance of brain-computer information fusion and the classification performance of single-modality image features

[0197]

[0198] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0199] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A brain-computer information fusion classification method for brain-out-of-loop applications, characterized in that: The brain-computer information fusion classification method for brain-out-of-loop applications includes: Step 1: On the training data set, use a data loading device to load paired brain response and stimulus image data, send them to a preprocessing device, preprocess the brain response, extract image features from the stimulus image, and form a brain response and image feature set; Step 2: training a prediction model from image features to brain responses on the paired feature set, and training feature reliability prediction models on the brain response and image feature sets respectively; Step 3, using the feature reliability of brain response and stimulus image to construct a weighted cascade fusion feature set, and training a linear SVM classification model on the fusion feature set; Step 4: On the test data set, only the stimulus image data is loaded by using a data loading device, and sent to a preprocessing device to extract image features; Step 5, input the extracted image features into the brain response generation device, load the model parameters, perform forward reasoning, and reconstruct the brain response corresponding to the stimulus image; Step 6, inputting the reconstructed brain response and image features into the feature reliability prediction device, loading the model parameters, performing forward reasoning, and outputting the reliability values ​​of the brain response and image features respectively; Step 7, sending the reconstructed brain response, the extracted image features, and the feature reliability values ​​obtained in step 6 to the brain-computer information fusion classification device, weighting the brain response and image features according to the feature reliability values, obtaining the fusion features, loading the linear SVM classification model parameters, and outputting the category to which the fusion features belong and the classification probability of each category; The specific process of the feature reliability prediction device to construct the feature reliability prediction models of brain response and stimulus image respectively is as follows: The obtained feature reliability values ​​are used as labels, and training is performed on the brain response and image feature sets respectively, and the trained two feature reliability prediction model parameters are stored in the feature reliability prediction device; wherein the steps of constructing and training the feature reliability prediction model are as follows: An image feature reliability prediction model was constructed. The model adopted a fully connected neural network structure and consisted of three layers: input layer, hidden layer, and output layer. The number of neurons was 168, 32, and 1 respectively. The activation function adopted ELU, and batch normalization was added between the hidden layer and the output layer. The output value of the network can be considered as the feature reliability value predicted by the network. The calculated reliability value is used as the reliability label of the image feature, and the mean square error loss function L is used. MSE Supervised prediction model training process: Where y represents the feature reliability prediction value, y′ represents the feature reliability label value, and n represents the batch size. The prediction model training uses the mean square error loss function and Adam optimizer, with an initial learning rate of 1e-3 and a learning rate decay of 0.

1. A total of 100 epochs are trained, and the learning rate decays every 40 epochs. The feature reliability prediction models of image features and brain responses are trained separately, and the respective model parameters are saved after successful training; The brain response and image features are weighted by the calculated feature reliability labels to construct a brain-computer information fusion feature set, and a linear SVM classification model is trained on the fusion feature set. The successfully trained model parameters are stored in the brain-computer feature fusion classification device. The specific steps of feature weighting method are as follows: Load paired data B from the brain response and image feature set i and I i , and the calculated reliability values ​​are used as reliability labels for brain responses and image features and Calculate the respective weighting coefficients according to the reliability values ​​of the two and Obtain the fusion feature F according to the weighted coefficients of the two i , forming a fusion feature set: A linear SVM classification model is trained on the obtained fusion feature set as the classification output of the fusion feature set, and the parameters of the successfully trained SVM model are saved.

2. The brain-computer information fusion classification method for brain-out-of-loop applications as claimed in claim 1, characterized in that: The data loading device simultaneously loads the paired brain response and stimulation image data, and sends them to the brain response and stimulation image preprocessing devices respectively, extracts the brain response preprocessing features and image features respectively, and constructs paired brain response and stimulation image feature sets; The data loading device inherits from the DataSet class and DataLoader class in the PyTorch deep learning framework, and reads paired training data from paired brain response and stimulus image data sets in turn during the training phase, and performs subsequent processing respectively; The preprocessing device is divided into two parts, which respectively preprocesses the brain response and extracts the stimulus image features using the ResNet34 pre-trained model to construct a brain response and image feature set.

3. The brain-computer information fusion classification method for brain-out-of-loop application as claimed in claim 2, characterized in that: The brain response is preprocessed as follows: From the original brain response data, along the stimulus presentation dimension, the brain responses collected by multiple presentations of the same stimulus image are obtained, and the brain responses belonging to the same stimulus image are averaged; Select brain response data from electrodes placed in the inferior temporal region of the ventral visual pathway and extract the corresponding brain responses; For the extracted brain response data, the brain response data of 70-170ms were averaged along the time dimension to obtain the spatial response data of each electrode point; The acquired brain response data is standardized and flipped to obtain the preprocessed 1*168-dimensional brain response data to construct a brain response feature set.

4. The brain-computer information fusion classification method for brain-out-of-loop application as claimed in claim 2, characterized in that: The feature extraction and preprocessing process of the stimulus image preprocessing device is as follows: Use the PyTorch deep learning framework to build a ResNet34 model, remove its last fully connected layer, load the ImageNet pre-trained model parameters in the torchvison model library, and set the model parameter "pretrained = True"; The stimulus image was loaded using a data loading device and input into the ResNet34 model for forward inference, the output features of the convolutional layer were obtained as 512 dimensions, and an image feature set was constructed; In order to eliminate the dimensionality difference, the acquired image feature set is reduced to 128 dimensions using principal component analysis to construct the image feature set.

5. The brain-computer information fusion classification method for brain-out-of-loop application as claimed in claim 1, characterized in that: The prediction model of image features to brain responses is trained on the paired feature set, and the feature reliability prediction model is trained on the brain response and image feature set respectively. The specific process is as follows: On the paired brain response and stimulus image feature sets, a feature domain prediction model from image features to brain response is constructed, the brain response feature set is used as the true value, the paired image feature set is used to train the brain response generation model, and the successfully trained prediction model parameters are stored in the brain response prediction device; The specific steps for building and training the brain response generation model are as follows: Construct a brain response denoising autoencoder structure, which uses a fully connected neural network structure. The encoding module consists of three hidden layers with 168, 96 and 32 neurons respectively. The decoding module also consists of three hidden layers with 32, 96 and 168 neurons. The hidden layer uses the ELU activation function, and a batch normalization layer is added between every two layers. The output of the decoding module is consistent with the dimension of the brain response and is used to reconstruct the brain response. Load the brain response feature set, add a random Gaussian noise to each input brain response, the random Gaussian noise is generated by Python's numpy toolkit, with a mean of 0 and a variance of 0.2; output the noisy brain response to the denoising autoencoder structure, and use the mean square error loss function to supervise the loss between the reconstructed brain response and the real brain response: Where y represents the brain response reconstructed by the decoding module, y' represents the real brain response, and n represents the batch size. The Adam optimizer is used with an initial learning rate of 1e-3 and a learning rate decay of 0.

1. A total of 50 epochs are trained, and the learning rate decays every 20 epochs. A feature domain mapping model from image features to brain responses was constructed, using a fully connected neural network structure with only a feature encoding structure consisting of three hidden layers with 168, 96, and 32 neurons, respectively. The decoding structure shared parameters with the decoding module of the denoising autoencoder model of the brain response, and was used to construct an information flow pipeline from the image feature domain to the reconstruction of the brain response. Load the parameters of the successfully trained autoencoder model, set the encoder parameters of the autoencoder model to be fixed, and the decoder parameters to be learnable; Load the paired brain responses and image features into the encoder of the denoising autoencoder model and the constructed image feature encoder to obtain the feature domain features f b and f i , and the decoder output of the denoised autoencoder model simultaneously produces the reconstructed brain response r from two sources b and r i , and the real brain response is recorded as g; when training the model, the main idea is to make the image features and brain responses show similarity in the feature domain after the encoder nonlinear mapping, and the decoder decoding and reconstruction should also be consistent with the real brain response, and use the absolute error to supervise the training process of the feature domain and the reconstruction result: L total =L f +L b +L i L f =L MAE (in b ,f i ) L b =L MAE (r b ,g) L i =L MAE (r i ,g); The Adam optimizer was used to optimize the model, with an initial learning rate of 1e-3 and a learning rate decay of 0.

1. A total of 50 epochs were trained, and the learning rate decayed every 20 epochs. Load the successfully trained image encoder model parameters and the decoder parameters of the denoising autoencoder model, build a complete information flow pipeline from image features to brain response reconstruction, and save the model parameters.

6. The brain-computer information fusion classification method for brain-out-of-loop application as claimed in claim 1, characterized in that: The weighted cascade fusion feature set is constructed by utilizing the feature reliability of brain response and stimulus image, and a linear SVM classification model is trained on the fusion feature set. The specific process is as follows: The linear SVM models are trained on the brain response and image feature sets respectively, and the classification sensitivity values ​​of the brain response and image features are calculated respectively as the reliability labels of the brain response and image features; The calculation steps of the classification sensitivity value of the feature are as follows, The image feature set is randomly divided into a training set and a test set, with each set accounting for half of the data. Based on the categories in the training set, image features of corresponding categories are selected in pairs to train a binary classification linear SVM model. All possible binary classifier combinations are trained in turn, and the model parameters are retained. For any test set image feature f, load the binary classifier model parameters w(i, j) related to its category i in turn, input f, record the distance d from f to the decision boundary of the binary classifier w(i, j), and the binary classification results of f on the binary classifier w(i, j) for other categories j The correct classification is 1, and the wrong classification is 0; for each binary classifier w(i, j), the decision distance d between the two types of brain response features f in the test set i and j is statistically analyzed, and the maximum value is d max , the minimum value is d min , calculate the classification confidence C of f, where C = (dd min ) / (d max -d min ); By exchanging the training dataset and the test dataset and repeating the above process, a two-fold cross validation can be completed to obtain the results of all samples in the entire image feature dataset. and its classification confidence C f ; According to all the samples obtained and its classification confidence C f , calculate the true positive rate of each sample f in the entire sample set in multi-classification And the average false positive rate FPR of each category i , where the calculation formula for the true positive rate of sample f is: The average false positive rate of the i-th category is FPR i : Repeat the above process 10 times, complete 10 two-fold cross validations, and calculate the average true positive rate of sample f in 10 repeated tests and the average false positive rate for each category i According to the average true positive rate of the image feature f calculated and the average false positive rate of category i Calculate the classification sensitivity index of image feature f Where Z(.) represents the inverse of Gaussian cumulative distribution; the classification sensitivity has reflected the separability of feature f in the representation space. As the reliability evaluation index of its feature, the obtained reliability index is the statistical characteristic of the reliability of the entire feature set, which serves as the label data of the subsequent prediction model.

7. The brain-computer information fusion classification method for brain-out-of-loop applications as claimed in claim 1, characterized in that: The specific process of forward reasoning is as follows: Only the stimulus image data is loaded by the data loading device and sent to the preprocessing device to extract the image features and reduce the dimension to 168 dimensions; The extracted image features are sent to a brain response prediction device, the brain response prediction model parameters are loaded, and forward reasoning is performed to obtain the reconstructed brain response; The extracted image features and the obtained reconstructed brain responses are sent to the feature reliability prediction device in pairs, the brain response feature reliability model parameters and the image feature reliability prediction model parameters are loaded respectively, and forward reasoning is performed to obtain the feature reliability prediction values ​​of the two respectively; The acquired image features, the corresponding reconstructed brain responses and the acquired feature reliability values ​​are sent to the brain-computer information fusion classification device, the brain responses and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

8. A brain-computer information fusion classification system for brain-out-of-the-loop applications implementing the brain-computer information fusion classification method for brain-out-of-the-loop applications as described in any one of claims 1 to 7, characterized in that: The brain-computer information fusion classification system for brain-out-of-loop applications comprises: a data loading device, an image preprocessing device, a brain response generating device, an image feature reliability prediction device and a brain-computer feature fusion classification device; A data loading device, connected to the input image and the input brain response data, for loading the stimulus image and the original brain response to be input into the system; An image preprocessing device, connected to the data loading device, for extracting image features of the stimulus image and preprocessing the brain response; A brain response generating device is connected to the preprocessing device and establishes a feature domain reconstruction model from image features to brain responses, so as to reconstruct brain responses using image features; The image feature reliability prediction device is connected to the preprocessing device and is composed of a brain response reliability prediction and an image feature reliability prediction, and predicts the reliability of the input brain response and image feature respectively; The brain-computer feature fusion classification device is respectively connected to the preprocessing device, the brain response generation device and the feature reliability prediction device, and uses the predicted value of feature reliability to weight the cascade reconstructed brain response and image features to form a fusion feature, and classifies the fusion feature to output the result.

9. A storage medium for receiving user input programs, wherein the stored computer program enables an electronic device to execute the brain-computer information fusion classification method for brain-out-of-loop application as claimed in any one of claims 1 to 7, comprising the following steps: Step 1: On the training data set, use a data loading device to load paired brain response and stimulus image data, send them to a preprocessing device, preprocess the brain response, extract image features from the stimulus image, and form a brain response and image feature set; Step 2: training a prediction model from image features to brain responses on the paired feature set, and training feature reliability prediction models on the brain response and image feature sets respectively; Step 3, using the feature reliability of brain response and stimulus image to construct a weighted cascade fusion feature set, and training a linear SVM classification model on the fusion feature set; Step 4: On the test data set, only the stimulus image data is loaded by using a data loading device, and sent to a preprocessing device to extract image features; Step 5, input the extracted image features into the brain response generation device, load the model parameters, perform forward reasoning, and reconstruct the brain response corresponding to the stimulus image; Step 6, inputting the reconstructed brain response and image features into the feature reliability prediction device, loading the model parameters, performing forward reasoning, and outputting the reliability values ​​of the brain response and image features respectively; In step seven, the reconstructed brain response, the extracted image features, and the feature reliability values ​​obtained in step six are sent to the brain-computer information fusion classification device, the brain response and image features are weighted and cascaded according to the feature reliability values, the fusion features are obtained, the linear SVM classification model parameters are loaded, and the category to which the fusion features belong and the classification probability of each category are output.

Images