Visual stimulation multi-classification method and system based on deep reinforcement learning

Abstract

The invention discloses a visual stimulation multi-classification method and system based on deep reinforcement learning, and the method comprises the steps: taking an EEG signal as a sample from the perspective of the EEG signal, coding a time-varying visual feature through a deep long-short-term memory network, depicting the selection of a time period as a Markov decision process, and carrying out the deep reinforcement learning. Autonomous selection of time periods is realized through a deep reinforcement learning algorithm, and time domain information of features is aggregated and input into a classifier to stimulate classification of images. According to the method, the time dynamic information in the EEG signals can be effectively captured, and compared with an existing method, a better visual stimulation classification result can be obtained.

Description

Technical Field

[0001] The present invention relates to the research on electroencephalogram (EEG) signal processing, feature extraction and selection, and specifically, to EEG time-varying feature extraction based on a deep long short-term memory (LSTM) network and EEG feature selection based on deep reinforcement learning (DRL). Background Art

[0002] Decoding the human brain is one of the most attractive problems in brain computer interface (BCI) research. Over the past few decades, researchers have been actively studying how to decode the human brain from brain signals to facilitate communication or recover past memories. For example, developing BCI systems helps people with disabilities communicate with others by identifying their thoughts. In addition, researchers have also paid great attention to deciphering visual information in the human brain and identifying the visual stimuli experienced by humans from brain signals. Recent studies have performed well in identifying the categories of visual stimuli based on brain signals. With the continuous development of neuroimaging technology, there are several types of brain signals used to develop BCI systems, such as EEG, magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI). Among them, EEG is considered to be the most advantageous because it can be recorded non-invasively from different locations of the scalp by measuring voltage fluctuations. Due to the convenience of the acquisition process, EEG-based BCI systems have been widely used in real-world applications such as games or brain training. In addition, classification tasks based on EEG have also attracted attention, including emotion recognition, sleepiness detection, and decoding visual memory. We mainly study the task of visual stimulus recognition based on image-evoked EEG signals.

[0003] However, when the subject is not fully focused during the visually evoked EEG acquisition process due to unfamiliarity with the BCI or uncomfortable conditions (such as long calibration time), the collected EEG data may have not only information related to the visual stimulation, but also irrelevant information, which is considered as unreliable EEG segments in the following description. In addition, visually evoked EEG also has different physiological noises, such as heartbeat, eye movement, etc. Therefore, it is unreasonable to assume that the obtained EEG trials have complete reliability. Therefore, the present invention proposes a visual stimulation multi-classification method based on deep reinforcement learning for the automatic selection of EEG reliable segments, in order to improve the multi-classification performance of visual stimulation. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to address the fact that the current multi-classification methods of visual stimuli ignore the full utilization of EEG temporal dynamic information, and to propose a multi-classification method and system for visual stimuli based on deep reinforcement learning, which can automatically select reliable EEG segments, in order to improve the multi-classification performance of visual stimuli.

[0005] Technical solution: To solve the above problems, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a visual stimulus multi-classification method based on deep reinforcement learning, the method comprising the following steps:

[0007] Step 1, band-pass filtering the raw EEG data recorded by the EEG system, and intercepting the signal with the time point of stimulus presentation during the data acquisition experiment as the reference point to form an EEG trial;

[0008] Step 2, use a deep LSTM network to extract time-varying visual features of multi-channel EEG time series;

[0009] Step 3: Design the time period selection as a Markov decision process (MDP), and realize autonomous time period selection through deep reinforcement learning algorithm; in MDP, the state is defined as whether there is a feature change in the current time step, the action is defined as whether to keep the current time step, and the reward is defined as the difference between the selected feature and the global feature classification loss up to the current moment;

[0010] Step 4: Aggregate the temporal information of the time-varying features and input them into the classifier for classifying the stimulus images.

[0011] Furthermore, in step 1, a fifth-order Butterworth filter is used for band-stop filtering and band-pass filtering, and complete data of all channels and all time periods are obtained for subsequent processing.

[0012] Furthermore, in step 2, a deep LSTM network is used to extract time-varying visual features of multi-channel EEG time series. The method is as follows: Assume that the EEG data preprocessed in step 1 is , where C is the number of channels and T is the number of sampling points. represents a real number; the input multi-channel EEG time series is converted into a feature vector through a deep LSTM network, in which the output of the first layer is provided as the input of the second layer, and so on, the output matrix of all time steps of the deepest LSTM layer is used as the feature representation of the entire EEG input sequence.

[0013] Furthermore, in step 3, the time period selection is designed as an MDP, and the autonomous time period selection is realized through the reinforcement learning algorithm. The method is as follows: Assume that the time-varying feature matrix of the input signal x extracted by step 2 is Where T is the number of sampling points, D is the dimension of the feature vector corresponding to each time point, {.} T Represents transposition; the selection of time periods is described as MDP, and autonomous selection is achieved by modeling the DRL framework. The key elements of MDP, state, action and reward are defined as follows: 1) The state is Among them I t-1 is the index set of the feature vectors selected until time t-1, Ave is the aggregation operator, that is, averaging the feature matrix in the time dimension; 2) The action is That is, when the action is 1, it includes the current time t; 3) The reward is in and They are Ave(F) and Classification loss; Calculate classification loss As a criterion, the cumulative discount reward is then defined as Here, the discount factor γ∈(0, 1] is a hyperparameter used to enforce delayed rewards.

[0014] Furthermore, the actor-critic AC policy gradient algorithm is used to realize autonomous time segment selection. The algorithm combines the ideas of value function approximation and policy gradient, and includes two networks: policy network and value network. The policy network selects action a according to state s, that is, a = π(s|θ π ), θ π represents the parameters of the policy network; the value network calculates the state value V(s; θ v ) to evaluate the choice of policy network, θ v Represents the parameters of the value network; the policy network calculates the policy gradient according to the state value and iteratively updates the adjustable parameter θ π , to realize the training of the policy network; the training of the value network is synchronized with the policy network, and the adjustable parameter θ is updated through the error between the target value and the predicted value v , in order to achieve the training of the value network.

[0015] Furthermore, the process of optimizing network parameters in the AC algorithm is as follows:

[0016] First, according to the policy gradient theorem, the parameters of the policy network are learned along the following directions:

[0017]

[0018] Since it is impossible to directly know the state value V(s) in a specific state t θ v), using the return G t To approximate it, it is defined as:

[0019]

[0020] Among them, r t is the reward value at the current moment, γ is the discount factor, The state value predicted by the value network at the next moment; introduce the advantage function A t As G t Alternative:

[0021]

[0022] Where b(s t ) is a baseline function that does not depend on action a, using the state value predicted by the current value network The policy gradient is updated by the following formula:

[0023]

[0024] And the loss function of the policy network is naturally defined as:

[0025]

[0026] The loss function of the value network is defined as:

[0027]

[0028] Furthermore, in step 4, the temporal information of the time-varying features is aggregated and input into the classifier for the classification of the stimulus image. The method is as follows: Assume that the action obtained in step 3 is used as a mask for the corresponding selection of the time-varying visual features, that is, Represents point-to-point product; the aggregation function Ave is used to integrate the feature time domain information and input it into the classifier φ to realize multi-classification of visual stimuli:

[0029]

[0030] Among them, θ φ is a learnable parameter, The classification prediction results.

[0031] In a second aspect, the present invention provides a visual stimulus multi-classification system based on deep reinforcement learning, comprising:

[0032] The EEG data preprocessing module is used to perform bandpass filtering on the original EEG data recorded by the EEG system, and intercept the signal with the time point of stimulus presentation during the data acquisition experiment as the reference point to form an EEG trial;

[0033] A time-varying feature encoding module for extracting time-varying visual features of multi-channel EEG time series using a deep long short-term memory (LSTM) network;

[0034] The autonomous time period selection module is used to design the time period selection as a Markov decision process (MDP) and realize autonomous time period selection through a deep reinforcement learning algorithm. In the MDP, the state is defined as whether there is a feature change in the current time step, the action is defined as whether the current time step is retained, and the reward is defined as the difference between the selected feature up to the current moment and the global feature classification loss.

[0035] The time-varying feature aggregation and multi-classification module is used to aggregate the time domain information of the time-varying features and input it into the classifier for classification of stimulus images.

[0036] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program / instructions stored in the memory and executable on the processor, wherein when the computer program / instructions are executed by the processor, the steps of the visual stimulus multi-classification method based on deep reinforcement learning are implemented.

[0037] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the visual stimulus multi-classification method based on deep reinforcement learning.

[0038] Beneficial effects: Most existing methods rely on fixed time windows or specific time periods set by experts, or use simple time series modeling methods, which ignore the complex temporal dynamic information in EEG signals. The present invention integrates deep reinforcement learning technology to realize the autonomous selection of EEG time periods, effectively capturing the temporal dynamic characteristics of EEG signals. By aggregating the temporal information of features and inputting them into the classifier for the classification of stimulus images, the designed end-to-end neural network simplifies the task process. Compared with traditional methods of distribution optimization including preprocessing, feature extraction and classification, the present invention ensures the consistent optimization of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 Schematic diagram of extracting time-varying visual features for a deep LSTM network;

[0041] Figure 3 Schematic diagram of DRL's autonomous selection of time periods;

[0042] Figure 4 Collect the EEG power spectrum of a specific channel when the subject is stimulated by the external environment;

[0043] Figure 5 Action sequence diagram for spectrum distribution and its corresponding agent module decision. DETAILED DESCRIPTION

[0044] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the model in the present invention should be trained and optimized according to specific data.

[0045] like Figure 1 As shown, a method for multi-classification of visual stimuli based on deep reinforcement learning disclosed in an embodiment of the present invention mainly includes four steps: (1) EEG data preprocessing; (2) time-varying feature encoding; (3) autonomous selection of time periods; (4) time-varying feature aggregation and multi-classification. Each step is described in detail below.

[0046] Step (1): bandpass filter the raw EEG data recorded by the EEG system, and intercept the signal with the time point of stimulus presentation during the data acquisition experiment as the reference point to form an EEG trial.

[0047] This embodiment uses a public large-scale dataset EEG-ImageNet 40 for illustration. The dataset consists of 128 channels of EEG signals with a duration of 440 milliseconds from six different subjects: one female and five males. The stimulus images consist of 40 object categories in ImageNet, with approximately 50 images per category. For evaluation, we follow the 4:1:1 ratio provided by the official dataset contributors for training, validation, and testing. We follow the official 6 segmentations for experiments and report the mean and standard deviation of all segmentation results. In order to remove noise signals and capture the frequency bands related to visual perception (i.e., β and γ), a fifth-order Butterworth filter is used for bandstop filtering (49-51Hz) and bandpass filtering (14-72Hz), and complete data of all channels and all time periods are taken for subsequent processing.

[0048] Step (2): Use a deep LSTM network to extract time-varying visual features of multi-channel EEG time series.

[0049] Specifically, assume that the EEG data after preprocessing in step (1) is Where C is the number of channels and T is the number of sampling points. The input multi-channel EEG time series is converted into a low-dimensional feature vector. LSTM is a special recurrent neural network (RNN) that can effectively capture long-term dependencies when processing sequence data. Compared with ordinary RNN, LSTM has stronger memory and better gradient propagation, so it performs better when processing long sequence data. The core idea of ​​LSTM is to control the flow of information through a gating mechanism, including input gate, forget gate and output gate. These gates control the input, forgetting and output of information, so that LSTM can selectively remember and forget previous information, so as to better process sequence data. The previous method only connects the time series of multiple channels into a feature vector, ignoring the temporal dynamics, which contains basic information for understanding brain electrical activity. In order to incorporate this dynamic feature into the model, the present invention uses a deep LSTM recursive neural network ψ stacked by LSTM layers to track long-term dependencies in the input data. The number of stacked layers of the LSTM network is determined by comprehensively considering computing resources, training data scale and task requirements, and 2-4 layers can be used. This embodiment uses a 4-layer LSTM network. Specifically, the output of the first layer is provided as the input of the second layer, and so on. The output matrix of all time steps of the deepest LSTM layer is used as the feature representation of the entire EEG input sequence, as shown in the schematic diagram. Figure 2 As shown, and Respectively represent the hidden state and cell state of the lth layer at time step t. It is worth noting that since EEG data contains a lot of information that is not related to visual stimulation, in order to solve the multi-classification task of visual stimulation, the output of the last time step is not used as the feature representation of the entire EEG sequence, but the data of all time steps is output to retain the dynamic characteristics of the visual features to the greatest extent. The model can be expressed by the following formula:

[0050] F = ψ(x; θ ψ )

[0051] Among them, θ ψ It is an adjustable parameter.

[0052] Step (3): Design the time period selection as an MDP, and implement autonomous time period selection through a deep reinforcement learning algorithm.

[0053] Specifically, the time-varying feature matrix of the input signal x extracted by step (2) is where D is the dimension of the feature vector corresponding to each time point, {·} T We hope to design a method for autonomously selecting signal time periods so that the selected feature vector It can carry the most relevant information induced by external visual stimulation. However, visual stimulation involves internal cognitive processes in the brain, so there is no clear label to indicate the relevant information actually included at each time point. To this end, the present invention creatively describes the selection of time periods as MDP, and realizes autonomous voxel selection by modeling the DRL framework. MDP contains five basic elements, namely (S, A, P, r, γ), which respectively represent the state set, action set, state transition matrix, reward signal and discount factor. Since there are no specific factors that may interfere with the selection of time periods in this process, the present invention adopts a model-free DRL framework that does not require environment modeling. The key elements state, action and reward in MDP are defined as follows: 1) state is Among them I t-1 is the index set of the feature vectors selected until time t-1, Ave is the aggregation operator, that is, averaging the feature matrix in the time dimension. The specific formula is for That is, when the action is 1, it includes the current time t; 3) Reward is We define the basic information by applying global average pooling (GAP) to the entire feature vector in our experiments, where and They are Ave(F) and Calculate the classification loss. As a criterion, the cumulative discount reward is then defined as Among them, the discount factor γ∈(0, 1] is a hyperparameter for executing delayed rewards. The core of DRL is to train a strategy that maximizes the return in the entire MDP. The advantage of policy gradient algorithms such as REINFORCE, actor-critic, PPO, and DDPG is that it directly iterates the strategy without evaluating the strategy. In this embodiment, the classic actor-critic policy gradient algorithm is used for specific implementation. The algorithm combines the ideas of value function approximation and policy gradient, and includes two networks: a policy network and a value network. On the one hand, the policy network plays the role of an actor and is responsible for updating the policy parameters. On the other hand, the value network plays the role of a critic and is responsible for calculating the state value. The policy network selects action a through state s, that is, a=π(s|θ π ), θ π represents the parameters of the policy network. Then, the value network calculates the state value V(s; θ v ) to evaluate the choice of policy network, θ v Represents the parameters of the value network. Finally, the policy network calculates the policy gradient based on the state value and iteratively updates the adjustable parameter θ π, in order to realize the training of the policy network. The training of the value network must be synchronized with the policy network, using the feedback r generated by each interaction with the environment t+1 and t+1 To calculate the temporal difference (TD) target value, use the value network to calculate the predicted value, and update the adjustable parameter θ by the error between the target value and the predicted value v , in order to achieve the training of the value network. In this embodiment, by considering the algorithm applicability and computational cost, an AC network with synchronization advantage is realized. The specific process of the algorithm to optimize the network parameters is as follows:

[0054] First, according to the policy gradient theorem, the parameters of the policy network are learned along the following directions:

[0055]

[0056] Since it is impossible to directly know the state value V(s) in a specific state t θ v ), we use the return G t To approximate it, it is defined as:

[0057]

[0058] Among them, r t is the reward value at the current moment, γ is the discount factor, is the state value predicted by the value network at the next moment. t Realized the V(s t θ v ), but it brings significant noise and variance. For this reason, the advantage function A is introduced t As G t Alternative:

[0059]

[0060] Where b(s t ) is a baseline function that does not depend on action a, such as the state value predicted by the current value network used here The introduction of the advantage function does not change the direction of the gradient estimate θ π Finally, the policy gradient is updated as follows:

[0061]

[0062] And the loss function of the policy network is naturally defined as:

[0063]

[0064] At the same time, through the analysis of the above strategy network training process, it is not difficult to find that the actors just cater to the preferences of the judges. Therefore, in order for the actors to perform better, the level of the judges also needs to be higher. In other words, the state value V(s t θ v ) requires a more accurate approximation of G t Therefore, the value network is trained synchronously and the training loss is defined as:

[0065]

[0066] The specific algorithm process is as follows Figure 3 shown.

[0067] Step (4): Take the action a obtained in step (3) t As a mask, it is used to select corresponding time-varying visual features, that is, Subsequently, the feature temporal information is integrated through the aggregation function Ave and input into the classifier φ to achieve multi-classification of visual stimuli:

[0068]

[0069] Among them, θ φ is a learnable parameter, is the classification prediction result. In this embodiment, φ consists of two linear layers and a Softmax activation function. The above end-to-end model is trained by the cross entropy loss between the predicted value and the true value to obtain the optimal network parameters and The present invention provides a pseudo code for a batch of data (n samples) on a training set to participate in training, as shown in Table 1.

[0070] Table 1 Pseudo code for model training (taking a batch of data as an example)

[0071]

[0072] In addition, the specific network architecture configuration involved in the model is shown in Table 2:

[0073] Table 2 Network architecture configuration

[0074]

[0075] In this example, the EEG-ImageNet 40 dataset is used as an example. Figure 4The power spectrum of a specific channel of a subject when receiving external stimulation is shown, and different visual stimulation images for different behaviors. It can be clearly found that even for the same subject, when receiving different visual stimulations at different times, or even the same visual stimulation, the signal patterns are very different, and reliable EEG segments do not cover the entire time segment. This also illustrates the necessity of effective selection of time segments.

[0076] In addition, Table 3 reports the performance achieved by three configurations with various LSTM architecture details. Among them, LSTM uses only one layer of LSTM network, and the output of the last time step is used as the feature; LSTMs is a stack of four layers of LSTM network, and the output of the last time step is used as the feature; LSTMs-full is a stack of four layers of LSTM network, and the output of the full time step is used as the feature.

[0077] Table 3 Classification accuracy of various LSTM architectures

[0078]

[0079] It can be found that on this dataset, the LSTMs-full method significantly outperforms the remaining methods, which shows the necessity of retaining temporal dynamic information. Figure 5 The spectrogram of a channel of two random visual stimuli viewed by the subject (specifically implemented by short-time Fourier transform) and the corresponding action sequence of our agent module decision are visualized in Figure 4. Considering the power spectrum coupling and the agent's choice, we can observe a positive relationship between them, that is, the selected signal segment shows higher spectral power in the μ and β bands. Finally, to illustrate the effectiveness of the proposed method, we compare the proposed algorithm with the above baseline method, and the specific classification accuracy is shown in Table 4.

[0080] Table 4 Comparison of classification accuracy between the proposed algorithm and the baseline method

[0081]

[0082] Based on the same inventive concept, an embodiment of the present invention discloses a visual stimulation multi-classification system based on deep reinforcement learning, including: an EEG data preprocessing module, used to bandpass filter the original EEG data recorded by the EEG system, and intercept the signal with the time point of stimulus presentation during the data acquisition experiment as a reference point to form an EEG trial; a time-varying feature encoding module, used to extract time-varying visual features of a multi-channel EEG time series using a deep long short-term memory (LSTM) network; a time period autonomous selection module, used to design the selection of time periods as a Markov decision process (MDP), and realize autonomous time period selection through a deep reinforcement learning algorithm; in the MDP, the state is defined as whether there is a feature change in the current time step, the action is defined as whether to retain the current time step, and the reward is defined as the difference between the selected feature up to the current moment and the global feature classification loss; a time-varying feature aggregation and multi-classification module, used to aggregate the time domain information of the time-varying features, and input it into the fully connected layer for classification of stimulus images.

[0083] An embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program / instructions stored in the memory and executable on the processor, wherein the computer program / instructions, when executed by the processor, implement the steps of the visual stimulus multi-classification method based on deep reinforcement learning.

[0084] An embodiment of the present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the visual stimulation multi-classification method based on deep reinforcement learning.

[0085] Although the description of the present invention has been quite detailed and specifically describes one embodiment, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the appended claims in view of the prior art, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

Claims

1. A visual stimulus multi-classification method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1, band-pass filtering the raw EEG data recorded by the EEG system, and intercepting the signal with the time point of stimulus presentation during the data acquisition experiment as the reference point to form an EEG trial; Step 2, use a deep long short-term memory (LSTM) network to extract time-varying visual features of multi-channel EEG time series; Step 3: Design the time period selection as a Markov decision process (MDP), and realize autonomous time period selection through deep reinforcement learning algorithm; in MDP, the state is defined as whether there is a feature change in the current time step, the action is defined as whether to keep the current time step, and the reward is defined as the difference between the selected feature and the global feature classification loss up to the current moment; Step 4: Aggregate the temporal information of the time-varying features and input them into the classifier for classifying the stimulus images.

2. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 1, characterized in that: In step 1, a fifth-order Butterworth filter is used for band-stop filtering and band-pass filtering, and complete data of all channels and all time periods are obtained for subsequent processing.

3. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 1, characterized in that: In step 2, a deep LSTM network is used to extract time-varying visual features of multi-channel EEG time series. The method is as follows: Assume that the EEG data preprocessed in step 1 is Where C is the number of channels, T is the number of sampling points, represents a real number; the input multi-channel EEG time series is converted into a feature vector through a deep LSTM network, in which the output of the first layer is provided as the input of the second layer, and so on, the output matrix of all time steps of the deepest LSTM layer is used as the feature representation of the entire EEG input sequence.

4. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the time period selection is designed as an MDP, and the autonomous time period selection is realized through the reinforcement learning algorithm. The method is as follows: Assume that the time-varying feature matrix of the input signal x extracted by step 2 is Where T is the number of sampling points, D is the dimension of the feature vector corresponding to each time point, {·} T Represents transposition; the selection of time periods is described as MDP, and autonomous selection is achieved by modeling the DRL framework. The key elements of MDP, state, action and reward are defined as follows: 1) The state is Among them I t-1 is the index set of the feature vectors selected until time t-1, Ave is the aggregation operator, that is, averaging the feature matrix in the time dimension; 2) The action is That is, when the action is 1, it includes the current time t; 3) The reward is in and They are Ave(F) and Classification loss; Calculate classification loss As a criterion, the cumulative discount reward is then defined as Here, the discount factor γ∈(0,1] is a hyperparameter used to enforce delayed rewards.

5. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 4, characterized in that: The actor-critic AC policy gradient algorithm is used to realize autonomous time segment selection. The algorithm combines the ideas of value function approximation and policy gradient, and includes two networks: policy network and value network. The policy network selects action a through state s, that is, a=π(s|θ π ), θ π represents the parameters of the policy network; the value network calculates the state value V(s; θ v ) to evaluate the choice of policy network, θ v Represents the parameters of the value network; the policy network calculates the policy gradient according to the state value and iteratively updates the adjustable parameter θ π , to realize the training of the policy network; the training of the value network is synchronized with the policy network, and the adjustable parameter θ is updated through the error between the target value and the predicted value v , in order to achieve the training of the value network.

6. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 5, characterized in that: The process of optimizing network parameters in the AC algorithm is as follows: First, according to the policy gradient theorem, the parameters of the policy network are learned along the following directions: Since it is impossible to directly know the state value V(s) in a specific state t θ v ), using the return G t To approximate it, it is defined as: Among them, r t is the reward value at the current moment, γ is the discount factor, The state value predicted by the value network at the next moment; introduce the advantage function A t As G t Alternative: Where b(s t ) is a baseline function that does not depend on action a, using the state value predicted by the current value network The policy gradient is updated by the following formula: And the loss function of the policy network is naturally defined as: The loss function of the value network is defined as:

7. The visual stimulation multi-classification method based on deep reinforcement learning according to claim 1, characterized in that: In step 4, the temporal information of the time-varying features is aggregated and input into the classifier for the classification of the stimulus image. The method is as follows: Assume that the action obtained in step 3 is used as a mask for the corresponding selection of the time-varying visual features, that is, Represents point-to-point product; the aggregation function Ave is used to integrate the feature time domain information and input it into the classifier φ to realize multi-classification of visual stimuli: Among them, θ φ is a learnable parameter, The classification prediction result.

8. A visual stimulus multi-classification system based on deep reinforcement learning, characterized in that: include: The EEG data preprocessing module is used to perform bandpass filtering on the original EEG data recorded by the EEG system, and intercept the signal with the time point of stimulus presentation during the data acquisition experiment as the reference point to form an EEG trial; A time-varying feature encoding module for extracting time-varying visual features of multi-channel EEG time series using a deep long short-term memory (LSTM) network; The autonomous time period selection module is used to design the time period selection as a Markov decision process (MDP) and realize autonomous time period selection through a deep reinforcement learning algorithm. In the MDP, the state is defined as whether there is a feature change in the current time step, the action is defined as whether the current time step is retained, and the reward is defined as the difference between the selected feature up to the current moment and the global feature classification loss. The time-varying feature aggregation and multi-classification module is used to aggregate the time domain information of the time-varying features and input it into the classifier for classification of stimulus images.

9. A computer system comprising a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor, characterized in that: When the computer program / instructions are executed by a processor, the steps of the visual stimulus multi-classification method based on deep reinforcement learning are implemented according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the visual stimulus multi-classification method based on deep reinforcement learning are implemented according to any one of claims 1 to 7.

Images