A method for analyzing electroencephalogram data based on reinforcement learning

Abstract

The application discloses a kind of electroencephalogram data analysis methods based on reinforcement learning, comprising the following steps: S1 gathers each brain area electroencephalogram and is preprocessed to generate fragment;S2 calculates frequency domain features to generate characteristic vector sequence;S3 constructs and initializes growing prototype learning vector quantization network, and outputs nerve state code;S4 executes prototype addition, merging, deletion according to growth criterion and updates state code;S5 constructs reinforcement learning state by state code and performance parameter and determines action execution task;S6 updates action value table according to SARSA and generates group base model;S7 generates individualized action value by copying base and forms individualized model by SARSA update.The present application realizes electroencephalogram training self-adaptive decision and individualized continuous evolution, improves training efficiency and long-term stability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for analyzing electroencephalogram (EEG) data based on reinforcement learning. Background Technology

[0002] With the development of brain-computer interface (BCI) technology and intelligent interaction technology, cognitive state analysis and training methods based on electroencephalogram (EEG) signals are gradually being applied to fields such as attention training, cognitive ability assessment, and neurorehabilitation. Current technologies typically collect multi-channel EEG signals from users using devices such as EEG helmets, and then perform filtering, frequency domain analysis, or feature extraction on the signals to calculate single indicators such as focus and relaxation levels, which are used to drive simple training feedback or interactive control. Some systems set fixed thresholds, triggering corresponding feedback when EEG indicators reach predetermined conditions to guide the user's attention or state.

[0003] In existing applications, EEG training systems mostly employ general training models, lacking the ability to differentiate and adapt to the neural state changes of different users or the same user at different training stages. The training process is difficult to dynamically adjust based on real-time neural states. Existing technologies typically rely on single EEG features for judgment, lacking collaborative analysis of multiple neural features such as attention, relaxation level, and cognitive load. This makes it difficult to accurately characterize complex neural state structures, limiting the precision of training decisions. While some systems introduce reinforcement learning methods to regulate training tasks, they often employ static state modeling or train complete models for individual users, lacking an effective synergy mechanism between group learning and personalized learning. This results in models failing to continuously evolve with overall user data, and personalized training strategies easily becoming outdated after long-term use.

[0004] Therefore, how to provide a reinforcement learning-based EEG data analysis method is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a reinforcement learning-based EEG data analysis method. This invention comprehensively utilizes multi-channel EEG feature extraction, scalable prototype learning vector quantization networks, and reinforcement learning decision-making mechanisms to achieve the co-evolution of group learning and personalized learning, and has the advantages of strong adaptability, high degree of personalization, and good long-term training stability.

[0006] A reinforcement learning-based EEG data analysis method according to an embodiment of the present invention includes the following steps: S1. Obtain the EEG time-series signals of each brain region, perform preprocessing, and generate EEG segment sequences; S2. Perform frequency domain feature calculation on the EEG segment sequence to generate an EEG feature vector sequence; S3. Construct a learning vector quantization network with a growable prototype structure, initialize the prototype vector set, category label set, usage count set and distance statistics set, calculate the matching distance based on EEG feature vectors and output neural state codes; S4. During the neural state code generation process, prototype addition, prototype merging and prototype deletion operations are performed on the prototype vector set according to the preset growth criteria to update the learning vector quantization network structure and generate the updated neural state code. S5. Construct reinforcement learning states based on the updated neural state codes and training task performance parameters, determine training task actions based on reinforcement learning states, and execute training tasks. S6. Construct the SARSA population base action value function and update it based on reinforcement learning state and training task action execution to form a population base model; S7. Generate a user-personalized action value function based on the group base model, perform SARSA update on the user-personalized action value function, and form a user-personalized model.

[0007] Optionally, the learning vector quantization network structure in S3 specifically includes: The input layer receives EEG feature vectors; The competition layer consists of several prototype units, which together form a prototype vector set. Each prototype unit stores a prototype vector and is associated with a category label. The competition layer performs distance calculations based on the EEG feature vector and the prototype vector set to determine the matching prototype unit. The prototype management structure is set in the competition layer and connected to the prototype vector set. It performs prototype unit addition, prototype unit merging, and prototype unit deletion operations on the prototype vector set. During the initialization phase, the prototype vector set and the category label set are initialized, and the usage count parameter and distance statistics parameter corresponding to each prototype unit are set to preset initial values. After inputting the EEG feature vector into the competition layer and determining the matching prototype unit, the usage count parameter and distance statistics parameter corresponding to the matching prototype unit are written into the prototype management structure. The output layer is connected to the competition layer, receives the category label corresponding to the matching prototype unit, and outputs the neural state code corresponding to the category label.

[0008] Optionally, the growth criterion in S4 specifically includes: During the neural status code generation process, the distances between the current EEG feature vector and each prototype vector in the prototype vector set are sorted, and the first prototype vector corresponding to the smallest distance and the second prototype vector corresponding to the second smallest distance in the distance sorting results are determined. Within a preset time window, the index changes of the first prototype vector and the second prototype vector in the distance sorting result are recorded. When the first prototype vector and the second prototype vector are swapped in the sorting result, the degree of distinction is determined to be low. When the first prototype vector and the second prototype vector remain unchanged in the sorting result, the degree of distinction is determined to be high. Within a preset time window, the sign change of the reinforcement learning reward function output value corresponding to the neural state code is recorded. When the sign of the reward function output value changes, the stability is determined to be low. When the sign of the reward function output value remains consistent, the stability is determined to be high. Within a preset long time window, the occurrence of each prototype vector becoming a matching prototype vector is recorded. When a prototype vector does not become a matching prototype vector within the time window, the activity level is determined to be low. When a prototype vector becomes a matching prototype vector at least once within the time window, the activity level is determined to be high. Based on the combination of differentiation, stability and activity, a unique prototype structure adjustment operation type is determined. The prototype structure adjustment operation type is limited to prototype addition, prototype merging, prototype deletion or keeping the prototype structure unchanged. Adjust the operation type according to the prototype structure, perform the corresponding prototype addition operation, prototype merging operation, prototype deletion operation or keep the prototype structure unchanged operation on the prototype vector set, and output the updated neural state code.

[0009] Optionally, the prototype addition, prototype merging, and prototype deletion operations in S4 specifically include: Operation condition 1: When the activity level is determined to be low, the prototype deletion operation is executed, and the prototype vector with the low activity level is deleted. The prototype vector is removed from the prototype vector set and no longer participates in the distance calculation and neural state code generation process. Operation condition 2: When the discrimination level is judged to be low and the stability level is judged to be low, the prototype addition operation is determined to be executed. The current EEG feature vector is used as the initial value of the new prototype vector. The EEG feature vector is written into the prototype vector set to form the new prototype vector. The new prototype vector is assigned a category label that is consistent with the current matching prototype vector. The usage count parameter and distance statistics parameter corresponding to the new prototype vector are initialized to 0. Operation condition 3: When the discrimination level is judged to be low and the stability level is judged to be high, the prototype merging operation is determined to be executed. The prototype vector pairs with the same category label are selected to perform the merging operation. The merging operation generates the merged prototype vector by performing a weighted average on the corresponding prototype vector pairs using the counting parameter, and replaces the prototype vector pairs with the merged prototype vector. The prototype addition operation, prototype merging operation, and prototype deletion operation are judged in the order of operation condition 1 to operation condition 3. The corresponding operation is executed when the operation condition that is judged first is met, and the other operation conditions are not executed. The prototype vector set structure is kept unchanged, and neural status code generation continues to be performed based on the prototype vector set.

[0010] Optionally, S5 specifically includes: S51. The updated neural state code and the training task performance parameters are combined under the same time index to form a reinforcement learning state. The reinforcement learning state is composed of the discrete neural state code representing the current neural state and the training task performance parameters representing the completion status of the training task. S52. Input the reinforcement learning state as the current state into the reinforcement learning process, and determine a training task action from the training task action set based on the reinforcement learning state. S53. Execute the training task action. After the training task is completed, obtain the corresponding training task performance parameters and associate and store the training task performance parameters with the reinforcement learning state and training task action.

[0011] Optionally, S6 specifically includes: S61. Construct a SARSA population base action value function table based on reinforcement learning state and training task action. The action value function table uses the combination of reinforcement learning state and training task action as index, and initializes the action value of each index item. S62. After each training task action is completed, obtain the corresponding reward function output value, the reinforcement learning state at the next moment, and the next training task action. Update the corresponding action value in the action value function table according to the temporal difference update method of the SARSA algorithm. S63. After completing the preset rounds of training task interaction, the updated action value function table is stored to form a group base model.

[0012] Optionally, S7 specifically includes: S71. After the group base model is formed, the user personalized action value function table is generated based on the group base model. S72. Divide the training process into sequences, and in the initial stage, determine the training task actions based solely on the action value function corresponding to the group base model. S73. During the transition phase, based on the update of the user-personalized action value function table in the reinforcement learning state space, the action value functions corresponding to the group base model and the user-personalized model are used alternately to determine the training task actions. S74. In the personalization stage, the training task actions are determined solely based on the user's personalized action value function table. The user's personalized action value function table is then updated using the temporal differential update method of the SARSA algorithm to form a user-personalized model.

[0013] Optionally, the user personalization model described in S7 specifically includes: During the continuous updating of the group base model, a set of user-personalized action value offset tables is maintained for each user. During the training task interaction, the training task action is determined by the user's personalized action value offset table and the action value function table corresponding to the group base model. The action value of the training task action is obtained by superimposing the action value of the group base model and the corresponding user's personalized action value offset. When updating the user's personalized action value offset, the SARSA algorithm's temporal differential update method is used. The user's personalized action value offset table is updated only based on the corresponding user's reinforcement learning state, training task actions, and reward function output value, while the group base model action value function table is continuously updated. During the update of user-personalized action value offset, a forgetting factor mechanism is introduced to perform decay processing on the user-personalized action value offset corresponding to reinforcement learning states and training task actions that have not been accessed for more than a preset time interval.

[0014] Optionally, the partitioning sequence mentioned in S72 specifically includes: The initialization phase is the time when the process is initiated. During the training task interaction, when the number of non-zero values ​​of the offset corresponding to the state-action in the user's personalized action value offset table reaches a threshold, the training process is determined to enter the transition phase. During the training task interaction, when the number of non-zero offsets in the user's personalized action value offset table in the reinforcement learning state space reaches a stable condition, it is determined that the user personalization model has formed a stable personalized action value structure.

[0015] The beneficial effects of this invention are: (1) This invention constructs a learning vector quantization network with a growable prototype structure to dynamically model EEG feature vectors. During the training process, the prototype structure can be automatically adjusted according to the changes in the distribution of EEG features, so as to achieve adaptive characterization of EEG information in multiple brain regions and multiple frequency bands. Compared with the existing EEG analysis method that uses a fixed model structure, it improves the stability and discrimination ability of neural state modeling, which is conducive to accurately generating neural state codes that reflect the user's real neural state.

[0016] (2) The present invention constructs reinforcement learning state by combining neural state codes and training task performance parameters, and makes training task decisions based on the collaborative mechanism of group base model and user personalized model. While ensuring the continuous accumulation of group experience, it realizes the gradual adaptation of training strategy to individual differences, avoids the problem of separation between personalized model and group model in the prior art, and enables training decisions to maintain consistency and pertinence in different users and different training stages.

[0017] (3) In the process of personalized training, this invention introduces a personalized modeling method based on action value offset and combined with forgetting factor mechanism, so that the user's personalized characteristics are fully reflected in the short term, and gradually return to the group benchmark when the user is not used for a long time or when the behavior changes. This effectively balances the relationship between personalized adaptability and long-term model evolution, and improves the stability and practicality of the training system in the long-term operation process. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Fig. 1 This is a flowchart of an EEG data analysis method based on reinforcement learning proposed in this invention; Fig. 2 This is a diagram of the growable learning vector quantization network structure of an EEG data analysis method based on reinforcement learning proposed in this invention. Fig. 3 This is a diagram illustrating the personalized shift-forgetting mechanism coupling structure of an EEG data analysis method based on reinforcement learning proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figs. 1-3 A reinforcement learning-based EEG data analysis method includes the following steps: S1. Obtain the EEG time-series signals of each brain region, perform preprocessing, and generate EEG segment sequences; S2. Perform frequency domain feature calculation on the EEG segment sequence to generate an EEG feature vector sequence; S3. Construct a learning vector quantization network with a growable prototype structure, initialize the prototype vector set, category label set, usage count set and distance statistics set, calculate the matching distance based on EEG feature vectors and output neural state codes; S4. During the neural state code generation process, prototype addition, prototype merging and prototype deletion operations are performed on the prototype vector set according to the preset growth criteria to update the learning vector quantization network structure and generate the updated neural state code. S5. Construct reinforcement learning states based on the updated neural state codes and training task performance parameters, determine training task actions based on reinforcement learning states, and execute training tasks. S6. Construct the SARSA population base action value function and update it based on reinforcement learning state and training task action execution to form a population base model; S7. Generate a user-personalized action value function based on the group base model, perform SARSA update on the user-personalized action value function, and form a user-personalized model.

[0021] In this embodiment, S1 specifically includes: S11. Collect multi-channel EEG timing signals according to the preset EEG electrode layout scheme. The EEG electrode layout scheme is based on the international 10-20 electrode system, which maps electrode channels to predetermined brain regions. Each brain region corresponds to at least one fixed set of electrode channels. The correspondence between brain regions and electrode channels is set before the acquisition. During the acquisition process, each electrode channel synchronously records EEG signals at a uniform sampling frequency to form multi-channel EEG timing signal data. S12. Perform Butterworth bandpass filtering, notch filtering, independent component analysis artifact removal, and Z-score normalization on the EEG time series signal sets of each brain region. The preprocessing parameters are set before the processing begins and remain consistent during the processing. S13. Perform sliding window segmentation on the preprocessed EEG time sequence signal set of each brain region. The sliding window segmentation continuously divides the EEG time sequence signal according to the preset time window length and time step to generate EEG segment sequence.

[0022] In this embodiment, S2 specifically includes: S21. Perform fast Fourier transform on the EEG segment sequence in chronological order to convert the EEG segments of each brain region from time domain representation to frequency domain representation. The frequency domain representation includes the corresponding frequency component and amplitude component. Use uniform transformation parameters during the frequency domain transformation. S22. Based on the frequency domain representation of EEG segments of each brain region, calculate the power spectral density characteristics of each brain region in a preset frequency band, including the θ band, α band and β band; perform statistical calculations on the power spectral density of each brain region in each preset frequency band to generate the frequency domain feature vector of the corresponding brain region. S23. The frequency domain feature vectors corresponding to each brain region within the same time window are spliced ​​together in a preset order to form EEG feature vectors; the EEG feature vectors are arranged in chronological order to generate an EEG feature vector sequence.

[0023] In this embodiment, the learning vector quantization network structure in S3 specifically includes: The input layer receives EEG feature vectors; The competition layer consists of several prototype units, which together form a prototype vector set. Each prototype unit stores a prototype vector and is associated with a category label. The competition layer performs distance calculations based on the EEG feature vector and the prototype vector set to determine the matching prototype unit. The prototype management structure is set in the competition layer and connected to the prototype vector set. It performs prototype unit addition, prototype unit merging, and prototype unit deletion operations on the prototype vector set. During the initialization phase, the prototype vector set and the category label set are initialized, and the usage count parameter and distance statistics parameter corresponding to each prototype unit are set to preset initial values. After inputting the EEG feature vector into the competition layer and determining the matching prototype unit, the usage count parameter and distance statistics parameter corresponding to the matching prototype unit are written into the prototype management structure. The output layer is connected to the competition layer, receives the category label corresponding to the matching prototype unit, and outputs the neural state code corresponding to the category label. The category label is a combined label. In this embodiment, S3 specifically includes: The learning rate parameter is set to 0.05, and the distance calculation adopts the Euclidean distance metric. Each prototype unit is assigned a usage count parameter, which is set to 0 during the initialization phase. Each prototype unit is assigned a distance statistics parameter in the distance statistics storage unit, and all distance statistics parameters are set to 0 during the initialization phase. After inputting EEG feature vectors into the learning vector quantization network, the Euclidean distance between each EEG feature vector and each prototype vector in the prototype vector set is calculated to determine the matching prototype unit. The prototype vectors corresponding to the matching prototype units are updated according to the learning vector quantization update rules, and the update magnitude is controlled by the learning rate parameter. At the same time, the usage count parameter corresponding to the matching prototype unit is incremented by 1.

[0024] In this embodiment, the growth criterion in S4 specifically includes: During the neural status code generation process, the distances between the current EEG feature vector and each prototype vector in the prototype vector set are sorted, and the first prototype vector corresponding to the smallest distance and the second prototype vector corresponding to the second smallest distance in the distance sorting results are determined. Within a preset time window, the index changes of the first prototype vector and the second prototype vector in the distance sorting result are recorded. When the first prototype vector and the second prototype vector are swapped in the sorting result, the degree of discrimination is determined to be low. When the first prototype vector and the second prototype vector remain unchanged in the sorting result, the degree of discrimination is determined to be high. The preset time window is 20 consecutive neural status code generation cycles. Within a preset time window, the sign change of the reinforcement learning reward function output value corresponding to the neural state code is recorded. When the sign of the reward function output value changes, the stability is determined to be low. When the sign of the reward function output value remains consistent, the stability is determined to be high. Within a preset long time window, the occurrence of each prototype vector becoming a matching prototype vector is recorded. When a prototype vector does not become a matching prototype vector within the time window, the activity level is determined to be low. When a prototype vector becomes a matching prototype vector at least once within the time window, the activity level is determined to be high. The preset long time window length is set to 200 consecutive neural state code generation cycles of EEG feature vectors. Based on the combination of differentiation, stability and activity, a unique prototype structure adjustment operation type is determined. The prototype structure adjustment operation type is limited to prototype addition, prototype merging, prototype deletion or keeping the prototype structure unchanged. Adjust the operation type according to the prototype structure, perform the corresponding prototype addition operation, prototype merging operation, prototype deletion operation or keep the prototype structure unchanged operation on the prototype vector set, and output the updated neural state code.

[0025] In this embodiment, the prototype addition, prototype merging, and prototype deletion operations in S4 specifically include: Operation condition 1: When the activity level is determined to be low, the prototype deletion operation is executed, and the prototype vector with the low activity level is deleted. The prototype vector is removed from the prototype vector set and no longer participates in the distance calculation and neural state code generation process. Operation condition 2: When the discrimination level is judged to be low and the stability level is judged to be low, the prototype addition operation is determined to be executed. The current EEG feature vector is used as the initial value of the new prototype vector. The EEG feature vector is written into the prototype vector set to form the new prototype vector. The new prototype vector is assigned a category label that is consistent with the current matching prototype vector. The usage count parameter and distance statistics parameter corresponding to the new prototype vector are initialized to 0. Operation condition 3: When the discrimination level is judged to be low and the stability level is judged to be high, the prototype merging operation is determined to be executed. The prototype vector pairs with the same category label are selected to perform the merging operation. The merging operation generates a merged prototype vector by averaging the corresponding usage count parameters of the prototype vector pairs and replacing the prototype vector pairs with the merged prototype vector. The prototype addition operation, prototype merging operation, and prototype deletion operation are judged in the order of operation condition 1 to operation condition 3. The corresponding operation is executed when the operation condition that is judged first is met, and the other operation conditions are not executed. The prototype vector set structure is kept unchanged, and neural status code generation continues to be performed based on the prototype vector set.

[0026] In this embodiment, S5 specifically includes: S51. The updated neural state code and the training task performance parameters are combined under the same time index to form a reinforcement learning state. The reinforcement learning state is composed of the discrete neural state code representing the current neural state and the training task performance parameters representing the completion status of the training task. S52. Input the reinforcement learning state as the current state into the reinforcement learning process, and determine a training task action from the training task action set based on the reinforcement learning state. S53. Execute the training task action. After the training task is completed, obtain the corresponding training task performance parameters and associate and store the training task performance parameters with the reinforcement learning state and training task action.

[0027] In this embodiment, S6 specifically includes: S61. Construct a SARSA population base action value function table based on reinforcement learning state and training task action. The action value function table uses the combination of reinforcement learning state and training task action as index, and initializes the action value of each index item. S62. After each training task action is completed, obtain the corresponding reward function output value, the reinforcement learning state at the next moment, and the next training task action. Update the corresponding action value in the action value function table according to the temporal difference update method of the SARSA algorithm. S63. After completing the preset rounds of training task interaction, the updated action value function table is stored to form a group base model.

[0028] In this embodiment, S7 specifically includes: S71. After the group base model is formed, the user personalized action value function table is generated based on the group base model. S72. Divide the training process into sequences, and in the initial stage, determine the training task actions based solely on the action value function corresponding to the group base model. S73. During the transition phase, based on the update of the user-personalized action value function table in the reinforcement learning state space, the action value functions corresponding to the group base model and the user-personalized model are used alternately to determine the training task action; when the current reinforcement learning state appears an odd number of times, the training task action is determined based on the action value function corresponding to the group base model; when the current reinforcement learning state appears an even number of times, the training task action is determined based on the user-personalized action value function table. S74. In the personalization stage, the training task actions are determined solely based on the user's personalized action value function table. The user's personalized action value function table is then updated using the temporal differential update method of the SARSA algorithm to form a user-personalized model.

[0029] In this embodiment, the user personalization model mentioned in S7 specifically includes: During the continuous updating of the group base model, a set of user-personalized action value offset tables is maintained for each user. During the training task interaction, the training task action is determined by the user's personalized action value offset table and the action value function table corresponding to the group base model. The action value of the training task action is obtained by superimposing the action value of the group base model and the corresponding user's personalized action value offset. When updating the user's personalized action value offset, the SARSA algorithm's temporal differential update method is used. The user's personalized action value offset table is updated only based on the corresponding user's reinforcement learning state, training task actions, and reward function output value, while the group base model action value function table is continuously updated. During the update of user-personalized action value offset, a forgetting factor mechanism is introduced to perform decay processing on the user-personalized action value offset corresponding to reinforcement learning states and training task actions that have not been accessed for more than a preset time interval.

[0030] In this embodiment, the partitioning sequence mentioned in S72 specifically includes: The initialization phase is the time when the process is initiated. During the training task interaction, when the number of non-zero offset values ​​corresponding to states and actions in the user's personalized action value offset table reaches a threshold, the training process is determined to enter the transition phase; the threshold is set at 10% of the total number of all states and actions. During the training task interaction, when the number of non-zero offsets in the user's personalized action value offset table in the reinforcement learning state space reaches a stable condition, it is determined that the user personalization model has formed a stable personalized action value structure; the stable condition is that the change in the number of non-zero offsets is less than 5%.

[0031] The parameter settings are as follows: The learning rate parameter for reinforcement learning is set to a fixed value of 0.1, and the discount factor parameter is set to a fixed value of 0.9. During the training process, it is used to execute the ε-greedy policy in the action selection phase, and the exploration rate is set to 0.1. During the training of the group base model, the initial action value corresponding to all reinforcement learning states and training task actions in the action value function table is uniformly set to zero; during the construction of the user personalized model, the initial value of the offset corresponding to all states and actions in the user personalized action value offset table is uniformly set to zero. The reinforcement learning reward function is calculated based on the performance parameters of the training task, which include task completion accuracy, average reaction time, and operational stability. Task completion accuracy represents the proportion of training tasks correctly completed within the current training cycle; average reaction time represents the average time it takes for a user to effectively respond to training task stimuli within the current training cycle. Reaction time is linearly normalized by setting minimum and maximum reaction time thresholds, limiting the normalized reaction time value to between zero and one; operational stability index represents the fluctuation of a user's continuous task response time within the current training cycle. It is obtained by calculating the variance of the reaction time series and performing normalization. The operational stability index value is limited to between zero and one, with a larger value indicating more significant operational fluctuations. Based on the three normalized training task performance parameters mentioned above, a comprehensive training performance score is constructed. The comprehensive training performance score is calculated using a weighted summation method, where the weight corresponding to the task completion accuracy is set to 0.5, the weight corresponding to the reverse value of the normalized reaction time is set to 0.3, and the weight corresponding to the reverse value of the operational stability index is set to 0.2. By summing the three weighted results, the comprehensive training performance score corresponding to the current training cycle is obtained.

[0032] The output value of the reinforcement learning reward function is obtained by subtracting the combined training performance scores of two adjacent training cycles; Maintain a recent access timestamp for each user offset index, with the timestamp unit set to seconds and the value taken from Unix time; forgetting uses exponential decay, with a half-life of 90 days.

[0033] In this embodiment, the output of the learning vector quantization network is a discrete combination of neural state labels. After receiving the EEG feature vector, the learning vector quantization network determines the prototype unit that is closest to the current EEG feature by matching the distance with the prototype vector set, and outputs the category label associated with the prototype unit. The category label represents the state value under different neural dimensions in the form of discrete labels. The discrete labels of each neural dimension are combined in a predetermined order to form a neural state code.

[0034] The neural state code consists of multiple discrete state labels, each of which corresponds to neural state dimensions such as attention level, relaxation level, and cognitive load level, such as "high focus", "medium focus", "low focus", "high relaxation", "low relaxation", "high load", "medium load", and "low load". The discrete labels are combined to form a unique neural state code.

[0035] In subsequent decision-making, neural state codes are used as indexes to input into the rule table, triggering corresponding training task actions based on the neural state codes. For different neural state codes, the corresponding training task adjustment strategy is set in the state-action mapping table. High focus-low relaxation-medium load is judged as an "effective challenge state," maintaining the current task difficulty; low focus-high relaxation-low load is judged as a "lazy state," increasing the task difficulty and adding interference items. The rule table is pre-set and can be updated according to the reward function.

[0036] Example 1: To verify the feasibility of this invention in practice, it was applied to a cognitive training scenario based on electroencephalogram (EEG) signals. In this scenario, users participate in continuous cognitive training activities by wearing an EEG acquisition device. The training content includes various cognitive behaviors such as attention maintenance, information processing, and decision-making. Existing technologies generally suffer from problems such as a single training strategy, difficulty in reflecting individual differences, and a gradual decline in training effectiveness over extended periods, making it difficult to meet the adaptability and stability requirements of long-term cognitive training.

[0037] In this embodiment, time-series EEG signals from multiple brain regions of the user are acquired, and the acquired signals are filtered, artifact-free, and normalized to generate a stable sequence of EEG segments. Frequency domain feature calculations are performed on the EEG segment sequences to extract power spectral features from multiple frequency bands, and an EEG feature vector sequence is constructed. By introducing a learning vector quantization network with a growable prototype structure, the EEG feature vectors are matched and modeled, outputting a neural state code formed by a combination of multidimensional discrete labels to characterize the user's current comprehensive neural state.

[0038] During training, the learning vector quantization network dynamically adjusts its prototype structure based on changes in EEG feature distribution, enabling neural state modeling to continuously evolve with changes in the user's state. Reinforcement learning states are constructed based on neural state codes and training task performance parameters. The reinforcement learning process employs a collaborative approach of a group-based pedestal model and a personalized model. The group-based pedestal model is continuously updated during training with multiple users, forming stable group experience. The personalized model records individual characteristics in the form of action value offsets and gradually adjusts historical personalized information through a time-driven forgetting mechanism, avoiding policy lag caused by long-term inactivity.

[0039] In practical applications, when users begin training, the system primarily relies on the group-based model for training task scheduling, ensuring the fundamental rationality of the training strategy. As training continues, personalized action value biases gradually form and stabilize, and the training strategy gradually shifts towards individual characteristics, making the training content more aligned with the user's current cognitive state. If a user does not participate in training for an extended period, the forgetting mechanism gradually weakens the personalized biases, allowing for a rapid return to the latest group experience base upon re-engagement, preventing a decline in training efficiency due to the failure of historical strategies.

[0040] To verify the beneficial effects of this invention, the method of this invention was compared with traditional EEG training methods under the same training conditions. During the training process, the corresponding reward function output value was obtained for each training task. The individual matching score was calculated by statistically analyzing the average output value of the reward function of the user's personalized model during continuous training and performing normalized difference calculation with the average output value of the reward function of the group base model under the same reinforcement learning state. This score reflects the degree of fit of the training strategy to the individual. The long-term effectiveness index was calculated by performing normalized weighted average calculation of the mean and variance of the reward function output value over a long period. This index characterizes the stability of the training strategy during long-term use.

[0041] Table 1: Results of EEG Training

[0042] As can be seen from the numerical comparison in Table 1, after adopting the method of this invention, all core indicators show a continuous and stable upward trend. The accuracy of neural state recognition increased from 0.62 to 0.81, indicating that the growable prototype structure has a higher discriminative ability in multi-brain region EEG feature modeling. The training task completion rate increased from 0.58 to 0.83, indicating that the reinforcement learning strategy, after combining neural state codes, can better match the training difficulty with the user's current cognitive state. The average response consistency increased from 0.60 to 0.85, reflecting a reduction in operational fluctuations and enhanced behavioral stability during training.

[0043] The number of policy convergence steps decreased from 180 to 80, indicating that the group-based pedestal model and personalized bias mechanism can accelerate the formation of decision-making strategies. The individual matching score improved from 0.45 to 0.86, demonstrating that the system gradually develops decision preferences tailored to individual differences during training. The long-term effectiveness index reached 0.88 in the personalization stage, indicating that the personalized strategy did not degrade with prolonged use after the introduction of the time-driven forgetting mechanism. Overall, the data shows that this invention has advantages in improving training accuracy, adapting to individual differences, and maintaining long-term stability.

[0044] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for analyzing electroencephalogram (EEG) data based on reinforcement learning, characterized in that, Includes the following steps: S1. Obtain the EEG time-series signals of each brain region, perform preprocessing, and generate EEG segment sequences; S2. Perform frequency domain feature calculation on the EEG segment sequence to generate an EEG feature vector sequence; S3. Construct a learning vector quantization network with a growable prototype structure, initialize the prototype vector set, category label set, usage count set and distance statistics set, calculate the matching distance based on EEG feature vectors and output neural state codes; S4. During the neural state code generation process, prototype addition, prototype merging and prototype deletion operations are performed on the prototype vector set according to the preset growth criteria to update the learning vector quantization network structure and generate the updated neural state code. S5. Construct reinforcement learning states based on the updated neural state codes and training task performance parameters, determine training task actions based on reinforcement learning states, and execute training tasks. S6. Construct the SARSA population base action value function and update it based on reinforcement learning state and training task action execution to form a population base model; S7. Generate a user-personalized action value function based on the group base model, perform SARSA update on the user-personalized action value function, and form a user-personalized model.

2. The EEG data analysis method based on reinforcement learning according to claim 1, characterized in that, The learning vector quantization network structure in S3 specifically includes: The input layer receives EEG feature vectors; The competition layer consists of several prototype units, which together form a prototype vector set. Each prototype unit stores a prototype vector and is associated with a category label. The competition layer performs distance calculations based on the EEG feature vector and the prototype vector set to determine the matching prototype unit. The prototype management structure is set in the competition layer and connected to the prototype vector set. It performs prototype unit addition, prototype unit merging, and prototype unit deletion operations on the prototype vector set. During the initialization phase, the prototype vector set and the category label set are initialized, and the usage count parameter and distance statistics parameter corresponding to each prototype unit are set to preset initial values. After inputting the EEG feature vector into the competition layer and determining the matching prototype unit, the usage count parameter and distance statistics parameter corresponding to the matching prototype unit are written into the prototype management structure. The output layer is connected to the competition layer, receives the category label corresponding to the matching prototype unit, and outputs the neural state code corresponding to the category label.

3. The EEG data analysis method based on reinforcement learning according to claim 2, characterized in that, The growth criterion in S4 specifically includes: During the neural status code generation process, the distances between the current EEG feature vector and each prototype vector in the prototype vector set are sorted, and the first prototype vector corresponding to the smallest distance and the second prototype vector corresponding to the second smallest distance in the distance sorting results are determined. Within a preset time window, the index changes of the first prototype vector and the second prototype vector in the distance sorting result are recorded. When the first prototype vector and the second prototype vector are swapped in the sorting result, the degree of distinction is determined to be low. When the first prototype vector and the second prototype vector remain unchanged in the sorting result, the degree of distinction is determined to be high. Within a preset time window, the sign change of the reinforcement learning reward function output value corresponding to the neural state code is recorded. When the sign of the reward function output value changes, the stability is determined to be low. When the sign of the reward function output value remains consistent, the stability is determined to be high. Within a preset long time window, the occurrence of each prototype vector becoming a matching prototype vector is recorded. When a prototype vector does not become a matching prototype vector within the time window, the activity level is determined to be low. When a prototype vector becomes a matching prototype vector at least once within the time window, the activity level is determined to be high. Based on the combination of differentiation, stability and activity, a unique prototype structure adjustment operation type is determined. The prototype structure adjustment operation type is limited to prototype addition, prototype merging, prototype deletion or keeping the prototype structure unchanged. Adjust the operation type according to the prototype structure, perform the corresponding prototype addition operation, prototype merging operation, prototype deletion operation or keep the prototype structure unchanged operation on the prototype vector set, and output the updated neural state code.

4. The EEG data analysis method based on reinforcement learning according to claim 3, characterized in that, The prototype addition, prototype merging, and prototype deletion operations in S4 specifically include: Operation condition 1: When the activity level is determined to be low, the prototype deletion operation is executed, and the prototype vector with the low activity level is deleted. The prototype vector is removed from the prototype vector set and no longer participates in the distance calculation and neural state code generation process. Operation condition 2: When the discrimination level is judged to be low and the stability level is judged to be low, the prototype addition operation is determined to be executed. The current EEG feature vector is used as the initial value of the new prototype vector. The EEG feature vector is written into the prototype vector set to form the new prototype vector. The new prototype vector is assigned a category label that is consistent with the current matching prototype vector. The usage count parameter and distance statistics parameter corresponding to the new prototype vector are initialized to 0. Operation condition 3: When the discrimination level is judged to be low and the stability level is judged to be high, the prototype merging operation is determined to be executed. The prototype vector pairs with the same category label are selected to perform the merging operation. The merging operation generates the merged prototype vector by performing a weighted average on the corresponding prototype vector pairs using the counting parameter, and replaces the prototype vector pairs with the merged prototype vector. The prototype addition operation, prototype merging operation, and prototype deletion operation are judged in the order of operation condition 1 to operation condition 3. The corresponding operation is executed when the operation condition that is judged first is met, and the other operation conditions are not executed. The prototype vector set structure is kept unchanged, and neural status code generation continues to be performed based on the prototype vector set.

5. The EEG data analysis method based on reinforcement learning according to claim 4, characterized in that, S5 specifically includes: S51. The updated neural state code and the training task performance parameters are combined under the same time index to form a reinforcement learning state. The reinforcement learning state is composed of the discrete neural state code representing the current neural state and the training task performance parameters representing the completion status of the training task. S52. Input the reinforcement learning state as the current state into the reinforcement learning process, and determine a training task action from the training task action set based on the reinforcement learning state. S53. Execute the training task action. After the training task is completed, obtain the corresponding training task performance parameters and associate and store the training task performance parameters with the reinforcement learning state and training task action.

6. The EEG data analysis method based on reinforcement learning according to claim 5, characterized in that, S6 specifically includes: S61. Construct a SARSA population base action value function table based on reinforcement learning state and training task action. The action value function table uses the combination of reinforcement learning state and training task action as index, and initializes the action value of each index item. S62. After each training task action is completed, obtain the corresponding reward function output value, the reinforcement learning state at the next moment, and the next training task action. Update the corresponding action value in the action value function table according to the temporal difference update method of the SARSA algorithm. S63. After completing the preset rounds of training task interaction, the updated action value function table is stored to form a group base model.

7. The EEG data analysis method based on reinforcement learning according to claim 6, characterized in that, Specifically, S7 includes: S71. After the group base model is formed, the user personalized action value function table is generated based on the group base model. S72. Divide the training process into sequences, and in the initial stage, determine the training task actions based solely on the action value function corresponding to the group base model. S73. During the transition phase, based on the update of the user-personalized action value function table in the reinforcement learning state space, the action value functions corresponding to the group base model and the user-personalized model are used alternately to determine the training task actions. S74. In the personalization stage, the training task actions are determined solely based on the user's personalized action value function table. The user's personalized action value function table is then updated using the temporal differential update method of the SARSA algorithm to form a user-personalized model.

8. The EEG data analysis method based on reinforcement learning according to claim 7, characterized in that, The user personalization model mentioned in S7 specifically includes: During the continuous updating of the group base model, a set of user-personalized action value offset tables is maintained for each user. During the training task interaction, the training task action is determined by the user's personalized action value offset table and the action value function table corresponding to the group base model. The action value of the training task action is obtained by superimposing the action value of the group base model and the corresponding user's personalized action value offset. When updating the user's personalized action value offset, the SARSA algorithm's temporal differential update method is used. The user's personalized action value offset table is updated only based on the corresponding user's reinforcement learning state, training task actions, and reward function output value, while the group base model action value function table is continuously updated. During the update of user-personalized action value offset, a forgetting factor mechanism is introduced to perform decay processing on the user-personalized action value offset corresponding to reinforcement learning states and training task actions that have not been accessed for more than a preset time interval.

9. The EEG data analysis method based on reinforcement learning according to claim 8, characterized in that, The partitioning sequence mentioned in S72 specifically includes: The initialization phase is the time when the process is initiated. During the training task interaction, when the number of non-zero values ​​of the offset corresponding to the state-action in the user's personalized action value offset table reaches a threshold, the training process is determined to enter the transition phase. During the training task interaction, when the number of non-zero offsets in the user's personalized action value offset table in the reinforcement learning state space reaches a stable condition, it is determined that the user personalization model has formed a stable personalized action value structure.

Images