Brain-computer cooperative control optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of brain-computer cooperative control, in particular to a brain-computer cooperative control optimization method based on deep reinforcement learning, and the method comprises the steps: obtaining an electroencephalogram signal and a current task state of a user; based on the electroencephalogram signal and the current task state, determining an action selection strategy by using a deep reinforcement learning framework; executing an action and acquiring environment feedback according to an action selection strategy; calculating a reward value and uncertainty estimation based on environmental feedback; updating an action strategy space based on the reward value and the uncertainty estimation; the optimized action strategy is output and used for controlling the intelligent equipment to execute the task, and through innovative application of deep reinforcement learning, the performance and adaptability of the brain-computer interface system are remarkably improved. According to the method, the control strategy can be learned and optimized online in real time, and the response speed and accuracy of the system to the user intention are greatly improved. The method is of great significance to tasks needing quick response, such as intelligent artificial limb control or first-aid robot operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain-computer collaborative control, more specifically, to a brain-computer collaborative control optimization method based on deep reinforcement learning. BACKGROUND

[0002] In recent years, brain-computer interface technology as a revolutionary human-computer interaction method has shown great application potential in medical rehabilitation, auxiliary device control and other fields. Traditional brain-computer interface systems mainly rely on simple signal processing and classification algorithms, which are difficult to cope with complex and variable actual application scenarios. With the rapid development of deep learning technology, researchers have begun to try to apply deep neural networks to electroencephalogram decoding, which has improved the performance of the system to some extent. However, these methods often require a large amount of labeled data for training, which is difficult to adapt to individual differences and dynamic task requirements.

[0003] On the other hand, reinforcement learning as a method that can continuously optimize decisions through interaction with the environment has achieved remarkable results in the fields of robot control and game AI in recent years. Some researchers have tried to introduce reinforcement learning into brain-computer interface systems to achieve adaptive control strategy optimization. However, these methods usually face the problems of low learning efficiency and slow convergence speed, which makes it difficult to meet the needs of real-time interaction. In addition, due to the highly non-stationary and individual differences of electroencephalogram, how to achieve rapid adaptation while ensuring system stability is still a difficult problem to be solved.

[0004] There are still some other problems in existing brain-computer collaborative control methods. For example, most methods lack effective safety mechanisms and are difficult to prevent possible user errors or irrational behavior. At the same time, the lack of knowledge transfer ability also limits the generalization performance of the system, and a time-consuming training process is required every time a new task or new user is faced. In addition, how to balance exploration and utilization in complex and variable actual environments, and how to effectively utilize limited training data, are challenges faced by existing methods. SUMMARY

[0005] In view of the above problems, the present application proposes a brain-computer collaborative control optimization method based on deep reinforcement learning. This method aims to solve the problems of low learning efficiency, poor adaptability, insufficient safety, and weak generalization ability in existing technologies, and provides a new technical approach for efficient, safe, and flexible brain-computer collaborative control.

[0006] To solve the above technical problems, the present application adopts the following technical solutions: The brain-computer collaborative control optimization method based on deep reinforcement learning comprises: The acquisition step comprises: Acquiring the electroencephalogram of the user and the current task state; The processing steps include: Based on the EEG signals and the current task state, a deep reinforcement learning framework is used to determine the action selection strategy. Based on the action selection strategy, execute the action and obtain environmental feedback; Based on the environmental feedback, the reward value and uncertainty estimate are calculated; The update steps include: The action policy space is updated based on the reward value and the uncertainty estimate; Output steps, including: The optimized action strategy is output to control the smart device to perform tasks.

[0007] Preferably, the step of acquiring the user's electroencephalogram (EEG) signal specifically includes: By utilizing the event-related potential paradigm, target signals are detected by measuring the user's electroencephalogram (EEG) signals during the task. The target signal is preprocessed, including noise reduction and smoothing; Using the target signal, an online real-time prediction model is constructed.

[0008] Preferably, the step of constructing the online real-time prediction model specifically includes: Construct an offline support vector machine prediction model; The parameters in the offline support vector machine prediction model are determined by cross-validation. Based on the offline support vector machine prediction model, an online real-time prediction model is obtained.

[0009] Preferably, the step of determining the action selection strategy using a deep reinforcement learning framework specifically includes: Define the state space, including the state of the current task type, the state of the current action, the state of the current target position, the state of the current time, and the state of the current completion status; Define the action space, including the state at the next time step; Set a reward function; The online deep reinforcement learning algorithm DQN is used to learn a strategy for action selection.

[0010] Preferably, the reward function is as follows: in, The reward is for the completion of the current action. The reward is the average time it takes for the current user to complete the task. The penalty term is the average time. The reward is the rate of change over time. The penalty to be selected for the action.

[0011] Preferably, the step of calculating the uncertainty estimate specifically includes: The output of the action value function is calculated based on the difference between each action value. when Greater than the uncertainty threshold At that time, calculate the current time action strategy exploration level. calculate .

[0012] Preferably, the following steps are also included: A model-imagination replay mechanism is introduced. When an error occurs or the task is interrupted during the user's task, the same task is simulated in the simulation environment based on the event-related potential signals collected by the user in the current task. Update the action policy in the simulation environment.

[0013] Preferably, the following steps are also included: Design action command constraint functions to prevent irrational or erroneous user behavior; The action command constraint function is: in, The maximum permissible speed for different actions, This indicates the initial speed set by the user. To control command constraints, For the reward threshold, For noise level, As a reward, The number of samples for the action strategy. This represents the maximum number of samples.

[0014] Preferably, the following steps are also included: Transfer learning is performed during the control process to transfer the learned knowledge to similar tasks; The transfer learning includes: calculating a transfer weight w based on the similarity between the source task and the target task, and using the transfer weight w to transfer parameters, with the value of w varying from 0 to 1.

[0015] Preferably, the step of updating the action strategy space specifically includes: If the reward r calculated based on the event-related potential is greater than 0, the reward r and the action policy value v(a) are merged, and the current action policy in the action policy space is updated. If the calculated reward r is less than 0, add the current training sample to the learning sample set for the current action and wait for the next update; If the calculated reward r equals 0, the current action is not updated; The action value function is calculated using the updated current action policy value.

[0016] The method of the present invention has the following significant technical effects: The method of this invention significantly improves the performance and adaptability of brain-computer interface systems through the innovative application of deep reinforcement learning. First, this method enables online real-time learning and optimization of control strategies, greatly improving the system's response speed and accuracy to user intentions. This is of great significance for tasks requiring rapid response, such as intelligent prosthetic limb control or emergency robot operation. Second, the uncertainty estimation mechanism introduced in this method effectively balances the relationship between exploration and exploitation, enabling the system to continuously adapt to new situations and needs while maintaining stability. This adaptive capability allows the system to better handle the non-stationarity and individual differences in EEG signals.

[0017] Another significant advantage is the safety design of this method. By introducing action command constraint functions, the system can effectively prevent potential user errors, greatly improving reliability and safety in practical applications. This is particularly important for high-risk scenarios such as medical surgery assistance or industrial control. Simultaneously, the transfer learning technique employed in this method allows the system to quickly adapt to new tasks or new users, significantly reducing adaptation time and training costs. This flexibility and generalization ability pave the way for the widespread application of brain-computer interface technology.

[0018] Furthermore, the model visualization and playback mechanism of this method innovatively solves the problem of insufficient training data. By conducting offline learning in a simulation environment, the system can make full use of limited real-world interaction data, greatly improving learning efficiency and policy quality. This feature is particularly valuable for scenarios where frequent real-world training is difficult, such as hazardous environment operations or high-cost experiments.

[0019] In summary, the method proposed in this invention achieves a qualitative leap in brain-computer interface (BCI) collaborative control through the organic combination of innovative technologies such as deep reinforcement learning, security constraints, and knowledge transfer. It not only solves many problems in existing technologies but also opens up new directions for the further development and application of BCI technology. This efficient, safe, and flexible control method is expected to play a vital role in numerous fields such as medical rehabilitation, intelligent manufacturing, and emergency rescue, propelling human-machine collaboration to a higher level. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the method of the present invention.

[0021] Figure 2 This is a flowchart of the electroencephalogram (EEG) signal processing of the present invention.

[0022] Figure 3 The flowchart for determining the action selection strategy of the present invention is shown below.

[0023] Figure 4 This is a flowchart of the uncertainty estimation process of the present invention.

[0024] Figure 5 This is a flowchart of the action strategy space update process of the present invention.

[0025] Figure 6 This is a flowchart of the model imagination playback mechanism of the present invention. Detailed Implementation

[0026] like Figures 1-6 As shown, this invention provides a brain-computer interface (BCI) collaborative control optimization method based on deep reinforcement learning. This method combines deep learning, reinforcement learning, and brain-computer interface technologies to achieve efficient control of intelligent devices. The specific implementation methods of this invention will be described in detail below.

[0027] First, the method of this invention includes an acquisition step, a processing step, an update step, and an output step. In the acquisition step, this method acquires the user's electroencephalogram (EEG) signals and current task state. This step lays the foundation for subsequent processing and optimization.

[0028] In the processing steps, this method determines the action selection strategy based on the acquired EEG signals and the current task state using a deep reinforcement learning framework. Preferably, this invention employs a Deep Q-Network (DQN) as the core of the deep reinforcement learning algorithm. The DQN algorithm can effectively process high-dimensional input data and learn complex nonlinear strategies, making it particularly suitable for scenarios involving complex signal processing, such as brain-computer interfaces.

[0029] After determining the action selection strategy, this method executes the corresponding action and obtains environmental feedback. Environmental feedback can include information such as changes in the state of the smart device and task completion rate. Based on this environmental feedback, this method calculates a reward value and an uncertainty estimate. The reward value reflects the quality of the current action, while the uncertainty estimate represents the level of confidence in the current state-action pair.

[0030] In the update step, this method updates the action policy space based on the calculated reward value and uncertainty estimate. This step is crucial for policy optimization. Through continuous updates and learning, this method can gradually improve the action selection policy and enhance control performance.

[0031] Finally, in the output step, this method outputs an optimized action strategy to control the intelligent device to perform tasks. This optimized strategy can better adapt to the user's intentions and task requirements, improving the efficiency of human-machine collaboration.

[0032] Next, this invention further refines the steps for acquiring the user's electroencephalogram (EEG) signals. Specifically, this method utilizes the event-related potential (ERP) paradigm to detect the target signal by measuring the user's EEG signals during the task. The ERP paradigm is a widely used method in cognitive neuroscience research, capable of effectively capturing EEG activity associated with specific events or stimuli.

[0033] After acquiring the target signal, this method preprocesses it, including denoising and smoothing operations. The denoising process can employ methods such as wavelet transform or independent component analysis (ICA) to remove interference from electromyography (EMG), electrooculography (EOG), etc. Smoothing operations can use techniques such as moving average or Gaussian filtering to reduce random fluctuations in the signal. These preprocessing steps significantly improve the accuracy of subsequent analysis.

[0034] After preprocessing, this method uses the target signal to construct an online real-time prediction model. This model decodes the user's intent in real time, providing a basis for subsequent action selection.

[0035] To better construct online real-time prediction models, this invention proposes a method based on Support Vector Machines (SVM). Specifically, this method first constructs an offline SVM prediction model. SVM is a powerful machine learning algorithm, particularly suitable for handling high-dimensional data and small sample problems, which is highly compatible with the characteristics of brain-computer interfaces.

[0036] When constructing an offline SVM model, this method determines the parameters of the model through cross-validation. Cross-validation is a commonly used model selection and evaluation method that can effectively avoid overfitting. Preferably, this invention uses k-fold cross-validation, where k can be chosen to be 5 or 10. This parameter selection achieves a good balance between model performance and computational efficiency.

[0037] Based on an offline SVM model, this method obtains an online real-time prediction model. This online model can quickly predict the user's intention based on real-time input EEG signals. To improve the real-time performance of the prediction, this method can employ an incremental learning strategy to dynamically update the model parameters.

[0038] In practical applications, the method of this invention can significantly improve the performance of brain-computer interface systems. For example, in a scenario of intelligent prosthetic limb control, this method can adjust the movement of the prosthesis in real time based on the user's EEG signals. Through deep reinforcement learning, the system can gradually learn the user's intention patterns, improving the accuracy and naturalness of control. Simultaneously, the uncertainty estimation mechanism of this method can effectively handle noise and variability in EEG signals, improving the robustness of the system.

[0039] In summary, the brain-computer interface (BCI) collaborative control optimization method based on deep reinforcement learning provided by this invention achieves efficient and adaptive control of intelligent devices by combining advanced signal processing technology, machine learning algorithms, and reinforcement learning strategies. This method not only improves control accuracy and response speed but also continuously optimizes the control strategy based on user feedback, providing a new solution for the practical application of BCI technology. Next, this invention will describe in detail the specific steps of determining the action selection strategy using a deep reinforcement learning framework. In a preferred embodiment of this invention, this process includes defining a state space, defining an action space, setting a reward function, and learning the action selection strategy using an online deep reinforcement learning algorithm.

[0040] First, this method defines a multi-dimensional state space. This state space includes the state of the current task type, the state of the current action, the state of the current target position, the state of the current time, and the state of the current completion status. Through this comprehensive state representation, this method can capture various key information during task execution. For example, in a scenario of intelligent prosthetic limb control, the state may include the current movement type (such as walking, running, or climbing stairs), the current position and speed of the prosthetic limb, the target position, the time elapsed since the task was executed, and the completion status, etc.

[0041] Secondly, this method defines the action space, primarily including the state in the next time step. In the example of intelligent prosthetic limb control, actions might be represented as changes in the angle of the prosthetic joint or adjustments in movement speed. By defining actions as the state in the next time step, this method can directly learn the state transition function, which helps improve the continuity and smoothness of control.

[0042] Next, this invention establishes a comprehensive reward function. The design of this reward function is a key innovation of this method, as it comprehensively considers multiple factors to better guide policy learning. Specifically, the reward function can be expressed as: The meanings of each item are as follows: A reward for the completion of the current action; The reward is the average time it takes for the current user to complete the task. The penalty term is for the average time. Rewards for the rate of change over time; The penalty to be selected for the action; This multi-factor reward design allows for a more comprehensive evaluation of the quality of actions. For example, completion rewards encourage the system to complete tasks quickly, average time rewards and penalties balance speed and stability, time change rate rewards promote smoothness of actions, and action selection penalties inhibit overly frequent action changes.

[0043] In practical applications, the weights of these reward items can be adjusted according to specific task requirements. Preferably, this invention uses methods such as grid search or Bayesian optimization to automatically adjust these weights to adapt to different users and task types.

[0044] Finally, this method uses the online deep reinforcement learning algorithm DQN (Deep Q Network) to learn the action selection strategy. The DQN algorithm represents a significant breakthrough in reinforcement learning, combining Q-learning and deep neural networks to effectively handle high-dimensional state spaces. In this invention, the use of DQN enables the system to learn the optimal control strategy from complex EEG signals and environmental feedback.

[0045] Preferably, the present invention employs improved versions such as Double DQN or Prioritized Experience Replay to further improve learning efficiency and policy quality. For example, Double DQN effectively reduces the overestimation problem of Q-values ​​by decoupling action selection and evaluation; while Prioritized Experience Replay accelerates the learning process by replaying important samples more frequently.

[0046] In the policy learning process, this method also introduces an uncertainty estimation mechanism. The core of this mechanism is calculating the difference in the action-value function between different action values. Specifically, this method calculates: in, The parameters of a deep neural network, This indicates a possible action. When this difference exceeds a preset uncertainty threshold... In such cases, the system will increase exploratory behavior. This mechanism helps to strike a balance between strategy optimization and environmental exploration, and is particularly suitable for handling non-stationary environments or situations where user intent changes frequently.

[0047] In practical applications, uncertain threshold The choice of threshold is crucial. An excessively large threshold may cause the system to become overly conservative and unable to adapt to new situations; while an excessively small threshold may cause frequent policy fluctuations. Based on experience, this invention suggests setting the threshold accordingly. The value can be set between 0.1 and 0.3 and can be dynamically adjusted according to the specific task.

[0048] When uncertainty is high, this method calculates the exploration level L_p of the current action strategy. Introducing the exploration level allows the system to make more attempts under uncertain conditions, thereby obtaining more environmental information. The specific calculation formula is as follows: in, The value typically ranges from 0 to 1 and can be dynamically adjusted based on the complexity of the task and the learning stage. For example, a larger value can be set in the early stages of learning. Values ​​are used to encourage exploration, and can be reduced once the strategy has stabilized. It is worthwhile to focus on utilizing the knowledge already learned.

[0049] Furthermore, this invention introduces a model visualization and playback mechanism to address situations where users encounter errors or interruptions during the task. The core idea of ​​this mechanism is to utilize collected data for offline learning in a simulation environment, thereby improving the system's robustness and learning efficiency.

[0050] Specifically, when a user error or task interruption is detected, this method simulates the same task in a simulation environment based on the event-related potential signals collected by the user during the current task. This process can be viewed as a "mental rehearsal," allowing the system to explore different strategies and possible outcomes without affecting actual control.

[0051] In the simulation environment, this method continues to use the aforementioned deep reinforcement learning algorithm to update the action policy. This offline learning can significantly accelerate the policy optimization process, especially for high-risk actions that are difficult to attempt frequently in real-world environments. Simultaneously, by comparing simulation results with actual execution results, the system can continuously improve its internal model and enhance prediction accuracy.

[0052] It is worth noting that the design of the simulation environment is crucial to the effectiveness of the model's visualization and playback mechanism. This invention proposes using a physics-based simulation engine to simulate the real-world environment as accurately as possible. Simultaneously, random perturbations can be introduced to increase the diversity of the simulation and improve the learning's generalization ability.

[0053] In summary, by defining appropriate state and action spaces, designing a comprehensive reward function, employing advanced deep reinforcement learning algorithms, introducing uncertainty estimation mechanisms, and incorporating model visualization and playback innovations, the method provided by this invention can effectively learn and optimize brain-computer collaborative control strategies. This method not only adapts to complex and ever-changing task environments but also makes full use of limited training data, providing strong technical support for achieving more intelligent and natural human-computer interaction control. In another embodiment of this invention, to further improve the system's security and reliability, this method introduces an action command constraint function. This constraint function is designed to avoid irrational or erroneous user behavior, thereby ensuring the stability and security of the entire control process.

[0054] Specifically, the action command constraint function proposed in this invention can be expressed as: In this formula: Indicates the maximum permissible speed for different actions. This indicates the initial speed set by the user. To control command constraints For the reward threshold, noise level As a reward, The number of samples for the action strategy. This represents the maximum number of samples.

[0055] The design of this constraint function embodies several key innovations of this invention. First, by comparing the difference between the actual speed and the maximum permissible speed, this method can effectively limit excessively fast or violent movements, preventing potential safety hazards. Second, an exponential decay term is introduced. This causes the constraint strength to decrease as positive rewards increase, allowing the system to maintain a degree of flexibility while remaining safe. Finally, the sampling rate ratio... The introduction of this ensures that the constraint strength gradually decreases as learning progresses, enabling the system to gradually adapt to the user's control style.

[0056] In practical applications, the choice of parameters is crucial to the effectiveness of the constraint function. For example, The value can be adjusted based on the risk level of the task, typically between 0.1 and 1. For high-risk tasks, a larger value can be selected. A higher value can be chosen to strengthen constraints; however, for low-risk tasks, a smaller value can be selected to improve the system's responsiveness. Noise parameter It is typically set between 0.01 and 0.1 to simulate environmental uncertainty and the randomness of user input.

[0057] Another innovation of this invention is the introduction of transfer learning technology, which enables the system to transfer learned knowledge to similar tasks. This method greatly improves learning efficiency and reduces training time on new tasks.

[0058] In the transfer learning process, this method first calculates the transfer weight w based on the similarity between the source task and the target task. This weight determines how much knowledge can be transferred from the source task to the target task. The calculation of the transfer weight can be based on the cosine similarity of task features or the correlation coefficient of task performance. Preferably, this invention employs an adaptive weight calculation method, enabling the transfer process to be dynamically adjusted according to the specific circumstances of the task.

[0059] Specifically, the transfer weight w ranges from 0 to 1. When w is close to 0, it indicates that the two tasks are not very similar, and the system mainly relies on learning from the new task. When w is close to 1, it indicates that the two tasks are highly similar, and the system can directly apply most of the knowledge learned from the source task. In practical applications, the initial value of w can be set to 0.5, and then dynamically adjusted based on the performance of the task.

[0060] Transfer learning is not only applicable to knowledge transfer between tasks, but also to model transfer between different users. For example, in the scenario of intelligent prosthetic limb control, a user's control model can be used as a starting point to quickly customize personalized control strategies for new users. This method greatly shortens the adaptation time for new users and improves the system's versatility.

[0061] Finally, this invention describes in detail the specific steps of updating the action policy space. This update process is the core of the entire method and directly affects the system's learning effect and control performance.

[0062] First, the system calculates the reward r based on the event-related potential. If the reward r is greater than 0, it indicates that the current action is valid. The system then merges this reward r with the current action policy value v(a) and updates the current action policy in the action policy space. This positive reinforcement increases the probability of selecting valid actions.

[0063] If the calculated reward r is less than 0, it indicates that the current action may not be ideal. The system will add this training sample to the learning sample set for the current action, but will not immediately update the policy. This approach avoids over-adjusting the policy due to a single negative feedback, improving the stability of learning. These negative samples will be used in subsequent batch learning, helping the system to comprehensively understand the optimal action selection under various conditions.

[0064] When the calculated reward r equals 0, the system maintains the current action. This approach reflects a key design principle of this invention: maintaining policy stability under uncertain conditions.

[0065] After each update, the system recalculates the action value function using the updated current action policy value. This action value function reflects the long-term value of choosing a particular action in the current state and is an important basis for subsequent action selection.

[0066] Preferably, this invention employs a Temporal Difference (TD) learning method to update the action-value function. TD learning can update the estimated value in real time at each time step, making it particularly suitable for online learning scenarios. The specific update formula can be expressed as: in, It's the learning rate. These are discount factors, which control the learning speed and the degree of emphasis on future rewards, respectively. In practical applications, the learning rate... The discount factor is typically set between 0.01 and 0.1 and can be gradually decreased as learning progresses to ensure eventual convergence. It is usually set between 0.9 and 0.99, reflecting the system's emphasis on long-term rewards.

[0067] Through this meticulous update mechanism, the method of this invention can continuously optimize action selection strategies while maintaining learning stability, gradually improving the effectiveness of brain-computer collaborative control. This method is particularly suitable for handling complex and dynamically changing control tasks, such as intelligent prosthetic limb control and brain-controlled robot operation, and other advanced human-computer interaction scenarios.

[0068] In summary, the brain-computer interface (BCI) collaborative control optimization method based on deep reinforcement learning provided by this invention achieves a safe, efficient, and transferable intelligent control system by introducing action command constraint functions, transfer learning techniques, and a refined policy update mechanism. This method not only adapts to the characteristics of individual users but also flexibly transfers knowledge across different tasks, providing strong technical support for the widespread application of BCI technology.

[0069] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto; any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the scheme and improved concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A brain-computer collaborative control optimization method based on deep reinforcement learning, characterized in that, include: The acquisition steps include: Acquire the user's brainwave signals and current task status; The processing steps include: Based on the EEG signals and the current task state, a deep reinforcement learning framework is used to determine the action selection strategy. Based on the action selection strategy, execute the action and obtain environmental feedback; Based on the environmental feedback, the reward value and uncertainty estimate are calculated; The update steps include: The action policy space is updated based on the reward value and the uncertainty estimate; Output steps, including: The optimized action strategy is output to control the smart device to perform tasks.

2. The method according to claim 1, characterized in that, The steps for acquiring the user's electroencephalogram (EEG) signals specifically include: By utilizing the event-related potential paradigm, target signals are detected by measuring the user's electroencephalogram (EEG) signals during the task. The target signal is preprocessed, including noise reduction and smoothing; Using the target signal, an online real-time prediction model is constructed.

3. The method according to claim 2, characterized in that, The steps for constructing the online real-time prediction model specifically include: Construct an offline support vector machine prediction model; The parameters in the offline support vector machine prediction model are determined by cross-validation. Based on the offline support vector machine prediction model, an online real-time prediction model is obtained.

4. The method according to claim 1, characterized in that, The steps for determining the action selection strategy using a deep reinforcement learning framework specifically include: Define the state space, including the state of the current task type, the state of the current action, the state of the current target position, the state of the current time, and the state of the current completion status; Define the action space, including the state at the next time step; Set a reward function; The online deep reinforcement learning algorithm DQN is used to learn a strategy for action selection.

5. The method according to claim 4, characterized in that, The reward function is specifically as follows: in, The reward is for the completion of the current action. The reward is the average time it takes for the current user to complete the task. The penalty term is the average time. The reward is for the rate of change over time. The penalty to be selected for the action.

6. The method according to claim 1, characterized in that, The steps for calculating the uncertainty estimate specifically include: The output of the action value function is calculated based on the difference between each action value. when Greater than the uncertainty threshold At that time, calculate the current time action strategy exploration level. calculate .

7. The method according to claim 1, characterized in that, It also includes the following steps: A model-imagination replay mechanism is introduced. When an error occurs or is interrupted during a user task, the same task is simulated in the simulation environment based on the event-related potential signals collected by the user in the current task. Update the action policy in the simulation environment.

8. The method according to claim 1, characterized in that, It also includes the following steps: Design action command constraint functions to prevent irrational or erroneous user behavior; The action command constraint function is: in, The maximum permissible speed for different actions, This indicates the initial speed set by the user. To control command constraints, For the reward threshold, For noise level, As a reward, The number of samples for the action strategy. This represents the maximum number of samples.

9. The method according to claim 1, characterized in that, It also includes the following steps: Transfer learning is performed during the control process to transfer the learned knowledge to similar tasks; The transfer learning includes: calculating a transfer weight w based on the similarity between the source task and the target task, and using the transfer weight w to transfer parameters, with the value of w varying from 0 to 1.

10. The method according to claim 1, characterized in that, The steps for updating the action strategy space specifically include: If the reward r calculated based on the event-related potential is greater than 0, the reward r and the action policy value v(a) are merged, and the current action policy in the action policy space is updated. If the calculated reward r is less than 0, add the current training sample to the learning sample set for the current action and wait for the next update; If the calculated reward r equals 0, the current action is not updated; The action value function is calculated using the updated current action policy value.

Images