Basic ganglion Beta rhythm regulation and control device and system based on reinforcement learning
By constructing a stimulus current recommendation model using a soft actor-critic algorithm based on reinforcement learning, the shortcomings of the DBS system in the dynamic adjustment of neural activity are addressed, enabling fine-grained control of Beta rhythms and improving the adaptability and stability of the control.
Patent Information
- Application Number
- CN202511932147.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-01-23
AI Technical Summary
Existing deep brain stimulation (DBS) systems cannot dynamically adjust to real-time changes in neural activity, resulting in an inability to stably regulate neural rhythmic activity. Furthermore, existing closed-loop regulation strategies suffer from poor adaptability, low control precision, and a lack of adaptive capabilities.
The soft actor-critic (SAC) algorithm based on reinforcement learning is adopted. Local field potential signals are collected in real time through STN electrodes to construct a stimulation current recommendation model. The actor network recommends the optimal stimulation current intensity, and the critic network evaluates its value to achieve fine control of Beta rhythm.
It achieves fine and smooth adjustment of DBS parameters, adapts to individual differences and nonlinear dynamic changes in neural signals, improves the response speed and stability of the closed-loop system, outputs continuous stimulation current, and avoids abrupt changes caused by discrete control.
Smart Images
Figure CN121371493A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of neural engineering, in particular to a basal ganglia beta rhythm regulation device and system based on reinforcement learning. BACKGROUND
[0002] Rhythmic activity in the electrophysiological activity of deep brain nuclei is related to motor function and disease symptoms, for example, the beta band (8-35 Hz) in the local field potential (LFP) of the subthalamic nucleus (STN) in the basal ganglia circuit is related to the motor slowness of Parkinson's disease. At present, deep brain stimulation (DBS) is an effective means of regulating such neural rhythms. However, most traditional DBS uses fixed stimulation parameters, which cannot be dynamically adjusted according to real-time changes in neural activity, resulting in the inability to stably regulate neural rhythm activity.
[0003] In recent years, the concept of closed-loop DBS has been proposed, aiming to dynamically adjust stimulation parameters according to real-time monitoring of biomarkers. Existing closed-loop control strategies are mostly based on simple proportional-integral-derivative control or threshold triggering mechanisms, which have fixed control logic and are difficult to adapt to complex nonlinear neural dynamics. In view of the above problems, existing research has attempted to apply reinforcement learning to DBS parameter optimization. However, existing methods mostly use algorithms with discrete action spaces, which cannot output continuous and fine current adjustment instructions; or use algorithms with poor stability, which are difficult to guarantee the effect of long-term regulation.
[0004] In summary, the existing technology has the problems of poor adaptability of closed-loop DBS regulation strategy, low control precision, and lack of adaptive ability. SUMMARY
[0005] In order to solve the above problems, the present application provides a basal ganglia beta rhythm regulation device and system based on reinforcement learning, which regulates the beta rhythm in the brain to the expected level based on reinforcement learning.
[0006] According to some embodiments, the present application adopts the following technical solution: The first aspect of the present application provides a basal ganglia beta rhythm regulation device based on reinforcement learning, comprising an STN electrode and an intelligent regulator, wherein the STN electrode and the intelligent regulator are connected; The intelligent regulator is configured to: acquire a local field potential signal collected by the STN electrode in real time, extract a Beta rhythm amplitude at a current time from the local field potential signal; based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, input a current state into a trained stimulation current recommendation model to obtain an optimal stimulation current intensity; and apply the stimulation current intensity to the STN electrode to perform basal ganglia Beta rhythm regulation. The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, includes an actor network and a critic network, takes a negative value of a deviation of the Beta rhythm amplitude at the current time from a target value as a reward function, and takes the stimulation current intensity as an action.
[0007] The second aspect of the present application provides a basal ganglia Beta rhythm regulation system based on reinforcement learning, including the basal ganglia Beta rhythm regulation device based on reinforcement learning.
[0008] The third aspect of the present application provides a computer readable storage medium.
[0009] A computer readable storage medium, which stores a computer program, the program being executed by a processor to perform the following steps: acquire a local field potential signal collected by the STN electrode in real time, extract a Beta rhythm amplitude at a current time from the local field potential signal; based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, input a current state into a trained stimulation current recommendation model to obtain an optimal stimulation current intensity; apply the stimulation current intensity to the STN electrode to perform basal ganglia Beta rhythm regulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, includes an actor network and a critic network, takes a negative value of a deviation of the Beta rhythm amplitude at the current time from a target value as a reward function, and takes the stimulation current intensity as an action.
[0010] The fourth aspect of the present application provides a computer device.
[0011] A computer device, including a memory, a processor and a computer program stored on the memory and executable on the processor, the processor performing the following steps when executing the program: Acquire the local field potential signal collected in real time by the STN electrode, and extract the Beta rhythm amplitude at the current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the last time, construct the current state input into the trained stimulation current recommendation model to obtain the optimal stimulation current intensity; Apply the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm regulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, includes an actor network and a critic network, takes the negative value of the deviation of the Beta rhythm amplitude at the current time from the target value as a reward function, and takes the stimulation current intensity as an action.
[0012] The fifth aspect of the application provides a program product.
[0013] A program product, which is a computer program product, includes a computer program, and the computer program performs the following steps when executed by a processor: Acquire the local field potential signal collected in real time by the STN electrode, and extract the Beta rhythm amplitude at the current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the last time, construct the current state input into the trained stimulation current recommendation model to obtain the optimal stimulation current intensity; Apply the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm regulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, includes an actor network and a critic network, takes the negative value of the deviation of the Beta rhythm amplitude at the current time from the target value as a reward function, and takes the stimulation current intensity as an action.
[0014] Compared with the prior art, the application has the following beneficial effects: The SAC algorithm is used for learning, and continuous stimulation current can be output, fine and smooth adjustment of the DBS parameter is realized, and sudden changes and discomfort caused by discrete control are avoided.
[0015] The SAC algorithm maximizes the cumulative reward and the policy entropy in the optimization process, so that the agent maintains good exploration ability and robustness while learning an effective control policy, and can adapt to individual differences and nonlinear dynamic changes of neural signals.
[0016] The application constructs a state feedback mechanism based on current and historical Beta amplitude, which can capture the dynamic trend of neural activity, realize more accurate predictive regulation, and improve the response speed and stability of the closed-loop system. BRIEF DESCRIPTION OF DRAWINGS
[0017] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments of the application, together with its
[0018] Figure 1 The total flowchart of the stimulation current recommendation model for Example 1; Figure 2 The subflowchart of the learning phase of the stimulation current recommendation model for Example 1; Figure 3 The subflowchart of the application phase of the stimulation current recommendation model for Example 1. DETAILED DESCRIPTION The application will be further described below with reference to the drawings and embodiments.
[0019] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0020] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be further understood that when the terms "comprise" and / or "comprising" are used in this specification, they indicate the presence of a feature, step, operation, device, component and / or combinations thereof.
[0021] Glossary LFP: Local Field Potential DBS: Deep Brain Stimulation AT: Target Beta Amplitude, refers to the amplitude of the Beta rhythm (8-35 Hz) in the thalamic local field potential that is expected to be maintained, in microvolts (μV).
[0022] AC(t): Beta Amplitude at Current Time t, refers to the instantaneous amplitude of the Beta rhythm extracted from the LFP signal of the STN at time t.
[0023] st: state vector at time t, defined as st = [AC(t), AC(t-1)], i.e., a two-dimensional vector composed of the current and previous Beta rhythm amplitudes, used to describe the dynamic state of neural activity.
[0024] I(t): stimulation current at time t, refers to the deep brain stimulation (DBS) current intensity applied to the STN electrode at time t, which is the control action of the present invention.
[0025] Imin: minimum allowed stimulation current, is the lower limit of the action variable I(t).
[0026] Imax: maximum allowed stimulation current, is the upper limit of the action variable I(t).
[0027] r(t): reward at time t, is the feedback signal of reinforcement learning, its calculation formula is r(t) = -|AC(t) - AT|, which represents the deviation of the current Beta rhythm amplitude from the target value, the smaller the deviation, the greater the reward.
[0028] γ: discount factor, a hyperparameter in reinforcement learning algorithm, used to measure the importance of future rewards relative to immediate rewards, its value is between 0 and 1, set to 0.99 in the present invention.
[0029] τ: soft update coefficient, used for updating the parameters of the target network in SAC algorithm, to avoid unstable training, set to 0.005 in the present invention.
[0030] θ: actor network parameters, refer to the weights and bias parameters of the actor network in SAC algorithm, saved after training in the learning stage, and used to generate control policies in the application stage.
[0031] φ1, φ2: critic network parameters, refer to the weights and bias parameters of the two independent critic networks Q1 and Q2 in SAC algorithm, used to evaluate the value of state-action pairs.
[0032] π(s|θ): actor network, is the policy network in SAC algorithm used to generate actions (i.e., stimulation current I(t)) based on state s, its parameters are θ.
[0033] Q1(s, a|φ1), Q2(s, a|φ2): critic network, is the network in SAC algorithm used to evaluate the cumulative reward (Q-value) obtained by performing action a in state s, using a double critic structure to improve stability.
[0034] μ(·): action mean function, refers to the mean of the action distribution (Gaussian distribution) output by the actor network, representing the most likely deterministic action to be taken in a given state.
[0035] Q value: refers to the expected value of the cumulative discounted reward that the agent can obtain in the future after performing a certain action in a certain state. In the SAC algorithm, the output of the critic network Q(s, a) is an estimate of the Q value, which is the core indicator for evaluating the merits of actions.
[0036] done: termination flag, a Boolean variable indicating whether a training round is over.
[0037] Embodiment one In one embodiment of the present application, a basal ganglia Beta rhythm regulation device based on reinforcement learning is provided, comprising an STN electrode and an intelligent regulator, wherein the STN electrode and the intelligent regulator are connected; The intelligent regulator is configured to: acquire the local field potential signal collected by the STN electrode in real time, extract the Beta rhythm amplitude at the current time from the local field potential signal; based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the last time, input the current state into the trained stimulation current recommendation model to obtain the optimal stimulation current intensity; and apply the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm regulation. The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, including an actor network and a critic network, and the negative value of the deviation between the Beta rhythm amplitude at the current time and the target value is used as a reward function, and the stimulation current intensity is used as an action. The actor network recommends the optimal stimulation current intensity according to the current state, and the critic network evaluates the value of the stimulation current intensity recommended by the actor network in the current state based on the reward function.
[0038] As an embodiment, the basal ganglia Beta rhythm regulation device based on reinforcement learning of the present application regulates the Beta rhythm in the brain to the expected level based on reinforcement learning. The most critical part is the learning and application stage of the stimulation current recommendation model. In this embodiment, an advanced deep reinforcement learning framework, the soft actor-critic (SAC) algorithm, is used to construct an intelligent regulator that can input continuous neural states and output continuous stimulation currents. By maximizing the weighted sum of cumulative rewards and policy entropy, the agent can learn precise regulation while maintaining sufficient exploration ability, thereby adapting to individual differences and state changes. The objective function is defined as
[0039] wherein, is the expected value, representing the average performance under the trajectory (state and action sequence) generated by the policy π, is the discount factor, and r(t) is the reward function, is the state at time t, is the policy entropy, which is calculated as:
[0040] where, is the expected operation, which represents the statistical weighted average performance of all possible actions a generated by the current policy distribution π in the sampled state space. The weighted average performance of all candidate stimulation current intensities a output according to the policy distribution π is shown below; α is the temperature coefficient, which can be learned through the following objective function:
[0041] where, is the expected operation, which represents the statistical weighted average performance of all possible actions a generated by the current policy distribution π in the sampled state space. is the target entropy, which is a preset constant, usually set according to the dimension of the action space, and in this paper it is set to -1.
[0042] The stimulation current recommendation model is described in detail from the learning phase and the application phase as shown in the following: Figure 1 I. Learning phase (1) According to clinical evaluation or historical data, set the target control value AT, for example, set the Beta rhythm amplitude when the patient's symptoms are significantly relieved as AT, so AT is the amplitude of the Beta rhythm (8-35 Hz) in the thalamic nucleus local field potential signal that is expected to maintain, with the unit of microvolts (μV).
[0043] (2) Define the state variable. Collect the LFP (local field potential) signal of STN, extract the instantaneous Beta rhythm amplitude through band-pass filtering (8-35 Hz) and Hilbert transform (Hilbert Transform). Define the state variable as the state vector composed of the Beta rhythm amplitude at the current time and the last time, i.e. st = [AC(t), AC(t-1)], where AC(t) and AC(t-1) are the Beta rhythm amplitudes at the current time and the last time, respectively.
[0044] (3) Define the action variable. The action is the stimulation current intensity I(t) applied to the STN electrode, whose value range [Imin, Imax] is determined by the user, such as I(t) ∈ [0, 5] mA.
[0045] (4) Define the reward function. The negative value of the deviation between the current Beta rhythm amplitude and the target value is used as the reward function, i.e. r(t) = -|AC(t) - AT|, which directly reflects the immediate effect of the control, and the smaller the deviation, the greater the reward.
[0046] (5) Initialize the neural network structure of the soft actor-critic (SAC) algorithm, including the actor network π(s|θ), two critic networks Q1(s, a|φ1), Q2(s, a|φ2), the corresponding target networks, and the experience replay pool, and set the discount factor γ, the soft update coefficient τ, the learning rate LR, the temperature coefficient α, the batch size BS, and the replay pool capacity RC.
[0047] In this embodiment, the actor and critic are implemented using a deep neural network. The input of the actor network is a two-dimensional state st, and the output is the mean μ and standard deviation σ of a Gaussian distribution. The input of the critic network is the concatenation of the state st and the action I(t), and the output is the Q value. The hyperparameters are set as follows: γ = 0.99, τ = 0.005, LR = 3 × 10-4, α = 0.2, BS = 256, and RC = 106.
[0048] (6) Perform learning and training: At each time step, generate the action I(t) from the current state st using the actor network, perform the stimulation, obtain the next AC(t+1), calculate the reward r(t), construct the next state st+1 = [AC(t+1), AC(t)], and store the transition sample (st, I(t), r(t), st+1, done) in the experience replay pool. When the number of samples in the pool is sufficient, sample a batch of data to update the critic network, the actor network, and the temperature coefficient simultaneously, and soft-update the target network until the network converges. Save the trained actor network parameters θ. The specific steps are as follows: Figure 2 1) At each time step t, obtain the current state st.
[0049] 2) The actor network outputs the action distribution based on st to obtain the stimulation current intensity I(t).
[0050] 3) Perform I(t) to stimulate the patient.
[0051] 4) Collect the LFP signal after stimulation, extract AC(t+1), calculate the reward r(t), and construct the next state st+1 = [AC(t+1), AC(t)].
[0052] 5) Store the experience (st, I(t), r(t), st+1, done) in the experience replay pool.
[0053] 6) When the number of samples in the replay pool is greater than the batch size, randomly sample a batch of data to update the parameters of the two critic networks to minimize the Bellman error.
[0054] 7) Based on the updated critic network, calculate the policy gradient, update the parameters of the actor network, so as to tend to select the action with high Q value and high entropy.
[0055] 8) Update the temperature coefficient a and the target network parameters synchronously.
[0056] 9) Repeat steps 1) to 8) until the performance of the actor network is stable, and save the parameters θ thereof.
[0057] II. Application stage, as shown in Figure 3 (1) Load the deployed actor network parameters θ after training to construct the deterministic policy I(t) = μ(st|θ), wherein μ(·) is the action mean value output by the actor network; (2) Real-time acquisition of LFP signals of the STN, and extraction of the Beta rhythm amplitude AC(t) at the current time; (3) Construct the current state st = [AC(t), AC(t-1)] and input it to the actor network to calculate the mean value μ(st|θ) of the output action; (4) Clip the mean value μ(st|θ) to the range [Imin, Imax] to generate the final stimulation current intensity I(t) and apply it to the STN electrode to perform electrical stimulation; (5) Wait for the next sampling period, and loop steps (2) to (4) to realize continuous closed-loop regulation of the Beta rhythm amplitude.
[0058] Embodiment Two The embodiment provides a basal ganglia Beta rhythm regulation system based on reinforcement learning, comprising the basal ganglia Beta rhythm regulation device based on reinforcement learning as described in embodiment one.
[0059] It should be noted that the specific implementation mode of the basal ganglia Beta rhythm regulation device system based on reinforcement learning of the embodiment of the application is similar to that of the basal ganglia Beta rhythm regulation device based on reinforcement learning of the embodiment of the application, and specific reference can be made to the description of the device part. In order to reduce redundancy, this place will not be repeated.
[0060] Embodiment Three The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the following steps: Acquire the local field potential signal collected by the STN electrode in real time, and extract the Beta rhythm amplitude at the current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the last time, construct the current state and input it into the trained stimulation current recommendation model to obtain the optimal stimulation current intensity; applying the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm modulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network. A negative value of a deviation between a Beta rhythm amplitude at a current time and a target value is used as a reward function, and a stimulation current intensity is used as an action. The actor network recommends an optimal stimulation current intensity according to a current state. The critic network evaluates a value of the stimulation current intensity recommended by the actor network in the current state based on the reward function.
[0061] Embodiment Four The embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: acquiring a local field potential signal collected in real time by an STN electrode, and extracting a Beta rhythm amplitude at a current time from the local field potential signal; constructing a current state based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, inputting the current state into a trained stimulation current recommendation model, and obtaining an optimal stimulation current intensity; applying the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm modulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network. A negative value of a deviation between a Beta rhythm amplitude at a current time and a target value is used as a reward function, and a stimulation current intensity is used as an action. The actor network recommends an optimal stimulation current intensity according to a current state. The critic network evaluates a value of the stimulation current intensity recommended by the actor network in the current state based on the reward function.
[0062] Embodiment Five The embodiment provides a program product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented: acquiring a local field potential signal collected in real time by an STN electrode, and extracting a Beta rhythm amplitude at a current time from the local field potential signal; constructing a current state based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, inputting the current state into a trained stimulation current recommendation model, and obtaining an optimal stimulation current intensity; applying the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm modulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network, wherein a negative value of a deviation of a Beta rhythm amplitude at a current time from a target value is taken as a reward function, and a stimulation current intensity is taken as an action, the actor network recommends an optimal stimulation current intensity according to a current state, and the critic network evaluates a value of the stimulation current intensity recommended by the actor network in the current state based on the reward function.
[0063] The present application is described in reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks or a combination of blocks.
[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed by the computer or other programmable device provide a process for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 a step of performing the functions specified in one or more blocks.
[0065] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.
Claims
1. A basal ganglia beta rhythm modulation device based on reinforcement learning, characterized by, The STN electrode and the intelligent regulator are connected; The intelligent regulator is configured to: acquire a local field potential signal collected by the STN electrode in real time, and extract a Beta rhythm amplitude at a current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, a current state is constructed and input into a trained stimulation current recommendation model to obtain an optimal stimulation current intensity; The stimulation current intensity is applied to the STN electrode for basal ganglia Beta rhythm regulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, includes an actor network and a critic network, takes a negative value of a deviation between the Beta rhythm amplitude at the current time and a target value as a reward function, and takes the stimulation current intensity as an action.
2. The reinforcement learning-based basal ganglia beta rhythm modulation apparatus of claim 1, wherein, The actor network recommends the optimal stimulation current intensity according to the current state, and the critic network evaluates a value of the stimulation current intensity recommended by the actor network in the current state based on the reward function.
3. The reinforcement learning-based basal ganglia beta rhythm modulation apparatus of claim 1, wherein, The current state is constructed by combining the Beta rhythm amplitude AC(t) at the current time and the Beta rhythm amplitude AC(t-1) at the previous time into a two-dimensional state vector st = [AC(t), AC(t-1)], which is used as an input of the stimulation current recommendation model. The reward function is expressed by a formula as follows: r(t) = -|AC(t) - AT| 4. The reinforcement learning-based basal ganglia beta rhythm modulation apparatus of claim 1, wherein, Wherein, AC(t) is the Beta rhythm amplitude AC(t) at the current time, and AT is a Beta rhythm amplitude target value to be maintained.
5. The reinforcement learning-based basal ganglia beta rhythm modulation apparatus of claim 1, wherein, The actor network is configured to output a Gaussian distribution, and the mean value of the Gaussian distribution is taken as the recommended stimulation current intensity.
6. The reinforcement learning-based basal ganglia beta rhythm modulation apparatus of claim 1, wherein, The intelligent regulator further includes an experience replay module configured to store a transition sample of a state, an action, a reward and a next state in a learning stage, and update parameters of the actor network and the critic network based on batch sampling data.
7. A basal ganglia beta rhythm modulation system based on reinforcement learning, characterized in that, The intelligent regulator is further configured to clip the stimulation current intensity recommended by the actor network to a preset minimum allowed value Imin and a maximum allowed value Imax to generate an actual output stimulation current.
8. A computer program product comprising a computer program, characterized in that, The basal ganglia Beta rhythm regulation device based on reinforcement learning includes the basal ganglia Beta rhythm regulation device according to any one of claims 1-6. The computer program is executed by a processor to implement the following steps: Acquiring a local field potential signal collected by the STN electrode in real time, and extracting a Beta rhythm amplitude at a current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and a Beta rhythm amplitude at a previous time, a current state is constructed and input into a trained stimulation current recommendation model to obtain an optimal stimulation current intensity; The stimulation current intensity is applied to the STN electrode for basal ganglia Beta rhythm regulation; The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network, and takes the negative value of the deviation of the Beta rhythm amplitude at the current time from a target value as a reward function, and takes the stimulation current intensity as an action.
9. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium is used to store computer instructions, which are executed by the processor to implement the following steps: Obtaining a local field potential signal collected by the STN electrode in real time, and extracting a Beta rhythm amplitude at the current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the previous time, constructing a current state and inputting the current state into the trained stimulation current recommendation model to obtain an optimal stimulation current intensity; Applying the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm regulation. The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network, and takes the negative value of the deviation of the Beta rhythm amplitude at the current time from a target value as a reward function, and takes the stimulation current intensity as an action.
10. An electronic device, comprising: The processor, the memory and the computer program are included, wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to implement the following steps: Obtaining a local field potential signal collected by the STN electrode in real time, and extracting a Beta rhythm amplitude at the current time from the local field potential signal; Based on the Beta rhythm amplitude at the current time and the Beta rhythm amplitude at the previous time, constructing a current state and inputting the current state into the trained stimulation current recommendation model to obtain an optimal stimulation current intensity; Applying the stimulation current intensity to the STN electrode for basal ganglia Beta rhythm regulation. The stimulation current recommendation model adopts a soft actor-critic algorithm in reinforcement learning, and includes an actor network and a critic network, and takes the negative value of the deviation of the Beta rhythm amplitude at the current time from a target value as a reward function, and takes the stimulation current intensity as an action.