Reinforcement learning-based multi-electrode time-domain interference stimulation parameter optimization method and system

Abstract

The application discloses a multi-electrode time-domain interference stimulation parameter optimization method and system based on reinforcement learning, relates to the field of time-domain interference stimulation, and comprises the following steps: constructing a neural network based on an encoder-decoder architecture as an intelligent agent; training the intelligent agent by using a reinforcement learning algorithm to obtain a trained intelligent agent; and performing optimal distribution of electrode positions and current intensity values according to the number of electrodes to be activated by using the trained intelligent agent. The application realizes efficient, flexible and accurate global optimization of a multi-electrode time-domain interference stimulation scheme.

Description

Technical Field

[0001] This invention relates to the field of temporal interferometry stimulation technology, and in particular to a method and system for optimizing multi-electrode temporal interferometry stimulation parameters based on reinforcement learning. Background Technology

[0002] Temporal Interference Stimulation (TIS) is a cutting-edge, non-invasive neuromodulation technique that aims to create a low-frequency envelope electric field by applying two sets of slightly different high-frequency currents (typically in the kilohertz range) to the scalp. This envelope field selectively activates deep neurons, while the high-frequency carrier current itself does not directly activate the superficial cortex, thus achieving precise, non-invasive stimulation of deep brain tissue.

[0003] To precisely focus the interference electric field on the target region and maximize its stimulating effect, researchers have extensively explored the optimization of the electrode parameters (including position and current intensity) of the TIS. Early optimization methods relied on exhaustive search, which involves traversing all possible combinations of electrode positions and currents to find the optimal solution. For example, Lee et al. attempted to select four electrodes from 61 candidate electrodes and test them by adjusting the current in small increments. However, this method is extremely computationally expensive; the computational cost increases exponentially with the number of candidate electrodes, making it impractical for real-world applications.

[0004] To address the computational efficiency issue of exhaustive search, subsequent research introduced heuristic algorithms, such as the Genetic Algorithm (GA). GA searches for the optimal solution by simulating natural selection and genetic processes, avoiding brute-force traversal of the entire solution space and significantly reducing computational costs. While GA performs reasonably well in optimizing traditional two-electrode TIS pairs, its state space expands dramatically when applied to multi-electrode TIS scenarios with a larger number of electrodes, posing a significant computational challenge and hindering its effective application.

[0005] In recent years, with the development of multi-electrode TIS technology, more optimization strategies based on machine learning have emerged. For example, Huang et al. proposed a method to maximize the envelope field strength of the target region using all available electrodes. Similarly, Bahn et al. proposed an optimization paradigm based on unsupervised neural networks (USNN), which significantly shortened the optimization time. However, these advanced optimization methods generally suffer from two core drawbacks: first, they cannot effectively control the number of electrodes involved in stimulation, usually requiring the activation of all available electrodes; second, they require the application of two different frequencies of current on the same electrode, which places high demands on the implementation of stimulation hardware and limits the feasibility of their clinical application.

[0006] To address the issue of the inability to control the number of electrodes using the aforementioned methods, Lee et al. proposed a Sequential Electrode-Pair Addition Strategy. This method constructs stimulation protocols by iteratively adding electrode pairs. While this strategy represents a breakthrough in electrode number control, it inherently relies on an exhaustive search for each newly added electrode pair, resulting in extremely time-consuming computation. More importantly, this "sequential" local optimization strategy cannot guarantee that the final electrode combination is the globally optimal solution.

[0007] In summary, while existing TIS electrode optimization algorithms have made some progress in improving the focal head of deep brain stimulation, they still generally suffer from key drawbacks such as high computational cost, poor scheme flexibility (especially in electrode number control), stringent hardware requirements, and difficulty in guaranteeing global optimization. Therefore, there is an urgent need in this field for a novel multi-electrode TIS parameter optimization method that is computationally efficient, flexible in configuration, and capable of global optimization. Summary of the Invention

[0008] To address the technical problems existing in the background art, this invention proposes a method and system for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning.

[0009] In a first aspect, the present invention proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, comprising: Construct a neural network based on an encoder-decoder architecture as an intelligent agent; The agent is trained using reinforcement learning methods to obtain a well-trained agent; The trained agent is used to optimally allocate electrode positions and current intensity values ​​based on the number of electrodes to be activated.

[0010] Preferably, the intelligent agent includes: an input layer, an encoder, a position decoder, and a current decoder; The input layer, encoder, and position decoder are connected in sequence, and the outputs of the encoder and position decoder are concatenated and then connected to the current decoder.

[0011] Preferably, the input layer is used to randomly generate an initial input tensor based on the number of electrodes to be activated; the encoder is used to encode the initial input tensor into a high-dimensional hidden state tensor; the position decoder is used to determine the electrode position information tensor of each electrode to be activated based on the hidden state tensor; and the current decoder is used to obtain the current intensity value assigned to each electrode to be activated based on the hidden state tensor and the electrode position information tensor of each electrode to be activated.

[0012] Preferably, the encoder includes multiple sets of coding units connected in sequence, each set of coding units including a first fully connected layer and a first GeLU activation function connected in sequence.

[0013] Preferably, the location decoder includes a second fully connected layer, a first hyperbolic tangent layer, a first hyperbolic tangent adjustment layer, a softmax function, and a class distribution sampling layer connected in sequence.

[0014] Preferably, the current decoder includes multiple decoding units, a first fully connected layer branch, a second fully connected layer branch, and a normal distribution sampling layer; the multiple decoding units are connected in sequence, and the last decoding unit is connected to the first fully connected layer branch and the second fully connected layer branch respectively, and the first fully connected layer branch and the second fully connected layer branch are connected to the input end of the normal distribution sampling layer respectively.

[0015] Preferably, the decoding unit includes a third fully connected layer and a second GeLU function layer connected in sequence.

[0016] Preferably, the first fully connected layer branch includes a fourth fully connected layer and a softplus layer connected in sequence; the second fully connected layer branch includes a reshaping layer, a fifth fully connected layer, a second hyperbolic tangent layer, and a second hyperbolic tangent adjustment layer connected in sequence; the input terminals of the fourth fully connected layer and the reshaping layer are respectively connected to the output terminal of the last decoding unit, and the softplus layer and the second hyperbolic tangent adjustment layer are respectively connected to the input terminal of the normal distribution sampling layer.

[0017] Preferably, the reinforcement learning algorithm is the policy gradient method.

[0018] Preferably, a reinforcement learning algorithm is used to train the agent to obtain a trained agent, specifically including: The number of electrodes to be activated is obtained and input into the agent to obtain the electrode position and current intensity value of the electrodes to be activated. The current intensity value is normalized; the electrode position and the normalized current intensity value are input into the lead field matrix of the pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain. Based on the TIS envelope electric field distribution of the whole brain, the current reward function is calculated; the reward function is obtained by subtracting the expected reward baseline from the reward function; the corresponding log probability is calculated based on the advantage function; the average log probability is calculated and used as the loss function. Determine if the loss function has converged; if yes, obtain the trained agent; if not, calculate the gradient of the loss function with respect to the agent's weights using the backpropagation algorithm, update the agent's weights using the optimizer, and re-input the number of electrodes to be activated into the updated agent for iteration until the loss function converges.

[0019] Preferably, the baseline of expected rewards is determined by the formula Perform the calculation; where, Here, represents the discount factor, and represents the set of reward values. This indicates the operation of finding the maximum value. This indicates the operation of averaging.

[0020] Preferably, before inputting the electrode positions and normalized current intensity values ​​into the lead field matrix of a pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain, the method further includes: An individualized head finite element model is constructed, and the leading field matrix of the individualized head finite element model is calculated.

[0021] Preferably, an individualized head finite element model is constructed, and the leading field matrix of the individualized head finite element model is calculated, including: Obtain head magnetic resonance imaging data from healthy subjects; Based on head magnetic resonance imaging data of healthy subjects, a high-resolution individualized head finite element model was constructed; the individualized head finite element model includes the scalp, skull, cerebrospinal fluid, gray matter, and white matter. Assign a corresponding conductivity attribute to each tissue; 74 candidate electrode positions that meet preset standards are pre-defined on the scalp surface; By applying a unit current to each candidate electrode location and using a distal electrode as a reference, the quasi-static Laplace equation is solved using the finite element method to calculate the electric field distribution generated by each candidate electrode in the whole brain. Based on the individualized head finite element model and the electric field distribution results of all 74 candidate electrodes, the lead field matrix is ​​obtained.

[0022] Preferably, the reward function is the peak-to-sum ratio; where, In the formula, PSR represents the peak-to-sum ratio, and H... max C represents the maximum envelope electric field amplitude calculated within the target region. sum This represents the average field strength of all voxels whose envelope electric field amplitude exceeds 90% of the maximum value in the non-target region.

[0023] Preferably, the reinforcement learning algorithm is an actor-critic class algorithm or a deep deterministic policy gradient method.

[0024] Secondly, this invention also proposes a multi-electrode temporal interference stimulus parameter optimization system based on reinforcement learning, comprising: The agent building module is used to construct neural networks based on an encoder-decoder architecture as agents. The agent training module is used to train the agent to obtain a trained agent; The allocation module is used to optimize the allocation of electrode positions and current intensity values ​​based on the number of electrodes to be activated using a trained agent.

[0025] In this invention, the proposed method and system for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning constructs a neural network based on an encoder-decoder architecture as an agent; the agent is trained using a reinforcement learning algorithm to obtain a trained agent; and the trained agent is used to optimally allocate electrode positions and current intensity values ​​according to the number of electrodes to be activated, so as to achieve efficient, flexible and accurate global optimization of multi-electrode TIS schemes. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating the training principle of a multi-electrode temporal interference stimulus parameter optimization method based on reinforcement learning in one embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the structure of an intelligent agent in one embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the simulation results of a simulation experiment using the multi-electrode temporal interference stimulus parameter optimization method based on reinforcement learning proposed in this invention to obtain electrode positions and current intensities with different numbers of electrodes.

[0029] Figure 4 The diagram shows the simulation results of the simulation experiment to obtain the electrode positions and current intensities of 16 electrodes using different methods; where (a) represents the multi-electrode temporal interference stimulus parameter optimization method based on reinforcement learning proposed in this invention, (b) represents the traditional algorithm GA, and (c) represents the traditional algorithm USNN. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] Reference Figure 1 and Figure 2 This invention proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, comprising: Construct a neural network based on an encoder-decoder architecture as an intelligent agent; The agent is trained using a reinforcement learning algorithm to obtain a well-trained agent; The trained agent is used to optimally allocate electrode positions and current intensity values ​​based on the number of electrodes to be activated.

[0032] This invention constructs a neural network based on an encoder-decoder architecture as an agent; trains the agent using a reinforcement learning algorithm to obtain a trained agent; and utilizes the trained agent to perform optimal allocation of electrode positions and current intensity values ​​based on the number of electrodes to be activated, thereby achieving efficient, flexible, and accurate global optimization of multi-electrode temporal interference stimulation (TIS) schemes.

[0033] In this embodiment, the intelligent agent includes: an input layer, an encoder, a position decoder, and a current decoder; The input layer, encoder, and position decoder are connected in sequence, and the outputs of the encoder and position decoder are concatenated and then connected to the current decoder.

[0034] The input layer is used to randomly generate an initial input tensor based on the number of electrodes to be activated; the encoder is used to encode the initial input tensor into a high-dimensional hidden state tensor; the position decoder is used to determine the electrode position information tensor of each electrode to be activated based on the hidden state tensor; and the current decoder is used to obtain the current intensity value assigned to each electrode to be activated based on the hidden state tensor and the electrode position information tensor of each electrode to be activated.

[0035] The agent in this embodiment adopts an encoder-decoder network architecture, which can simultaneously generate discrete electrode positions and continuous current values.

[0036] In a further embodiment, the encoder includes multiple sets of coding units connected in sequence, each set of coding units including a first fully connected layer and a first GeLU activation function connected in sequence.

[0037] In practice, the encoder receives the initial input tensor randomly generated by the input layer and encodes it into a high-dimensional hidden state tensor with dimensions B×H×N through the first fully connected layer, where H is the dimension of the hidden layer.

[0038] It should be understood that, in this embodiment, the fully connected layer refers to... Figure 2 Linear layers in the middle.

[0039] In a further embodiment, the location decoder includes a second fully connected layer, a first hyperbolic tangent layer, a first hyperbolic tangent adjustment layer, a softmax function, and a category distribution sampling layer connected in sequence.

[0040] In practice, the position decoder outputs the selection weights for each candidate electrode through a second fully connected layer, a first hyperbolic tangent layer, and a first hyperbolic tangent adjustment layer. The Softmax activation function is then used to convert these selection weights into a probability distribution, representing the probability that each candidate electrode will be selected into two different frequency channels. A category distribution sampling layer is then used to construct a category distribution based on the probability of each candidate electrode being selected into two different frequency channels, and samples are taken from this category distribution to determine the electrode positions (discrete parameters) of the electrodes to be activated. This design allows for the preset and control of the total number of electrodes to be activated. To ensure the effective operation of TIS, it is mandatory that at least two electrodes are allocated to each frequency channel.

[0041] In a further embodiment, the current decoder includes multiple sets of decoding units, a first fully connected layer branch, a second fully connected layer branch, and a normal distribution sampling layer; the multiple sets of decoding units are connected in sequence, and the last decoding unit is connected to the first fully connected layer branch and the second fully connected layer branch respectively, and the first fully connected layer branch and the second fully connected layer branch are connected to the input end of the normal distribution sampling layer respectively.

[0042] In specific implementation, the first fully connected layer branch is used to output the standard deviation σ of the current intensity value, the second fully connected layer branch is used to output the mean μ of the current intensity value, and the normal distribution sampling layer is used to construct a normal distribution based on the mean μ and the standard deviation σ, and to sample in the normal distribution to generate the initial current intensity value of the electrode to be activated.

[0043] In a further embodiment, the decoding unit includes a third fully connected layer and a second GeLU function layer connected in sequence.

[0044] In a further embodiment, the first fully connected layer branch includes a fourth fully connected layer and a softplus layer connected in sequence; the second fully connected layer branch includes a reshaping layer, a fifth fully connected layer, a second hyperbolic tangent layer, and a second hyperbolic tangent adjustment layer connected in sequence. The inputs of the fourth fully connected layer and the reshaping layer are connected to the output of the last decoding unit, respectively. The softplus layer and the second hyperbolic tangent adjustment layer are connected to the input of the normal distribution sampling layer, respectively.

[0045] In specific implementation, the current decoder generates the mean μ of a Gaussian distribution through the second hyperbolic tangent layer and the second hyperbolic tangent adjustment layer of the second fully connected layer branch, and generates the standard deviation σ through the Softplus layer in the first fully connected layer branch. It then constructs a normal distribution based on the mean μ and the standard deviation σ through the normal distribution sampling layer, and samples from this parameterized normal distribution to generate the initial current intensity value for the electrode to be activated.

[0046] Furthermore, the encoder, position decoder, and current decoder of the agent in this embodiment are not limited to traditional fully connected layers. They can be replaced or combined with more complex network structures, such as attention mechanisms or Transformer networks, to more effectively capture the complex dependencies between electrode positions and improve the quality of policy generation.

[0047] In this embodiment, in one specific embodiment, the reinforcement learning algorithm includes a policy gradient method.

[0048] Therefore, the training process of an agent includes: The process of training the agent using reinforcement learning algorithms to obtain a trained agent includes: The number of electrodes to be activated is obtained and input into the agent to obtain the electrode position and current intensity value of the electrodes to be activated. The current value is normalized. The electrode positions and normalized current intensity values ​​are input into the lead field matrix of a pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain. Based on the TIS envelope electric field distribution of the whole brain, the current reward function is calculated. The reward function is obtained by subtracting the reward expectation baseline from the reward function. Based on the advantage function, the corresponding log probability is calculated. The average value of the log probabilities is calculated and used as the loss function. Determine if the loss function has converged; if yes, obtain the trained agent; if not, calculate the gradient of the loss function with respect to the agent's weights using the backpropagation algorithm, update the agent's weights using the optimizer, and re-input the number of electrodes to be activated into the updated agent for iteration until the loss function converges.

[0049] With this configuration, the agent in this embodiment can gradually learn a strategy that maximizes the reward function based on the reinforcement learning algorithm, and finally converge and output the optimal combination of electrode position and current intensity. This achieves efficient, flexible and accurate global optimization of multi-electrode TIS schemes, and solves the problems of high computational cost, inflexible control of the number of electrodes, demanding hardware requirements and difficulty in obtaining the global optimal solution in existing time-domain interferometric stimulation (TIS) electrode parameter optimization techniques.

[0050] In calculating the TIS envelope electric field distribution of the whole brain, the process is first performed by... Calculate the position The corresponding frequency and electric field strength of the electrode at the specified location; where, For frequency serial number, Electrode serial number, For location; Represents frequency The k-th electrode at position The electric field strength generated at that location; and then according to Calculate the frequency The total field below; in the formula, This represents the set of selected electrodes corresponding to frequency j. ; in, In the formula, for and The included angle, The total field at the first frequency, The total field at the second frequency; This represents the maximum value of the envelope field; That is, according to and The magnitude of the envelope field determines its maximum value; if the former is smaller, then the maximum value of the envelope field is... Conversely, it is .

[0051] In this embodiment, the advantage function is typically defined as the current reward function minus a reward expectation baseline, which is obtained through... Dynamic updates are performed; among them, Here, is the discount factor, and rewards is the set of reward values. This indicates the operation of finding the maximum value. This indicates the averaging operation. This embodiment uses the advantage function to reduce the variance of the policy gradient, making training more stable.

[0052] In one specific embodiment, the reward function is the peak-to-sum ratio.

[0053] It's important to understand that the peak-to-sum ratio (PSR) is defined as the ratio of the maximum envelope field intensity within the target brain region to the average intensity of high-field regions in non-target brain regions (such as the cortex). A higher PSR value indicates better stimulation of the focal head.

[0054] in, In the formula, PSR represents the peak-to-sum ratio, and H... max C represents the maximum envelope electric field amplitude calculated within the target region. sum This represents the average field strength of all voxels whose envelope electric field amplitude exceeds 90% of the maximum value in non-target regions (such as the cerebral cortex).

[0055] At the same time, a validity threshold is set, namely The stimulus must be greater than 0.2 V / m; otherwise, it is considered an ineffective stimulus and will be given a very low reward. A higher PSR value indicates a better stimulus focus head and a lower off-target effect.

[0056] In another specific embodiment, the reward function can be extended from a single PSR metric to a multi-objective optimization function. For example, a penalty term can be added to the reward function to limit the field strength in specific risky brain regions (such as the brainstem, visual cortex, etc.), or a penalty can be added for total current consumption to achieve a more energy-efficient and safer stimulation scheme.

[0057] In a further embodiment, the number of electrodes to be activated is input into the intelligent agent to obtain the electrode positions and current intensity values ​​of the electrodes to be activated, specifically including: The initial input tensor is randomly generated using the input layer based on the number of electrodes to be activated. The initial input tensor is encoded into a high-dimensional hidden state tensor using an encoder. The position decoder is used to determine the electrode position information tensor of each electrode to be activated based on the hidden state tensor; The current decoder is used to obtain the current intensity value assigned to each electrode to be activated based on the hidden state tensor and the electrode position information tensor of each electrode to be activated.

[0058] To ensure the safety of the stimulation, this embodiment requires that the initial current value generated by the agent undergo strict normalization.

[0059] The normalization process needs to satisfy the following safety constraints: (1) The sum of the currents at each frequency is zero, so as to satisfy Kirchhoff's laws; (2) The current amplitude on a single electrode does not exceed the preset safety limit; (3) The sum of the absolute values ​​of the total current of all electrodes shall not exceed the total safety limit of the system.

[0060] Of course, in addition to basic constraints on current amplitude and total amount, safety constraints can also integrate more advanced safety strategies. For example, restrictions on the electric field gradient (E-field gradient) can be added to avoid potential damage to neural tissue; or individualized diffusion tensor imaging (DTI) data can be combined to actively avoid important nerve fiber bundles during the optimization process.

[0061] In a further embodiment, before inputting the electrode locations and normalized current intensity values ​​into the lead field matrix of a pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain, the method further includes: An individualized head finite element model is constructed, and the leading field matrix of the individualized head finite element model is calculated to facilitate the subsequent rapid calculation of the electric field generated by arbitrary electrode combinations.

[0062] This includes constructing an individualized head finite element model and calculating the leading field matrix of the individualized head finite element model, including: Acquire head magnetic resonance imaging (MRI) data from healthy subjects; A high-resolution, individualized finite element (FE) model of the head was constructed based on magnetic resonance imaging (MRI) data of healthy subjects. The individualized finite element model of the head includes the scalp, skull, cerebrospinal fluid, gray matter, and white matter. Assign a corresponding conductivity attribute to each tissue; 74 candidate electrode positions conforming to the international 10-10 system standard were pre-set on the scalp surface; By applying a unit current to each candidate electrode location and using a distal electrode as a reference, the electric field distribution generated by each candidate electrode in the whole brain was calculated by solving the quasi-static Laplace equation using the finite element method. Based on the individualized head finite element model and the electric field distribution results of all 74 candidate electrodes, the lead field matrix is ​​obtained.

[0063] The quasi-static Laplace equation is: In the formula, For electrical conductivity, It represents the electric potential.

[0064] Therefore, by inputting the electrode locations and normalized current intensity values ​​into the pre-constructed lead field matrix of the individualized head finite element model, the TIS envelope electric field distribution of the whole brain is calculated, specifically including: The electrode positions and normalized current intensity values ​​are input into the lead field matrix, and the three-dimensional TIS envelope electric field distribution of the whole brain is obtained through linear superposition and nonlinear envelope calculation.

[0065] Of course, in addition to the basic policy gradient method, this embodiment can also employ other more advanced reinforcement learning algorithms, such as actor-critic algorithms (e.g., A2C, PPO) and deep deterministic policy gradient (DDPG) methods. Using these algorithms can further improve training stability, convergence speed, and the final optimization effect.

[0066] In summary, the technical solution proposed in this invention has the following significant beneficial effects: High-Focus Head and High Flexibility: This invention can effectively explore the vast solution space of multi-electrode configurations, finding stimulation schemes superior to traditional genetic algorithms (GA) and achieving a higher stimulation focus head. Simultaneously, this method allows for pre-setting and flexible control of the number of activated electrodes, overcoming the limitation of methods like USNN in controlling the number of electrodes, thus achieving a better balance between performance and resource consumption.

[0067] Strong global optimization capability: As a powerful optimization tool for nonlinear problems, reinforcement learning can perform global search better than local optimization methods such as sequential addition, thus having a greater probability of finding the globally optimal or near-optimal electrode configuration scheme, and comprehensively improving the stimulation effect of TIS.

[0068] Practicality and efficiency: This invention is comparable to advanced methods such as USNN in terms of performance optimization, but it is more clinically practical because it can use fewer electrodes and adopt conventional hardware solutions (each electrode only needs one frequency).

[0069] Secondly, this invention also proposes a multi-electrode temporal interference stimulus parameter optimization system based on reinforcement learning, comprising: The agent building module is used to construct neural networks based on an encoder-decoder architecture as agents. The agent training module is used to train the agent to obtain a trained agent; The allocation module is used to optimize the allocation of electrode positions and current intensity values ​​based on the number of electrodes to be activated using a trained agent.

[0070] The intelligent agent includes an input layer, an encoder, a position decoder, and a current decoder. The input layer, encoder, and position decoder are connected in sequence, and the output of the encoder and the output of the position decoder are spliced ​​together and then connected to the current decoder.

[0071] The input layer is used to randomly generate an initial input tensor based on the number of electrodes to be activated; the encoder is used to encode the initial input tensor into a high-dimensional hidden state tensor; the position decoder is used to determine the electrode position information tensor of each electrode to be activated based on the hidden state tensor; and the current decoder is used to obtain the current intensity value assigned to each electrode to be activated based on the hidden state tensor and the electrode position information tensor of each electrode to be activated.

[0072] The encoder in this embodiment includes multiple sets of coding units connected in sequence. Each set of coding units includes a first fully connected layer and a first GeLU activation function connected in sequence.

[0073] The location decoder in this embodiment includes a second fully connected layer, a first hyperbolic tangent layer, a first hyperbolic tangent adjustment layer, a softmax function, and a class distribution sampling layer connected in sequence.

[0074] In this embodiment, the current decoder includes multiple decoding units, a first fully connected layer branch, a second fully connected layer branch, and a normal distribution sampling layer; the multiple decoding units are connected in sequence, and the last decoding unit is connected to the first fully connected layer branch and the second fully connected layer branch respectively. The first fully connected layer branch and the second fully connected layer branch are connected to the input end of the normal distribution sampling layer respectively. The decoding unit includes a third fully connected layer and a second GeLU function layer connected in sequence.

[0075] The first fully connected layer branch includes a fourth fully connected layer and a softplus layer connected in sequence; the second fully connected layer branch includes a reshaping layer, a fifth fully connected layer, a second hyperbolic tangent layer, and a second hyperbolic tangent adjustment layer connected in sequence; the input of the fourth fully connected layer and the input of the reshaping layer are respectively connected to the output of the last decoding unit, and the softplus layer and the second hyperbolic tangent adjustment layer are respectively connected to the input of the normal distribution sampling layer.

[0076] In one specific embodiment, the reinforcement learning algorithm includes a policy gradient method.

[0077] Therefore, the training process of the agent includes: obtaining the number of electrodes to be activated and inputting this number into the agent to obtain the electrode positions and current intensity values ​​of the electrodes to be activated; normalizing the current intensity values; inputting the electrode positions and normalized current intensity values ​​into the lead field matrix of a pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain; calculating the current reward function based on the TIS envelope electric field distribution of the whole brain; calculating the advantage function based on the current reward function; calculating the loss function based on the advantage function and the log probability of generating the scheme; determining whether the loss function has converged; if yes, the trained agent is obtained; if not, the gradient of the loss function with respect to the weights of the agent is calculated using the backpropagation algorithm, and the weights of the agent are updated using the optimizer. The number of electrodes to be activated is then input into the updated agent for iteration until the loss function converges.

[0078] Before inputting the electrode positions and normalized current intensity values ​​into the pre-constructed lead field matrix of the individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain, this embodiment also includes: constructing an individualized head finite element model and calculating the lead field matrix of the individualized head finite element model.

[0079] The process involves constructing an individualized head finite element model and calculating its leading field matrix. This includes: acquiring head magnetic resonance imaging (MRI) data from healthy subjects; constructing a high-resolution three-dimensional model of the individualized head finite element (FE) model based on the MRI data; the individualized head finite element model including the scalp, skull, cerebrospinal fluid, gray matter, and white matter; assigning corresponding conductivity properties to each tissue; pre-setting 74 candidate electrode locations on the scalp surface that meet pre-defined criteria; applying a unit current to each candidate electrode location and using a distal electrode as a reference, solving the quasi-static Laplace equation using the finite element method to calculate the electric field distribution generated by each candidate electrode in the whole brain; and obtaining the leading field matrix based on the individualized head finite element model and the electric field distribution results of all 74 candidate electrodes.

[0080] In this embodiment, the reward function is the peak-to-sum ratio; where, In the formula, PSR represents the peak-to-sum ratio, and H... max C represents the maximum envelope electric field amplitude calculated within the target region. sum This represents the average field strength of all voxels whose envelope electric field amplitude exceeds 90% of the maximum value in the non-target region.

[0081] In another embodiment, the reinforcement learning algorithm is an actor-critic class algorithm or a deep deterministic policy gradient (DDPG) method.

[0082] The present invention will now be described in conjunction with specific embodiments.

[0083] Example 1 This embodiment proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, including: Construct a neural network based on an encoder-decoder architecture as an agent; the agent includes: an input layer, an encoder, a position decoder, and a current decoder; the input layer, encoder, and position decoder are connected in sequence, and the output of the encoder and the output of the position decoder are concatenated and then connected to the current decoder; The encoder includes multiple sets of coding units connected in sequence, and each set of coding units includes a first fully connected layer and a GeLU activation function connected in sequence. The location decoder includes a second fully connected layer, a first hyperbolic tangent layer, a first hyperbolic tangent adjustment layer, a softmax function, and a class distribution sampling layer connected in sequence. The current decoder includes multiple decoding units, a first fully connected layer branch, a second fully connected layer branch, and a normal distribution sampling layer. The multiple decoding units are connected in sequence, with the last decoding unit connected to the first and second fully connected layer branches respectively. The first and second fully connected layer branches are connected to the input of the normal distribution sampling layer respectively. Each decoding unit includes a third fully connected layer and a second GeLU function layer connected in sequence. The first fully connected layer branch includes a fourth fully connected layer and a softplus layer connected in sequence; the second fully connected layer branch includes a reshaping layer, a fifth fully connected layer, a second hyperbolic tangent layer, and a second hyperbolic tangent adjustment layer connected in sequence; the input of the fourth fully connected layer and the input of the reshaping layer are respectively connected to the output of the last decoding unit, and the softplus layer and the second hyperbolic tangent adjustment layer are respectively connected to the input of the normal distribution sampling layer; The number of electrodes to be activated is obtained and input into the agent to obtain the electrode position and current intensity value of the electrodes to be activated. The current value is normalized. The electrode positions and normalized current intensity values ​​are input into the lead field matrix of a pre-constructed individualized head finite element model to calculate the TIS envelope electric field distribution of the whole brain. Based on the TIS envelope electric field distribution of the whole brain, the current reward function is calculated; the advantage function is obtained by subtracting the expected reward baseline from the reward function; the corresponding log probability is calculated based on the advantage function; the average of the log probabilities is calculated, and the average of the log probabilities is used as the loss function; wherein, the expected reward baseline is determined by the formula... Perform the calculation; where, Here, `rewards` represents the set of reward values, `max()` represents the maximum value operation, and `Mean()` represents the average value operation. Determine whether the loss function has converged; if yes, the trained agent is obtained; if not, the gradient of the loss function with respect to the agent's weights is calculated using the backpropagation algorithm, and the weights of the agent are updated using the optimizer. The number of electrodes to be activated is then input into the updated agent for iteration until the loss function converges. The trained agent is used to optimally allocate electrode positions and current intensity values ​​based on the number of electrodes to be activated.

[0084] The process involves constructing an individualized head finite element model and calculating its lead field matrix. This includes: acquiring head magnetic resonance imaging (MRI) data from healthy subjects; constructing a high-resolution three-dimensional model of the individualized head finite element (FE) model based on the MRI data; the individualized head finite element model including the scalp, skull, cerebrospinal fluid, gray matter, and white matter; assigning corresponding conductivity properties to each tissue; pre-setting 74 candidate electrode positions conforming to the international 10-10 system standard on the scalp surface; applying a unit current (1 mA) to each candidate electrode position and using a distal electrode as a reference, solving the quasi-static Laplace equation using the finite element method to calculate the electric field distribution generated by each candidate electrode in the whole brain; and obtaining the lead field matrix based on the individualized head finite element model and the electric field distribution results of all 74 candidate electrodes.

[0085] Therefore, the electrode positions and normalized current intensity values ​​are input into the pre-constructed individualized head finite element model's lead field matrix to calculate the TIS envelope electric field distribution of the whole brain. Specifically, this involves inputting the electrode positions and normalized current intensity values ​​into the lead field matrix, and obtaining the three-dimensional TIS envelope electric field distribution of the whole brain through linear superposition and nonlinear envelope calculation.

[0086] When the number of electrodes to be activated is four, the optimal electrode positions and current intensity values ​​for the four electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​for the four electrodes to be activated, a simulation experiment is conducted on the right globus pallidus (rGPi) of the deep brain region of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, as follows. Figure 3 As shown.

[0087] Example 2 The multi-electrode temporal interference stimulus parameter optimization method based on reinforcement learning proposed in this embodiment is the same as that in Embodiment 1.

[0088] When the number of electrodes to be activated is 8, the optimal electrode positions and current intensity values ​​for the 8 electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​for the 8 electrodes to be activated, a simulation experiment is performed on the deep brain region rGPi of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 3 As shown.

[0089] Example 3 This embodiment proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, which is the same as that in Embodiment 1.

[0090] When the number of electrodes to be activated is 12, the optimal electrode positions and current intensity values ​​of the 12 electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​of the 12 electrodes to be activated, a simulation experiment is performed on the deep brain region rGPi of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 3 As shown.

[0091] Example 4 This embodiment proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, which is the same as that in Embodiment 1.

[0092] When the number of electrodes to be activated is 16, the optimal electrode positions and current intensity values ​​for the 16 electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​for the 16 electrodes to be activated, a simulation experiment is performed on the deep brain region rGPi of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 3 and Figure 4 As shown in (a) and Table 1.

[0093] Example 5 This embodiment proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, which is the same as that in Embodiment 1.

[0094] When the number of electrodes to be activated is 24, the optimal electrode positions and current intensity values ​​for the 24 electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​for the 24 electrodes to be activated, a simulation experiment is performed on the deep brain region rGPi of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 3 As shown.

[0095] Example 6 This embodiment proposes a method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning, which is the same as that in Embodiment 1.

[0096] When the number of electrodes to be activated is 32, the optimal electrode positions and current intensity values ​​for the 32 electrodes are obtained using the multi-electrode temporal interference stimulation parameter optimization method based on reinforcement learning proposed in this embodiment. Based on the optimal electrode positions and current intensity values ​​for the 32 electrodes to be activated, a simulation experiment is performed on the deep brain region rGPi of the individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 3 As shown.

[0097] Comparative Example 1 When the number of electrodes to be activated is 16, the optimal electrode positions and current intensities for these 16 electrodes are obtained using the traditional genetic algorithm (GA). Based on these optimal electrode positions and current intensities, a simulation experiment is conducted on the deep brain region rGPi of an individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 4 As shown in (b) of the diagram.

[0098] Comparative Example 2 When the number of electrodes to be activated is 16, the optimal electrode positions and current intensities for these 16 electrodes are obtained using the traditional genetic algorithm USNN. Based on these optimal electrode positions and current intensities, simulation experiments are conducted on the deep brain region rGPi of an individualized head finite element model to obtain the simulation results of the TIS envelope electric field distribution, such as... Figure 4 As shown in (c) in the figure.

[0099] Table 1. PSR values ​​of different algorithms in simulation experiments from Figure 3 It is evident that the method proposed in this invention can flexibly control the number of activated electrodes. In stimulation of the right globus pallidus (rGPi), as the number of electrodes increased from 4 to 16, the stimulation focal head (measured by PSR value) showed a significant and sustained improvement (mean PSR increased from 1.04 to 1.35), and each increase from 4 to 8, 8 to 12, and 12 to 16 electrodes brought statistically significant improvements (p<0.05). This demonstrates the effectiveness and flexibility of this invention in improving stimulation accuracy using multiple electrodes.

[0100] from Figure 2As shown in Table 1, the proposed method (using 16 electrodes) exhibits an overwhelming advantage at the focal head compared to the traditional genetic algorithm (GA). In the rGPi stimulation task, the average PSR value of the RL method (1.35) is significantly higher than that of the GA (1.04), with a highly statistically significant difference (p=0.0002). This indicates that the proposed method has stronger global optimization capabilities and can find a better stimulation scheme.

[0101] Compared to advanced unsupervised neural network (USNN) methods, the method proposed in this invention achieves comparable performance in optimization. However, this invention has significant practical advantages: it achieves the same effect using only a few preset electrodes (e.g., 16), while USNN requires all available electrodes (e.g., 74); and this invention follows the conventional pattern of applying a single frequency current to each electrode, making hardware implementation simpler and easier to translate into clinical applications.

[0102] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing multi-electrode temporal interferometric stimulation parameters based on reinforcement learning, characterized in that, include: A neural network based on an encoder-decoder architecture is constructed as an agent. The agent includes: an input layer for randomly generating an initial input tensor based on the number of electrodes to be activated; an encoder for encoding the initial input tensor into a high-dimensional hidden state tensor; a position decoder for determining the electrode position information tensor of each electrode to be activated based on the hidden state tensor; and a current decoder for obtaining the current intensity value assigned to each electrode to be activated based on the hidden state tensor and the electrode position information tensor of each electrode to be activated. The current decoder includes multiple sequentially connected decoding units, a normal distribution sampling layer connected to the last decoding unit, and a first fully connected layer branch and a second fully connected layer branch connected in parallel between the decoding unit and the normal distribution sampling layer. The normal distribution sampling layer constructs a normal distribution based on the standard deviation of the current intensity values ​​output by the first fully connected layer branch and the mean of the current intensity values ​​output by the second fully connected layer branch, and samples from the normal distribution to generate the initial current intensity value for the electrode to be activated. A reinforcement learning algorithm is used to train the agent, resulting in a trained agent. The training process includes: inputting the number of electrodes to be activated into the agent to obtain the electrode positions and current intensity values; normalizing the current intensity values; inputting the electrode positions and normalized current intensity values ​​into the lead field matrix of a pre-constructed individualized head finite element model to calculate the temporal interference stimulus envelope electric field distribution of the whole brain, and calculating the current reward function based on it; subtracting the reward expectation baseline from the reward function to obtain the advantage function; calculating the corresponding log probability based on the advantage function; calculating the average of the log probabilities and using it as the loss function; determining whether the loss function has converged; if so, the trained agent is obtained; if not, the gradient of the loss function with respect to the agent's weights is calculated using the backpropagation algorithm, and the agent's weights are updated using the optimizer. The number of electrodes to be activated is then input into the updated agent for iteration until the loss function converges. The trained agent is used to optimally allocate electrode positions and current intensity values ​​based on the number of electrodes to be activated.

2. The method for optimizing multi-electrode temporal interferometric stimulation parameters based on reinforcement learning according to claim 1, characterized in that, The input layer, encoder, and position decoder are connected in sequence, and the outputs of the encoder and position decoder are concatenated and then connected to the current decoder.

3. The method for optimizing multi-electrode temporal interferometric stimulation parameters based on reinforcement learning according to claim 2, characterized in that, The encoder comprises multiple sets of coding units connected in sequence, each set of coding units comprising a first fully connected layer and a first GeLU activation function connected in sequence.

4. The method for optimizing multi-electrode temporal interferometric stimulation parameters based on reinforcement learning according to claim 3, characterized in that, The location decoder comprises a second fully connected layer, a first hyperbolic tangent layer, a first hyperbolic tangent adjustment layer, a softmax function, and a class distribution sampling layer connected in sequence.

5. The method for optimizing multi-electrode temporal interferometric stimulation parameters based on reinforcement learning according to claim 1, characterized in that, The decoding unit consists of a third fully connected layer and a second GeLU function layer connected in sequence.

6. The method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning according to claim 1, characterized in that, The first fully connected layer branch includes a fourth fully connected layer and a softplus layer connected in sequence; the second fully connected layer branch includes a reshaping layer, a fifth fully connected layer, a second hyperbolic tangent layer, and a second hyperbolic tangent adjustment layer connected in sequence; the inputs of the fourth fully connected layer and the reshaping layer are respectively connected to the output of the last decoding unit, and the softplus layer and the second hyperbolic tangent adjustment layer are respectively connected to the input of the normal distribution sampling layer.

7. The method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning according to claim 1, characterized in that, The baseline for expected rewards is determined by the formula. The calculation is performed; in the formula, Here, represents the discount factor, and represents the set of reward values. This indicates the operation of finding the maximum value. This indicates the operation of averaging.

8. The method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning according to claim 1, characterized in that, Before inputting the electrode locations and normalized current intensity values ​​into the pre-constructed lead field matrix of the individualized head finite element model to calculate the temporal interference stimulus envelope electric field distribution of the whole brain, the following steps are also included: An individualized head finite element model is constructed, and the leading field matrix of the individualized head finite element model is calculated.

9. The method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning according to claim 1, characterized in that, A personalized head finite element model is constructed, and the leading field matrix of the personalized head finite element model is calculated, including: Obtain head magnetic resonance imaging data from healthy subjects; Based on the magnetic resonance imaging data of the head of healthy subjects, a personalized head finite element model was constructed; the personalized head finite element model includes the scalp, skull, cerebrospinal fluid, gray matter and white matter; Assign a corresponding conductivity attribute to each tissue; 74 candidate electrode positions that meet preset standards are pre-defined on the scalp surface; By applying a unit current to each candidate electrode location and using a distal electrode as a reference, the electric field distribution generated by each candidate electrode in the whole brain was calculated by solving the quasi-static Laplace equation using the finite element method. Based on the individualized head finite element model and the electric field distribution of all 74 candidate electrodes, the lead field matrix is ​​obtained.

10. The method for optimizing multi-electrode temporal interference stimulation parameters based on reinforcement learning according to claim 1, characterized in that, The reward function is the peak-to-sum ratio; where, In the formula, PSR represents the peak-to-sum ratio. This represents the maximum envelope electric field amplitude calculated within the target region. This represents the average field strength of all voxels whose envelope electric field amplitude exceeds 90% of the maximum value in the non-target region.

11. A multi-electrode temporal interference stimulus parameter optimization system based on reinforcement learning, characterized in that, include: The intelligent agent construction module is used to construct a neural network based on an encoder-decoder architecture as an intelligent agent. The intelligent agent includes an input layer, an encoder, a position decoder, and a current decoder. The input layer, encoder, and position decoder are connected sequentially, and the outputs of the encoder and position decoder are concatenated and connected to the current decoder. The current decoder includes multiple decoding units, a first fully connected layer branch, a second fully connected layer branch, and a normal distribution sampling layer. The multiple decoding units are connected sequentially, with the last decoding unit connected to both the first and second fully connected layer branches. The first and second fully connected layer branches are connected to the input of the normal distribution sampling layer. The first fully connected layer branch outputs the mean of the current intensity value, the second fully connected layer branch outputs the standard deviation of the current intensity value, and the normal distribution sampling layer constructs a normal distribution based on the mean and standard deviation, samples from the normal distribution, and generates the initial current intensity of the electrode to be activated. The agent training module is used to train the agent to obtain a trained agent. The training process includes: obtaining the number of electrodes to be activated and inputting this number into the agent to obtain the electrode positions and current intensity values; normalizing the current intensity values; inputting the electrode positions and normalized current intensity values ​​into the lead field matrix of a pre-constructed individualized head finite element model to calculate the temporal interference stimulus envelope electric field distribution of the whole brain; calculating the current reward function based on the temporal interference stimulus envelope electric field distribution of the whole brain; subtracting the reward expectation baseline from the reward function to obtain the advantage function; calculating the corresponding log probability based on the advantage function; calculating the average of the log probabilities and using the average of the log probabilities as the loss function; determining whether the loss function has converged; if yes, the trained agent is obtained; if not, the gradient of the loss function with respect to the agent's weights is calculated using the backpropagation algorithm, and the agent's weights are updated using the optimizer. The number of electrodes to be activated is then input into the updated agent for iteration until the loss function converges. The allocation module is used to optimize the allocation of electrode positions and current intensity values ​​based on the number of electrodes to be activated using a trained agent.

Images