High-dimensional continuous control method and device based on RNA evolution adaptive maximum entropy

Abstract

The invention discloses a high-dimensional continuous control method and device based on RNA evolution adaptive maximum entropy. According to the method, through the population global search capability of RNA evolution, the exploration efficiency of high-dimensional continuous control tasks is improved, and the risk of falling into local optimum is reduced; through an adaptive entropy coefficient updating mechanism, the entropy coefficient is dynamically adjusted along with the training stage and the task dimension, and manual parameter adjustment is reduced; evolutionary search and maximum entropy gradient updating are cooperatively coupled, global optimization and local fine optimization are considered, and the convergence speed and stability are improved; and premature convergence is inhibited through diversity monitoring and a noise injection mechanism, so that the strategy robustness and generalization ability are improved.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, reinforcement learning and evolutionary computation technology, and specifically relates to a policy training method and device for high-dimensional continuous control scenarios such as robots. By combining RNA evolution mechanism with adaptive maximum entropy reinforcement learning, it achieves efficient exploration and stable convergence in high-dimensional continuous action space. Background Technology

[0002] High-dimensional continuous control tasks (such as multi-degree-of-freedom robots, continuous control simulation tasks, etc.) are usually characterized by high state dimension, high action dimension, sparse or multi-modal rewards, which leads to low exploration efficiency, easy getting trapped in local optima, unstable convergence and sensitivity to hyperparameters in traditional deep reinforcement learning methods during training.

[0003] Soft Actor-Critic (SAC) algorithms improve exploration by introducing an entropy regularization term into the objective function, but the entropy coefficient α typically relies on empirical parameter tuning or a fixed objective, making it difficult to adapt to different training stages and tasks with different dimensions. Evolutionary algorithms possess global optimization capabilities through population search, but lack the ability to utilize gradient information for fine-grained policy improvement.

[0004] Existing maximum entropy actor-critic class methods (such as SAC) improve exploration by introducing policy entropy into the objective function, but they mostly use fixed or empirically set entropy coefficients α, which are difficult to adaptively adjust with the training phase and action dimension. When environmental rewards are sparse or have high dimensionality, problems such as insufficient exploration, unstable convergence, and sensitivity to hyperparameters may still occur.

[0005] Existing proximal update methods based on policy gradients (such as PPO) improve training stability by truncating or constraining update magnitudes, but they often require fine-tuning in high-dimensional continuous control tasks and are still prone to falling into suboptimal policies when rewards are multi-peaked or there are many local optima.

[0006] Existing evolutionary reinforcement learning / neuroevolutionary methods possess a certain global optimization capability through population search, but lack the utilization of gradient information, resulting in low sample efficiency. If evolutionary search and gradient update are simply chained together or alternately executed, a coupling regulation mechanism between the intensity of evolutionary search and the training phase (or policy entropy level) is usually not established, leading to instability in the overall optimization process and sensitivity to hyperparameters.

[0007] In summary:

[0008] (1) In high-dimensional continuous action space, traditional reinforcement learning strategies are insufficient in exploration and are prone to getting trapped in local optima.

[0009] (2) The problem that the entropy regularization coefficient is difficult to adaptively adjust in different task dimensions and training stages.

[0010] (3) The fact that evolutionary search and gradient update are independent of each other leads to problems such as low training efficiency and unstable convergence.

[0011] (4) The decline in population diversity leads to the problem of premature convergence.

[0012] Therefore, there is an urgent need for a technical solution that can synergistically integrate "global search capability (evolution)" and "local gradient optimization capability (reinforcement learning)" and adaptively balance exploration and utilization, so as to improve the training efficiency and stability of high-dimensional continuous control tasks. Summary of the Invention

[0013] To address the shortcomings of existing technologies, this invention provides a high-dimensional continuous control method and apparatus based on RNA evolutionary adaptive maximum entropy. The method is executed on a computing device, which includes at least a processor, a memory, and an interface for interacting with the controlled object / simulation environment.

[0014] In a first aspect, embodiments of this application provide a high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy, comprising the following steps:

[0015] S1: Initialize the policy population and perform RNA encoding: Generate N policy network individuals, flatten and normalize the parameter vector of each individual in sequence, and map it to an RNA sequence; when evaluation or gradient update is required, decode the RNA sequence to restore the parameter vector and load it into the policy network.

[0016] S2: Fitness Evaluation: Each individual in the policy network interacts with the controlled object or the simulation environment to obtain a trajectory, and calculates the cumulative reward as the individual's fitness.

[0017] S3: Evolutionary Update: Based on fitness, the population is ranked, and an elite set E is selected. Selection, crossover, and base editing mutations are performed on non-elite individuals to generate a new generation of RNA sequences. The new sequences are decoded to obtain parameter vectors, forming a new generation of strategic individuals. The mutation probability is adaptively determined based on both the RNA folding free energy index and the relative fitness level.

[0018] S4: Adaptive Maximum Entropy Reinforcement Learning: Perform gradient updates based on the maximum entropy objective on the elite individual set E (or individuals extracted from E) to update the policy network and evaluation network; and adaptively update the entropy coefficients according to the target entropy in each gradient update to achieve an exploration-exploitation balance.

[0019] S5: Diversity monitoring and noise injection: Calculate the population diversity index D. If D is below the threshold, randomly select M individuals from non-elite individuals or select them according to their diversity contribution. Inject Gaussian noise into their complete parameter vectors and prune and re-evaluate them to restore diversity.

[0020] S6: Iterative Loop. Repeat steps S2-S5 until the convergence condition is met or the preset number of algebras is reached, and output the trained policy network for online control of the robot / controlled object.

[0021] In one possible implementation, the RNA encoding includes normalizing network parameters to a preset interval and mapping them into four symbols: A / U / C / G. The decoding includes mapping the symbols to their corresponding numerical intervals and assembling the recovered values ​​into a parameter vector θ_i in a preset order (where θ_i is the reconstruction result of the policy network parameter vector or its flattened scalar sequence {W_k}).

[0022] Preferably, the recovered value uses a representative value (interval center value) of the corresponding interval to ensure decoding determinism; when the exploration enhancement trigger condition is met, the recovered value is changed to a sampled value obtained by random sampling within the corresponding interval, wherein the exploration enhancement trigger condition includes: the population diversity index D is lower than the threshold D_min. Subsequently, the components of each dimension of the parameter vector θ_i are clipped using θ_i←clip(θ_i,θ_min,θ_max) to satisfy the effective range constraint; wherein the random sampling process uses a fixed random seed or a recorded seed to ensure repeatability.

[0023] In one possible implementation, the folding free energy index ΔG is calculated using the minimum free energy MFE.

[0024] In one possible implementation, the folding free energy index ΔG is calculated using an approximate stability index based on complementary pairing statistics, with a time complexity of O(L), where L is the length of the RNA sequence.

[0025] In one possible implementation, the mutation probability P_mut is determined through a two-stage coupling: first, the stability factor η is obtained based on the folding free energy index ΔG, and then the exploration coefficient c is obtained based on the relative fitness level. P_mut = clip(base_mut × η × c, P_min, P_max), where base_mut is the baseline mutation rate, and P_min and P_max are the lower and upper limits of the mutation probability, respectively.

[0026] In one possible implementation, the base editing variation includes at least base substitution, insertion, and deletion; the crossover includes at least single-point crossover, multi-point crossover, or complementary pairing crossover.

[0027] In one possible implementation, adaptive maximum entropy reinforcement learning includes maximum entropy policy optimization and adaptive update of entropy coefficients, used to achieve the maximum entropy objective optimization of the policy network and the adaptive adjustment of the entropy coefficient α. A soft actor-commentator structure is employed, and the policy optimization objective is to maximize the sum of the cumulative reward and the entropy term.

[0028] The critic network is updated by minimizing the soft Bellman error.

[0029] The actor network is updated using reparameterization techniques.

[0030] The entropy coefficient α is used to adjust the weight of the entropy regularization term, making the current policy entropy approximate the target entropy H_target. Preferably, α is updated by minimizing the temperature objective function.

[0031] In one possible implementation, the target entropy H_target = -dim(A), where dim(A) is the action dimension.

[0032] In one possible implementation, the online control is executed within a fixed control period Δt. The continuous control quantity output by the strategy network is limited and constrained to form a control command, which is then sent to the actuator driver via an EtherCAT, CAN, or RS485 communication interface.

[0033] Secondly, embodiments of this application also provide a high-dimensional continuous control device based on RNA evolution adaptive maximum entropy, which includes: a population initialization module, a fitness evaluation module, an evolutionary operation module, an adaptive maximum entropy reinforcement learning module, a diversity monitoring module, and a strategy output and deployment module; each module can be implemented by software, hardware, or a combination of software and hardware, and deployed in a computer device or an edge controller.

[0034] The population initialization module is used to generate an initial policy population, generate N policy network individuals, and flatten and normalize the parameter vector of each individual in sequence before mapping it to an RNA sequence.

[0035] The fitness evaluation module is used to obtain the trajectory of each individual in the policy network through interaction with the controlled object or simulation environment, and to calculate the cumulative reward as the individual's fitness.

[0036] The evolutionary operation module is used to select an elite set E of individuals based on fitness, and to perform selection, crossover, and base editing mutations on non-elite individuals to generate a new generation of RNA sequences; the new sequences are decoded to obtain parameter vectors and form a new generation of strategic individuals. The mutation probability is adaptively determined based on the RNA folding free energy index and the relative fitness level.

[0037] The adaptive maximum entropy reinforcement learning module is used to perform gradient updates based on the maximum entropy objective on the elite individual set E (or individuals extracted from E), update the policy network and the evaluation network; and in each gradient update, the entropy coefficient is adaptively updated according to the target entropy to achieve an exploration-exploitation balance.

[0038] The diversity monitoring module is used to calculate the population diversity index D. If D is lower than the threshold, M individuals are randomly selected from non-elite individuals or selected according to their diversity contribution. Gaussian noise is injected into their complete parameter vectors, and they are pruned and re-evaluated to restore diversity.

[0039] The strategy output and deployment module is used to control the fitness evaluation module, evolutionary operation module, adaptive maximum entropy reinforcement learning module and diversity monitoring module to perform repeated iterative training until the convergence condition is met or the preset number of generations is reached, and output the trained policy network for online control of the robot / controlled object.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] (1) This invention improves the exploration efficiency of high-dimensional continuous control tasks and reduces the risk of getting trapped in local optima by utilizing the global search capability of RNA evolution populations.

[0042] (2) The present invention uses an adaptive entropy coefficient update mechanism to dynamically adjust α with the training stage and task dimension, thereby reducing manual parameter tuning.

[0043] (3) This invention couples evolutionary search with maximum entropy gradient update, taking into account both global optimization and local fine optimization, thereby improving convergence speed and stability.

[0044] (4) This invention suppresses premature convergence through diversity monitoring and noise injection mechanisms, thereby improving the robustness and generalization ability of the strategy.

[0045] (5) In a high-dimensional continuous control benchmark task in one embodiment, the average reward is improved by about 20% compared with SAC-type methods and by about 28% compared with PPO-type methods. The number of interaction steps required to reach the same reward threshold is reduced by about 25%-30%, thereby obtaining a better control strategy under the same training time / computing power conditions. In an embodiment that targets trajectory tracking error and energy consumption, the control error and energy consumption can be further reduced and the stability can be improved.

[0046] Differences from the closest existing technology and unpredictable technical effects

[0047] Compared with the closest prior art, the present invention has at least the following differences: (1) the strategy network parameters are encoded with RNA base sequences, and the RNA folding free energy ΔG is introduced as a stability characterization to guide the mutation probability and editing operation intensity, thereby achieving directional exploration in high-dimensional parameter space; (2) the entropy coefficient α is adaptively updated under the maximum entropy target, and the phased exploration intensity is adjusted in combination with the target entropy threshold range to avoid insufficient exploration or training oscillation caused by fixing α; (3) a population diversity index D and a threshold D_min are set, and when diversity decays, a noise injection and resampling mechanism is triggered to suppress premature convergence. The above differences are not simply superimposed, but are coupled and coordinated in the same training cycle through "ΔG-guided evolutionary search - maximum entropy gradient optimization - diversity recovery", so that faster convergence and higher stability can be obtained simultaneously under the same training budget, which is an unpredictable technical effect.

[0048] Further Evidence on the Innovativeness and Technological Effectiveness of "RNA Encoding"

[0049] To avoid misinterpreting RNA four-symbol encoding as a simple replacement for binary / Gray code, the technical connotation and unpredictable technical effects of "RNA encoding" in this invention are further explained below:

[0050] (1) Structure-aware encoding-evaluation coupling: RNA encoding is not only used to carry network parameter sequences, but also serves as the input carrier for calculating the structural stability index ΔG. ΔG is used to adaptively adjust the mutation intensity, transforming the evolutionary search from "random perturbation" to "directed perturbation under stability constraints", thereby suppressing ineffective large perturbations and increasing the proportion of effective explorations in the high-dimensional parameter space.

[0051] (2) Sequence-level editing operators and continuity preservation: This invention defines base editing operators such as insertion / deletion / replacement at the RNA sequence level and combines them with complementary pairing crossover to form controllable genetic operations. By mapping each base to a representative value (or sampled value within the interval) of a preset continuous interval through decoding rules, it can be ensured that the parameter change caused by a single base substitution is limited to the upper bound of the interval width, thereby maintaining the "smoothness / continuity" of parameter perturbation in a statistical sense. In contrast, binary / Gray code usually uses bit flipping as the basic operator, and its Hamming distance lacks a stable correspondence with the parameter perturbation amplitude, which can easily cause uncontrolled parameter jumps in high-dimensional scenarios.

[0052] (3) Unpredictable synergistic effects: The above (1) and (2) make ΔG a unified coupling quantity of “encoding-operation-evaluation”, which works in synergy with the adaptive maximum entropy α update and diversity recovery mechanism in the same training cycle. This results in faster convergence and higher stability under the same training budget, rather than just obtaining algorithmic results with reward improvement.

[0053] (4) Comparison and verification suggestion: In the embodiments, “encoding mode comparison” ablation can be added, that is, the RNA four-symbol encoding is replaced with binary or Gray code encoding, while maintaining the same number of evolution operators and training budget, and the convergence speed, diversity decay and control indicators (such as trajectory RMSE, energy consumption, instability rate) are compared to further prove the necessity and technical effect of the RNA encoding-ΔG coupling mechanism described in this invention. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the device structure according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram illustrating the RNA coding, crossover, and base editing mutation mechanisms in an embodiment of the present invention.

[0056] Figure 3 This is a flowchart of the adaptive entropy coefficient α update and threshold adjustment in an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of the population diversity index D calculation and noise injection triggering mechanism in an embodiment of the present invention.

[0058] Figure 5 This is a performance comparison diagram for a typical continuous control task. Detailed Implementation

[0059] For ease of understanding, definitions of commonly used symbols in this manual are provided:

[0060] S: State space; A: Action space; P: State transition probability; R: Reward function. These four tuples together constitute a Markov decision process (MDP).

[0061] s_t: State at time t; a_t: Action at time t; r_t: Immediate reward obtained after executing a_t; γ: Discount factor, 0 < γ ≤ 1.

[0062] π_θ(a|s): The probability density / distribution of the policy (actor network) outputting action a in state s, represented by parameter θ; Q_φ(s,a): The action value function (commentator / evaluation network), represented by parameter φ.

[0063] H(π(·|s)): The entropy of the policy in state s, used to characterize the randomness of the policy; for a continuous action space, it can be written as H(π(·|s)) = -E_{a~π}[log π_θ(a|s)].

[0064] α: Entropy coefficient (temperature coefficient), used to balance cumulative reward and entropy regularization term.

[0065] H_target: Target entropy threshold, preferably set to -dim(A), where dim(A) is the action dimension.

[0066] N: Population size; G_max: Maximum number of generations.

[0067] f_i: Fitness (cumulative reward) of the i-th individual; f_avg: Average fitness of the population.

[0068] D: Population diversity index; D_min: Diversity lower limit threshold; M: Number of individuals injected with noise; σ: Noise standard deviation.

[0069] q_i: The RNA sequence corresponding to the i-th individual.

[0070] ΔG_i: The folding free energy index (scalar) of the RNA sequence q_i of the i-th individual, which serves as the input for the mutation probability determination process.

[0071] ΔG_MFE(q_i): The free energy value calculated based on the Minimum Free Energy (MFE) model.

[0072] ΔG_app(q_i): The estimated free energy calculated based on the approximate stability model.

[0073] Preferred setting ΔG_i = ΔG_MFE(q_i), or in an alternative implementation, let ΔG_i = ΔG_app(q_i).

[0074] If the free energy difference is required, then define Δ(ΔG)_i=ΔG_i−ΔG_i,ref, where ΔG_i,ref is the reference free energy value.

[0075] P_mut: mutation probability; base_mut: baseline mutation rate; P_min, P_max: lower and upper bounds of mutation probability; η: stability factor calculated from ΔG; c: exploration coefficient calculated from fitness relative level; clip(·): clipping function.

[0076] p_insert, p_delete, p_replace: base editing probabilities for insertion, deletion, and replacement; base_mut: base mutation probability.

[0077] This application provides a high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy, including the following steps:

[0078] S1: Initialize the policy population and perform RNA encoding: Generate N policy network individuals, flatten and normalize the parameter vector θ_i of each individual in sequence, and map it to an RNA sequence q_i composed of four symbols: A, U, C, and G; when evaluation or gradient update is required, decode the RNA sequence q_i to restore the parameter vector θ_i and load it into the policy network.

[0079] S2: Fitness Evaluation: Each individual in the policy network interacts with the controlled object or the simulation environment to obtain the trajectory τ_i={(s_t,a_t,r_t)}, and calculates the cumulative reward as the individual's fitness f_i; where the cumulative reward can be a discounted reward f_i=Σ_t γ^t·r_t, and can be further superimposed with constraint penalty terms to reflect energy consumption, amplitude limit or safety constraints.

[0080] S3: Evolutionary Update: Based on fitness, the population is ranked, and an elite set E is selected. Selection, crossover, and base editing mutations are performed on non-elite individuals to generate a new generation of RNA sequences. The new sequences are decoded to obtain parameter vectors, forming a new generation of strategic individuals. The mutation probability P_mut is adaptively determined based on the RNA folding free energy index ΔG and the relative fitness level.

[0081] S4: Adaptive Maximum Entropy Reinforcement Learning: Perform gradient updates based on the maximum entropy objective on the elite individual set E (or individuals extracted from E), updating the policy network π_θ and the evaluation network Q_φ; and in each gradient update, adaptively update the entropy coefficient α according to the target entropy H_target to achieve an exploration-exploitation balance.

[0082] S5: Diversity monitoring and noise injection: Calculate the population diversity index D. If D is lower than the threshold D_min (or decreases for several consecutive generations and is lower than the threshold), randomly select M individuals from non-elite individuals or select them according to their diversity contribution. Inject Gaussian noise into their complete parameter vectors and prune and re-evaluate them to restore diversity.

[0083] S6: Iterative Loop. Repeat S2-S5 until the convergence condition is met or the preset number of algebras G_max is reached, and output the trained policy network for online control of the robot / controlled object.

[0084] System / Device Examples

[0085] like Figure 1 As shown, this embodiment also provides an apparatus for training high-dimensional continuous control strategies for robots, which is deployed on a computing device. The computing device includes a processor, a memory, and an interface for communicating with a simulation environment / real robot.

[0086] The device includes:

[0087] (a) Population initialization module: used to generate the initial policy population, generate N policy network individuals, flatten and normalize the parameter vector of each individual in order and map it to an RNA sequence.

[0088] (b) Fitness evaluation module: used to obtain the trajectory of each individual in the policy network by interacting with the controlled object or the simulation environment, and calculate the cumulative reward as the individual fitness.

[0089] (c) Evolutionary Operation Module: This module is used to select an elite set E of individuals based on fitness, and to perform selection, crossover, and base editing mutations on non-elite individuals to generate a new generation of RNA sequences; it decodes the new sequences to obtain parameter vectors and forms a new generation of strategic individuals. The mutation probability is adaptively determined based on the RNA folding free energy index and the relative fitness level.

[0090] (d) Adaptive maximum entropy reinforcement learning module: used to perform gradient updates based on the maximum entropy objective on the elite individual set E (or individuals extracted from E), update the policy network and evaluation network; and adaptively update the entropy coefficient according to the target entropy in each gradient update to achieve exploration-exploitation balance.

[0091] (e) Diversity monitoring module: used to calculate the population diversity index D. If D is lower than the threshold, M individuals are randomly selected from non-elite individuals or selected according to their diversity contribution. Gaussian noise is injected into their complete parameter vectors and they are pruned and re-evaluated to restore diversity.

[0092] (f) Policy output and deployment module: used to output the trained policy network parameters and send them to the controlled object for execution.

[0093] System integration implementation example (real robot / actuator control link)

[0094] To further demonstrate that this invention is a technical solution that can be implemented in a real control system, this embodiment provides a system integration and deployment method. The training phase is completed in a simulation environment or a digital twin environment; during the deployment phase, control commands are output on the robot controller or edge computing device at a fixed control cycle to drive the actuators to form a closed-loop control.

[0095] (1) Hardware and software composition: including industrial PC / embedded computing unit (processor, memory), real-time controller (which can be RTOS or real-time kernel), sensor interface (such as encoder, IMU, force / torque sensor), actuator drive interface (such as EtherCAT, CAN, RS485 or PWM / analog interface) and network communication module.

[0096] (2) Data and signal flow: Sensor data is collected and a state vector is formed in each control cycle Δt (preferably 1ms–20ms). After state preprocessing (including filtering, normalization and coordinate transformation), the state vector s_t is obtained and the state vector s_t is input into the policy network π_θ to output action a_t.

[0097] In a preferred embodiment, the policy network π_θ is the parameter θ obtained and fixed during the training phase. * The trained network is used for deployment / online control.

[0098] In one optional implementation, the policy network π_θ is updated at a preset interval T during online operation. u The system receives parameter updates from the training module (i.e., training and deployment simultaneously), and both updates are verified by the safety constraint and motion limiting module to ensure control stability.

[0099] The strategy network outputs a continuous control quantity a_t (joint torque, desired speed, or current given value), which, after being limited and constrained by safety constraints (maximum torque / speed / acceleration constraints, joint limit, temperature rise / current protection), generates a control command u_t, which is sent to the servo driver / motor through the drive interface; after the actuator responds, the system enters the next cycle and feeds back the new sensor status.

[0100] (3) Example of system integration method: The policy network runs as a ROS node / middleware service (such as policy_node), subscribes to sensor topics or shared memory ( / joint_states, / imu, / ft_sensor), and publishes control command topics ( / cmd_torque or / cmd_vel); the real-time controller provides safety monitoring and constraint projection, and maps the final control command to the drive interface (EtherCAT PDO, CAN message or PWM duty cycle).

[0101] (4) Technical effects of online execution: Compared with the comparative scheme with fixed entropy coefficient or no evolutionary mechanism, the present invention can obtain a more robust control strategy under the same sampling budget / computing power constraints, thereby reflecting improvements in trajectory tracking error (e.g., RMSE), energy consumption (energy per unit task), and instability / fall rate in engineering indicators. The above technical effects correspond to the synergistic closed loop of "ΔG-guided evolutionary search - adaptive maximum entropy gradient optimization - diversity restoration".

[0102] (5) Safety and feasibility: To ensure the safety of online control, the strategy output can adopt a two-layer protection: the first layer is the action limit and constraint projection; the second layer is the monitoring module to detect abnormalities (such as over-temperature, over-current, attitude overturning risk) and switch to the safety controller or preset backup strategy.

[0103] Feasible calculation methods for RNA folding free energy ΔG:

[0104] To avoid insufficient disclosure due to "ambiguity in ΔG calculation methods," this invention provides at least two implementable ΔG calculation methods; one or a combination can be selected and used according to computing power and real-time requirements. ΔG is only used to adjust the mutation intensity and does not change the definition of fitness f_i.

[0105] In a preferred embodiment, the RNA folding free energy is calculated using the minimum free energy (MFE): for the RNA sequence q_i at a preset temperature parameter T, the MFE free energy ΔG_MFE(q_i) is calculated based on a preset set of thermodynamic parameters. When the sequence length is large, it is preferable to divide q_i into several subsequences q_i^m according to the window length L0, calculate ΔG_MFE(q_i^m) for each subsequence, and weight them with weight w_m to obtain the free energy index of individual i.

[0106] ΔG_i = ΔG_MFE(q_i),=Σ_m w_m·ΔG_MFE(q_i^m)

[0107] Among them, w_m can be 1 or preset according to the importance of the parameter group.

[0108] In an alternative implementation, ΔG_i is calculated using an approximate stability model. = ΔG_app(q_i).

[0109] Where ΔG_app(q_i) is the estimated free energy calculated based on the approximate stability model.

[0110] In either approach, ΔG_i is input into the two-stage mutation probability determination process to form the stability factor η_i, and clipping is used to ensure that the mutation probability P_mut is within a controllable range, thereby ensuring that the algorithm is feasible and does not introduce uncontrollable computational burden.

[0111] RNA encoding and decoding examples

[0112] (1) Mapping of parameter vectors to RNA sequences: The strategy network parameter vector θ_i is flattened into a real number sequence {w_k}_k=1^K in a preset order. Here, w_k represents the k-th scalar parameter (including network weights and bias terms) after flattening θ_i, and K is the total number of parameters in θ_i. Each w_k is normalized to the interval [-1,1] (or linearly mapped to [-1,1] according to the preset upper and lower bounds [w_min,w_max]), and then discretized and mapped to the four base sets {A,U,C,G}.

[0113] The preferred discretization method is as follows: if w_k∈[-1,-0.5) is mapped to A; w_k∈[-0.5,0) is mapped to U; w_k∈[0,0.5) is mapped to C; and w_k∈[0.5,1] is mapped to G. The above thresholds can also be adaptively set according to quantiles to improve representation accuracy.

[0114] (2) Decoding: The bases in the RNA sequence are mapped to interval center values ​​or random sampled values ​​according to the inverse rule, and restored to the parameter vector θ_i, which is used to construct the policy network. The inverse rule is the inverse mapping of the above interval mapping, that is, mapping the bases b_k∈{A,U,C,G} back to the corresponding numerical interval. It is preferred to use the interval center value as the deterministic restoration value to ensure that the decoding result of the same sequence is unique; when it is necessary to introduce additional perturbation to enhance the exploration, the restoration value can be randomly sampled in the corresponding interval according to a uniform distribution, and reproducibility can be ensured by fixing the random seed.

[0115] (3) Validity verification: After decoding, θ_i is pruned or regularized (after decoding to obtain the parameter vector θ_i, it is preferred to process it according to the preset physical / engineering constraint range [θmin, θmax]: when the parameter has clear upper and lower bound constraints, element-wise pruning θi←clip(θi, θmin, θmax) is adopted; when the upper and lower bounds of the parameter are not easy to give precisely, L2 regularization or norm normalization θi←θi / max(1,||θ_2|| / c) can be adopted to suppress excessively large values) to ensure that the parameter is within the preset valid range; if numerical abnormalities (NaN / Inf) or constraints are still present after the above processing, it is preferred to roll back to the parent parameter; if the number of consecutive rollbacks exceeds the threshold R_max, the individual is resampled (with a fixed random seed to control reproducibility) to maintain the continuity of the parameter space and the stability of the algorithm.

[0116] Adaptive determination of evolutionary operations and mutation probabilities (e.g.) Figure 2 (As shown)

[0117] (1) Selection: Sort the population according to fitness f_i, select the top K elite individuals and keep them directly to the next generation; non-elite individuals obtain their parents through tournament selection.

[0118] (2) Crossover: Perform complementary pairing crossover or double-strand crossover on the selected parent RNA sequence to generate daughter sequences in order to maintain coding continuity.

[0119] (3) Mutation (base editing): Insert a new base into the RNA sequence with probability p_insert, delete a base with probability p_delete, and replace a base with probability p_replace; these probabilities can adaptively change with the generation g. For example, increase p_insert and p_replace in the early stage of evolution to enhance exploration, and decrease them in the later stage of evolution to enhance convergence.

[0120] The mutation of the RNA sequence is described by "two-layer probabilities". The first layer determines whether the i-th individual performs mutation in the current generation with the mutation trigger probability P_mut; the second layer, under the condition of triggering mutation, performs insertion / deletion / replacement operations according to the editing type probability distribution. Preferably, let: p_insert + p_delete + p_replace = 1; and when mutation is triggered, use p_insert, p_delete, p_replace as conditional probabilities to select specific editing operators; when mutation is not triggered, keep the sequence unchanged.

[0121] (4) Two-stage determination of the mutation probability P_mut:

[0122] The first stage: Calculate the structure stability factor η_i = exp(-ΔG_i / (kT)) based on the RNA folding free energy difference ΔG_i, and normalize it to [η_min, η_max].

[0123] The second stage: Determine the exploitation / exploration coefficient c_i according to the relative fitness level: if f_i ≥ f_avg, then c_i = c_good (preferably 0.5); if f_i < f_avg, then c_i = c_bad (preferably 2).

[0124] The mutation probability of the i-th individual is taken as P_mut,i = clip(base_mut × η_i × c_i, P_min, P_max), where clip(·) represents clipping to the [P_min, P_max] interval, and P_min and P_max are preset upper and lower limits (such as 0.001 and 0.5).

[0125] Adaptive maximum entropy reinforcement learning and entropy coefficient update

[0126] In this embodiment, the adaptive maximum entropy reinforcement learning module, also known as the maximum entropy policy optimization and entropy coefficient adaptive update module, includes a maximum entropy policy optimization module and an entropy coefficient adaptive update module, which are used to achieve the maximum entropy objective optimization of the policy network and the adaptive adjustment of the entropy coefficient α. The adaptive maximum entropy reinforcement learning module adopts a soft actor-critic structure, and the policy optimization objective is to maximize the sum of the cumulative reward and the entropy term:

[0127] J(π) = E[ Σ_{t=0}^{∞} ( r_t + α · H(π(·|s_t)) ) ]

[0128] Where E represents the expectation of the state-action trajectory distribution sampled during training; H(π(·|s_t)) is the entropy of the policy in state s_t.

[0129] The critic network updates by minimizing the soft Bellman error:

[0130] L(φ) = E[ ( Q_φ(s_t,a_t) - y_t )^2 ]

[0131] Where Q_φ(s_t,a_t) is the action value function characterized by parameter φ; s_t: the state at time t; a_t: the action at time t.

[0132] y_t = r_t + γ( min(Q'_1(s_{t+1},a_{t+1}), Q'_2(s_{t+1},a_{t+1})) - αlog π_θ(a_{t+1}|s_{t+1}) )

[0133] Where Q'_1 and Q'_2 are the target commentator network outputs, respectively composed of the target parameters. , Characterization; soft updates are preferred for the target parameters: .

[0134] The actor network is updated using reparameterization techniques (such as a Gaussian strategy), and its gradient can be written as:

[0135] ∇J(θ)=E[∇_θlogπ_θ(a_t|s_t)·(Q_φ(s_t,a_t)-αlogπ_θ(a_t|s_t))]

[0136] Where π_θ(a_t|s_t) is the probability density / distribution of the policy (actor network) outputting action a_t in state s_t, characterized by parameter θ.

[0137] The entropy coefficient α is used to adjust the weight of the entropy regularization term, making the current policy entropy approximate the target entropy H_target. Preferably, α is updated by minimizing the temperature objective function:

[0138] J(α) = [ α · ( -log π_θ(a_t|s_t) - H_target ) ]

[0139] Where D is the experience replay pool, and π_θ is the policy network. Gradient descent is preferred for updating α.

[0140]

[0141] Where λ α The learning rate is the entropy coefficient. The target entropy H_target can be preset to a value related to the action dimension, for example, H_target = -dim(A), where dim(A) is the action dimension.

[0142] Diversity monitoring and noise injection rules

[0143] To avoid premature convergence of the population, this embodiment monitors population diversity. Let θ_i be the parameter vector of the i-th individual, then the diversity index D is defined as:

[0144] D = (2 / (N(N-1))) · Σ_{i <j} || θ_i - θ_j ||_2

[0145] Noise injection is triggered when D < D_min (e.g., 0.1) or when the G_drop generations show a decreasing trend and fall below the threshold: M individuals are randomly selected, and their parameters are updated to θ_i ← θ_i + ε, where ε ~ N(0, σ^2 I), and σ is preferably 0.01. After noise injection, parameter range pruning and necessary re-evaluation are required.

[0146] Application Example (High-Dimensional Continuous Control of Robots)

[0147] Taking multi-degree-of-freedom robot control as an example, the robot controller collects the state vector s_t (containing sensor information such as joint angle, angular velocity, force / torque, etc.) in each control cycle, and the policy network outputs the continuous control quantity a_t (joint torque or desired speed), which drives the robot to move through the actuator.

[0148] The training phase can be carried out in a simulation environment (such as MuJoCo), and the reward function r_t comprehensively reflects indicators such as trajectory tracking error, energy consumption and stability. After training, the policy network parameters are deployed to the robot controller, and control commands are output in real time, thereby achieving stable control in a high-dimensional continuous motion space.

[0149] Experimental setup

[0150] In one embodiment, a high-dimensional continuous control benchmark task is selected as the controlled object / simulation environment. This high-dimensional continuous control benchmark task environment includes, but is not limited to, continuous control tasks such as Humanoid, Ant, HalfCheetah, Hopper, and Walker2d (with a continuous real-valued vector action space). Humanoid and Ant tasks have higher action dimensions, used to verify the convergence speed and stability of the present invention in high-dimensional continuous control scenarios. The state vector s_t represents environmental observations (e.g., joint angles, angular velocities, contact information, etc.), and the action is a continuous torque vector a_t. The baseline for comparison includes at least: maximum entropy actor-critic methods (e.g., SAC), proximal policy optimization methods based on policy gradients (e.g., PPO), and evolutionary reinforcement learning variants without RNA folding free energy ΔG guidance. Comparative evaluations are performed under the same training budget, network structure, and evaluation protocol. Evaluation metrics include average cumulative reward, number of interaction steps required to reach a preset reward threshold (convergence steps), reward variance (stability), and tracking error and energy consumption under selectable real control links.

[0151] Comparative Examples and Technical Effects

[0152] In the comparative embodiments, the method of the present invention achieves a higher average cumulative reward and a smaller reward variance compared to the baseline with the same number of interaction steps, demonstrating a more stable training process. In some tasks, the present invention significantly reduces the number of interaction steps required to reach the same reward threshold, demonstrating higher sample efficiency. The above effects mainly come from: ① the constraint and guidance of the RNA folding free energy ΔG on the mutation intensity, improving the effectiveness of evolutionary search; ② the adaptive update of the entropy coefficient α, which dynamically adjusts the exploration intensity according to the training stage and action dimension; ③ the diversity monitoring and recovery mechanism, which inhibits premature convergence of the population.

[0153] In the ablation examples, ΔG guidance can be removed (using only a fixed mutation rate), α adaptive update can be removed (using a fixed α or a fixed target entropy), and diversity recovery can be removed (without noise injection), while keeping the other settings unchanged. Comparative results show that removing any of these key mechanisms leads to a decrease in average reward or an increase in the number of convergence steps. Specifically, removing ΔG guidance typically significantly reduces the effective exploration quality during the evolutionary phase; removing α adaptive update requires retuning for different task dimensions; and removing diversity recovery results in premature convergence and increased performance volatility in the later stages of training. The above ablation comparisons are used to establish a causal relationship between "technical features" and "technical effects."

[0154] It should be understood that the above comparison and ablation data can be used as example results; in the actual submission and review process, the corresponding training logs, model weights and evaluation scripts can be provided for review as required by the reviewer, so as to support the creativity and beneficial effects.

[0155] Task / Scenario Indicators (Examples) Method of the present invention Comparison methods (example) HalfCheetah-v2 Average reward 5400 SAC: 4500; PPO: 4200 Walker2d-v2 Number of interaction steps required to reach the threshold reward Reduced by approximately 25%-30% SAC: Baseline; PPO: Increase of approximately 10%-20% General (Multitasking) Stability / Premature convergence More stable after diversity triggers recovery The comparison method is more susceptible to hyperparameter effects.

[0156] (3) Third-party platform verification (optional): reproduce and output evaluation reports / ranking records under a public benchmark platform or third-party evaluation framework; or review the same evaluation protocol and issue verification records by a credible third-party institution / laboratory.

[0157] The aforementioned materials are intended to demonstrate that the technical effects described in the specification, such as "increased rewards, reduced convergence steps, and improved stability," are reproducible and verifiable. Even if all materials are not disclosed on the application date, they can be submitted during the examination process as required by the examiner to support the technical effects and the demonstration of inventiveness.

[0158] Optional Implementation Gradient

[0159] To improve the applicability of the present invention under different engineering constraints, the present invention provides the following optional implementation gradient; each of the following items can be used alone or in any combination.

[0160] (1) Calculate the gradient of ΔG: It is preferred to use the RNA folding free energy MFE calculation based on dynamic programming; when computing power is limited or real-time requirements are high, the approximate stability index (O(L)) based on complementary pairing statistics can be used instead; or a window segmentation weighting method can be used to compromise between accuracy and computational cost.

[0161] (2) Encoding / decoding gradient: A / U / C / G to parameter intervals can be bidirectionally mapped by interval partitioning mapping, table lookup mapping or random sampling mapping within the interval; preferably, the decoded parameters are pruned and normalized to meet numerical stability and physical constraints.

[0162] (3) Evolutionary operation gradient: Mutation operations can include base substitution, insertion and deletion; crossover operations can include single-point crossover, multi-point crossover and complementary pairing crossover; selection strategies can include elite retention and roulette wheel selection. To avoid disrupting stable structures, the insertion and deletion probabilities can be reduced for segments with smaller ΔG (more stable).

[0163] (4) Online control deployment gradient: The policy network can be deployed on the host computer, edge computing unit or embedded controller; the control interface can be EtherCAT, CAN, RS485, etc.; a safety constraint layer (amplitude limit, rate limit, obstacle / collision constraint) can be superimposed after the policy output to meet the engineering safety requirements.

[0164] (5) Low-resource training gradient: When the cost of obtaining training samples is high, the replay pool can be initialized by offline replay data; a smaller population size N and fewer generations G_max can be used, and the gradient update steps can be increased to maintain convergence; distributed parallel evaluation can be used to shorten the wall clock time.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent substitutions or variations made in accordance with this specification and claims should fall within the scope of protection of the present invention.

Claims

1. A high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy, characterized in that, The steps include the following: S1: Initialize the policy population and perform RNA encoding: Generate N policy network individuals, flatten and normalize the parameter vector of each individual in order, and map it to an RNA sequence; when evaluation or gradient update is required, decode the RNA sequence to restore the parameter vector and load it into the policy network; S2: Fitness evaluation: The trajectory of each individual in the policy network is obtained by interacting with the controlled object or the simulation environment, and the cumulative reward is calculated as the individual's fitness. S3: Evolutionary Update: Based on fitness, the population is ranked to select an elite set E, and selection, crossover, and base editing mutations are performed on non-elite individuals to generate a new generation of RNA sequences; the new sequences are decoded to obtain parameter vectors and form a new generation of strategy individuals; The mutation probability is adaptively determined based on both the RNA folding free energy index and the relative fitness level. S4: Adaptive Maximum Entropy Reinforcement Learning: Perform gradient updates based on the maximum entropy objective on the elite individual set E to update the policy network and evaluation network; and adaptively update the entropy coefficients according to the target entropy in each gradient update; S5: Diversity monitoring and noise injection: Calculate the population diversity index D. If D is lower than the threshold, randomly select M individuals from non-elite individuals or select them according to their diversity contribution. Inject Gaussian noise into their complete parameter vectors and prune and re-evaluate them to restore diversity. S6: Iterative loop; repeat S2-S5 until the convergence condition is met or the preset number of algebras is reached, and output the trained policy network for online control of the controlled object.

2. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The RNA encoding includes normalizing network parameters to a preset interval and mapping them into four symbols: A / U / C / G. The decoding includes mapping the symbols to their corresponding numerical ranges and assembling the recovered values ​​into a parameter vector in a preset order. The recovery value is a representative value of the corresponding interval; when the exploration enhancement trigger condition is met, the recovery value is changed to a sampled value obtained by random sampling within the corresponding interval, wherein the exploration enhancement trigger condition includes: the population diversity index D is lower than the threshold; subsequently, the components of each dimension of the parameter vector are pruned to meet the effective range constraint; wherein the random sampling process uses a fixed random seed or record to ensure repeatability.

3. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The folding free energy index ΔG is calculated using the minimum free energy MFE.

4. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The folding free energy index ΔG is calculated using an approximate stability index based on complementary pairing statistics.

5. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The mutation probability P_mut is determined through a two-stage coupling: first, the stability factor η is obtained based on the folding free energy index ΔG, and then the exploration coefficient c is obtained based on the relative fitness level. P_mut = clip(base_mut × η × c, P_min, P_max), where base_mut is the baseline mutation rate, and P_min and P_max are the lower and upper limits of the mutation probability, respectively.

6. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The base editing variations include at least base substitution, insertion, and deletion; the crossovers include at least single-point crossovers, multi-point crossovers, or complementary pairing crossovers.

7. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, Adaptive maximum entropy reinforcement learning includes maximum entropy policy optimization and adaptive update of entropy coefficients, which are used to achieve the maximum entropy objective optimization of the policy network and the adaptive adjustment of the entropy coefficient α. A soft actor-critic structure is adopted, and the strategy optimization objective is to maximize the sum of cumulative reward and entropy. The critic network is updated by minimizing the soft Bellman error; The actor network is updated using reparameterization techniques; The entropy coefficient α is used to adjust the weight of the entropy regularization term so that the current policy entropy approaches the target entropy H_target; preferably, α is updated by minimizing the temperature objective function.

8. The high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The target entropy H_target = -dim(A), where dim(A) is the action dimension.

9. A high-dimensional continuous control method based on RNA evolutionary adaptive maximum entropy according to claim 1, characterized in that, The online control is executed within a fixed control period Δt. The continuous control quantity output by the strategy network is limited and constrained to form a control command, which is then sent to the actuator driver via EtherCAT, CAN, or RS485 communication interface.

10. A high-dimensional continuous control device based on RNA evolutionary adaptive maximum entropy, characterized in that, It includes: The system includes a population initialization module, a fitness evaluation module, an evolutionary operation module, an adaptive maximum entropy reinforcement learning module, a diversity monitoring module, and a policy output and deployment module. Each module can be implemented by software, hardware, or a combination of both and deployed in a computer device or edge controller. The population initialization module is used to generate an initial policy population, generate N policy network individuals, and flatten and normalize the parameter vector of each individual in sequence before mapping it to an RNA sequence. The fitness evaluation module is used to obtain the trajectory of each individual in the policy network by interacting with the controlled object or the simulation environment, and to calculate the cumulative reward as the individual's fitness. The evolutionary operation module is used to select an elite set E of individuals based on fitness, and to perform selection, crossover, and base editing mutations on non-elite individuals to generate a new generation of RNA sequences; and to decode the new sequences to obtain parameter vectors and form a new generation of strategy individuals. The mutation probability is adaptively determined based on both the RNA folding free energy index and the relative fitness level. The adaptive maximum entropy reinforcement learning module is used to perform gradient updates based on the maximum entropy objective on the elite individual set E, update the policy network and the evaluation network, and adaptively update the entropy coefficients according to the target entropy in each gradient update. The diversity monitoring module is used to calculate the population diversity index D. If D is lower than the threshold, M individuals are randomly selected from non-elite individuals or selected according to their diversity contribution. Gaussian noise is injected into their complete parameter vectors and they are pruned and re-evaluated to restore diversity. The strategy output and deployment module is used to control the fitness evaluation module, evolutionary operation module, adaptive maximum entropy reinforcement learning module and diversity monitoring module to perform repeated iterative training until the convergence condition is met or the preset number of generations is reached, and output the trained policy network for online control of the controlled object.

Images