A high-dimensional robot control method and device based on dimension entropy modulation

Abstract

The embodiment of the specification discloses a high-dimensional robot control method and device based on dimension entropy modulation. The method comprises inputting current state information into a pre-trained Actor network to obtain an action instruction output by the Actor network. The Actor network is used to extract current state features corresponding to the current state information, and sequentially performs temperature hyperparameter-based scaling and Softmax normalization on logits corresponding to each action dimension of the current state features, so as to obtain adjustment weights of each action dimension according to the product of the probability value output by the Softmax and the total number of action dimensions, and modulate the basic standard deviation according to the adjustment weights, so as to sample the action instruction according to the action distribution constructed by the modulated target standard deviation; and the robot is controlled to perform an action based on the action instruction. In the embodiment, the global optimal solution can be better found by the dimension entropy modulation method, and the performance is better in a complex control task.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of robot reinforcement learning control, and specifically relate to a high-dimensional robot control method and device based on dimensional entropy modulation. Background Technology

[0002] Deep reinforcement learning has become a core technology for solving complex control tasks in robots. However, with the development of high-degree-of-freedom systems such as humanoid robots, the dimension of the action space has increased dramatically, leading to a serious "curse of dimensionality".

[0003] Currently, mainstream methods for controlling high-dimensional humanoid robots fall into two main categories: deterministic policy gradient methods and traditional maximum entropy stochastic policy methods. Deterministic policies, combined with large-scale parallel simulation, exhibit good stability in simple movement tasks. However, they inherently lack exploration diversity and are prone to getting trapped in local optima. When faced with tasks requiring fine-grained full-body coordination, such as dribbling and shooting or balancing on complex terrain, they often fail to discover effective control strategies. Traditional maximum entropy stochastic policy methods, while theoretically possessing stronger exploration capabilities and robustness, are difficult to apply effectively in high-dimensional spaces due to low exploration efficiency, overestimation of value, and limitations in the accuracy of discretization methods. Therefore, a control method that can balance efficient exploration and high-precision value estimation in high-dimensional continuous control is urgently needed. Summary of the Invention

[0004] The embodiments of this disclosure provide a high-dimensional robot control method and apparatus based on dimensional entropy modulation, which aims to solve one or more of the above-mentioned problems and other potential problems.

[0005] According to a first aspect of this disclosure, a high-dimensional robot control method based on dimensional entropy modulation is provided. The method includes acquiring the robot's current state information, inputting the current state information into a pre-trained Actor network to obtain action commands output by the Actor network; the Actor network is used to extract current state features corresponding to the current state information, and the logits corresponding to each action dimension of the current state features are sequentially scaled based on a temperature hyperparameter and normalized using Softmax, so as to obtain the adjustment weights for each action dimension based on the product of the probability value output by Softmax and the total number of action dimensions, and modulating the base standard deviation according to the adjustment weights, sampling action commands based on the action distribution constructed according to the modulated target standard deviation, wherein the arithmetic mean of the adjustment weights for all dimensions is 1; and controlling the robot based on the action commands; wherein, Actor network... The Actor network is jointly trained with the Critic network; the Actor network is used to guide updates based on the variance of the Critic network's output; during the exploration phase of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment, which is used to modulate the unnormalized output of the Actor network; the Critic network is used to model the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, so as to output the mean and variance that minimize the difference between the predicted value distribution and the target value distribution; the Critic network uses inverse variance-weighted gradient estimation when updating parameters. The gradient estimation includes a mean update term and a variance update term. The mean update term is used to anchor the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

[0006] According to a second aspect of this disclosure, a high-dimensional robot control device based on dimensional entropy modulation is provided. The device includes an information acquisition module configured to acquire the robot's current state information and input the current state information into a pre-trained Actor network to obtain action commands output by the Actor network. The Actor network is used to extract current state features corresponding to the current state information, and to perform temperature-based hyperparameter scaling and Softmax normalization on the logits corresponding to each action dimension of the current state features. The adjustment weights for each action dimension are obtained by multiplying the probability value output by Softmax by the total number of action dimensions, and the base standard deviation is modulated based on the adjustment weights. Action commands are sampled based on the action distribution constructed from the modulated target standard deviation, and the arithmetic mean of the adjustment weights for all dimensions is 1. An action control module is configured to control the robot based on the action commands. Control is implemented; the Actor network is jointly trained with the Critic network; the Actor network is used to guide updates based on the variance of the Critic network's output; during the exploration process of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment, which is used to modulate the unnormalized output of the Actor network; the Critic network is used to model the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, so as to output the mean and variance that minimize the difference between the predicted value distribution and the target value distribution; the Critic network uses inverse variance-weighted gradient estimation when updating parameters. The gradient estimation includes a mean update term and a variance update term. The mean update term is used to anchor the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

[0007] According to a third aspect of this disclosure, an electronic device is provided, including one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform a method provided according to a first scheme.

[0008] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to the first aspect.

[0009] The scheme provided in the embodiments of this specification achieves autonomous focusing of exploration variance towards the critical task dimension through dimensional entropy modulation, while strictly adhering to the exploration budget conservation principle (i.e., without increasing total exploration noise). This avoids resource dissipation in redundant dimensions by stochastic policies, enabling them to escape convergence stagnation in complex action spaces with dimensions exceeding 60. Simultaneously, during the joint training of the Actor and Critic networks, the gradient is decoupled into mean and variance estimators. Combined with expectation replacement and the removal of the variance objective's dynamic shearing boundary, the stability of large-scale statistical data is utilized to eliminate quantization errors and suppress overestimation. This allows for stable and effective policy learning for the Actor network, leading to better discovery of the global optimum. The Actor network achieves a balance between efficient exploration and high-precision value estimation in high-dimensional continuous control, resulting in better performance in practical complex control tasks. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A flowchart illustrating a high-dimensional robot control method based on dimensional entropy modulation according to some embodiments of this disclosure is shown.

[0012] Figure 2 A schematic diagram of the training architecture of the Actor network in some embodiments of this disclosure is shown during the training process;

[0013] Figure 3 The diagram illustrates a performance comparison between the algorithm of this application and other existing baseline algorithms in different environments, based on some embodiments of this disclosure.

[0014] Figure 4 A schematic diagram of the structure of a high-dimensional robot control device based on dimensional entropy modulation according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0017] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “in response to determination”, or “in response to detection”.

[0018] Figure 1 A flowchart illustrating a high-dimensional robot control method 100 based on dimensional entropy modulation, according to some embodiments of this disclosure, is shown. Method 100 can be executed by a terminal, which may include, but is not limited to, mobile phones, tablets, desktop computers, servers, etc. Figure 1 As shown, in method 100, step 102 can obtain the current state information of the robot and input the current state information into the pre-trained Actor network to obtain the action instructions output by the Actor network. The Actor network is used to extract the current state features corresponding to the current state information. The current state features are scaled and normalized by temperature hyperparameter based on logits corresponding to each action dimension in turn. The adjustment weight of each action dimension is obtained by multiplying the probability value output by Softmax with the total number of action dimensions. The base standard deviation is modulated according to the adjustment weight. The action instructions are sampled according to the action distribution constructed based on the modulated target standard deviation. The arithmetic mean of the adjustment weights of all dimensions is 1.

[0019] Step 104 allows for the control of the robot based on motion commands.

[0020] The Actor network is jointly trained with the Critic network. The Actor network is used to guide updates based on the variance of the Critic network's output. During the exploration phase of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment. This scaling factor is used to modulate the unnormalized output of the Actor network. The Critic network is used to model the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, so as to output the mean and variance that minimize the difference between the predicted value distribution and the target value distribution. When updating parameters, the Critic network uses inverse variance-weighted gradient estimation. The gradient estimation includes a mean update term and a variance update term. The mean update term is used to anchor the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

[0021] In this embodiment, based on a pre-trained Actor network, the current state information of the robot to be controlled can be processed. The weights of each action dimension are adjusted according to the current state features corresponding to the current state information, thereby modulating the base standard deviation. An action distribution is constructed based on the modulated target standard deviation, and standard random sampling library functions are called to sample action commands from it for robot action control. This method differs from existing approaches that independently predict the variance of each dimension. It introduces a structured exploration budget constraint to avoid the inefficient budget allocation and "curse of dimensionality" problem caused by independent parameterization of the standard Gaussian policy. Specifically:

[0022] First, the budget-constrained variance is calculated, that is, for the features collected by the Actor backbone network. Calculate each action dimension Adjustment weight The adjustment weights can be calculated using a temperature-scaled Softmax normalization operation. This operation involves introducing a temperature hyperparameter to scale the logits, adjusting the sharpness of the weight distribution (for example, when the temperature hyperparameter is low, small differences between logits are amplified, leading to a more extreme weight distribution; conversely, when the temperature hyperparameter is high, the weight distribution tends to be smoother, resulting in a more balanced weight distribution across dimensions). The scaled values ​​are then Softmax normalized to transform them into a probability distribution. Finally, the output probability value is multiplied by the total number of action dimensions N to obtain the final adjustment weights. The calculation formula is as follows:

[0023]

[0024] in, It's a temperature over-parameter. These are logits (i.e., the unnormalized output of a neural network). This process enforces constraints. This means that the total exploration budget is conserved by ensuring that the arithmetic mean is 1. This allows the agent to autonomously reallocate the exploration budget, increasing variance (as a sink of entropy) in task-independent dimensions, thereby achieving near-deterministic and precise control in key control dimensions.

[0025] Next, the final standard deviation, i.e., the target standard deviation, can be modulated. Target standard deviation of each action dimension for:

[0026]

[0027] in, It is the basic logarithmic scaling parameter learned by the network.

[0028] The schematic diagram of the training architecture principle of the Actor network used above during the training process can be seen as follows: Figure 2 As shown, during training, the Actor network can utilize a large-scale parallel simulation environment to collect data by running multiple simulation instances simultaneously, thereby improving training efficiency. To prevent pattern collapse in the parallel population, a heterogeneous exploration strategy can be employed during the training exploration process, that is, for each parallel environment... Assign a scaling factor This scaling factor is used to modulate logits, allowing agents in different environments to exhibit different shapes of exploration anisotropy.

[0029]

[0030] This randomness ensures that some agents in the parallel population conduct extensive exploration, while others focus on fine-grained policy optimization in specific subspaces, thereby effectively escaping local optima through population diversity.

[0031] Furthermore, joint training can be achieved by integrating the Actor network and the Critic network into a Distributional Soft Policy Iteration (DSPI) loop. In this way, the continuous Gaussian Critic eliminates quantization errors and adaptively regularizes the Q-value updates using the distribution's prediction variance, thus solving the common overestimation problem in high-dimensional spaces while ensuring high-fidelity control signals. The Critic network can then adjust the current policy... In state Take action below This application models the long-term return distribution (i.e., the predicted value distribution). Unlike traditional methods that only estimate the expected value of returns, this application can accurately capture the random uncertainty of the environment by minimizing the difference between the predicted value distribution and the preset target value distribution, thus providing a more robust value signal for policy improvement. This fitting process can be measured by the Kullback-Leibler divergence. The Actor network will use the mean estimate determined by the Critic network to guide the update, and the update method is as follows:

[0032]

[0033] in, For Actor network parameters, Indicates from the buffer Sample state and by policy The expectation of generating the action, To balance the entropy temperature coefficient between exploration and utilization, This represents the average value output by the Critic. Because the Actor network uses dimensional entropy modulation... The item implicitly includes the reallocated exploration budget.

[0034] To mitigate quantization errors and overestimation, the Critic network can parameterize the reward distribution using a continuous Gaussian distribution.

[0035]

[0036] Critic parameters The update is achieved by minimizing the difference between the predicted value distribution and the target value distribution. The gradient can be decomposed into mean-driven and variance-driven components:

[0037]

[0038] in, These are the training parameters for the Critic neural network; The objective function of the Critic network with respect to the parameters The gradient; This is the gradient scaling factor, which approximates the average variance of the batch data. This is used to eliminate the sensitivity of gradients to reward scale; Let $\mathematical$ be the expected value, representing the average of the batch data in the experience replay buffer. The current predicted average return (Q value) of the Critic network. This represents the standard deviation of the current predictions made by the Critic network. The target mean is conservative; A random return sample taken from the target value distribution; A very small constant (e.g.) ), used to prevent numerical protection terms where the denominator is zero.

[0039] To mitigate overestimation, the conservative target mean is taken as the minimum Q-value predicted by the two twin target networks:

[0040]

[0041] in, The immediate reward that is fed back after an action is performed in a given state; This is the discount factor, which is usually a preset constant between 0 and 1; For the environment to transition to the next state And generate the next action. The minimum Q-value of the dual-objective network at that time; This is an entropy reward item.

[0042] In the gradient estimation formula, term (I) is the mean update term, which replaces the expected value and anchors the mean update to a conservative target mean instead of a random sample. Term (II) is the variance update term, which is used to capture the inherent randomness of the returns. In this term, the variance shear boundary in the original DSAC-T is removed, allowing the Critic to learn the true scale of the randomness of the environment unbiasedly.

[0043] By observing the gradient formula (I) term It can be seen that the update step size is related to the prediction variance. Inversely proportional. When an agent accesses an out-of-distribution state-action pair, due to the lack of historical data, the network typically predicts with extremely high uncertainty, i.e., variance. It is very large. At this point, the denominator is... This significantly reduces the gradient value. This means that for uncertain states, Critic will automatically reduce the update magnitude. This inverse variance weighting mechanism inherently acts as a regularizer, preventing the network from blindly making large value updates in unexplored regions, thus effectively curbing the "overestimation of Q-value" phenomenon common in high-dimensional spaces.

[0044] By employing the above method, when calculating the variance of the Critic distribution, the statistical stability provided by large-scale parallel data can be utilized to significantly reduce the numerical stability constant for preventing division-by-zero errors from 0.1 required by the traditional DSAC-T algorithm to [a much lower value]. This tiny numerical adjustment resulted in a significant performance improvement. The extremely low stability constant allows Critic to fit low-variance states with extremely high fidelity, thus keenly capturing subtle reward structures that would normally be smoothed out by large numerical constants, enabling refined modeling of rewards in complex environments.

[0045] Compared to existing technologies, which rely on a Critic network to perform single-point estimation and output a scalar Q-value as a predetermined expected reward, the Actor network explores based on this fixed variance. In other words, the existing approach involves the Critic providing a score to the Actor, which the Actor adjusts accordingly. This approach is prone to errors when the Critic assigns an incorrectly high Q-value to an unexplored state-action pair, causing the Actor to blindly imitate that action, leading to policy collapse. Furthermore, the Actor's exploration variance is typically globally uniform and cannot be dynamically adjusted based on the importance of the task dimension. This application, however, changes the Critic's update mechanism from equal error backpropagation to inverse variance weighting, achieving automatic regularization of the update process. This transforms the Critic's value representation from single-point estimation to continuous distribution modeling, shifting value estimation from numerical transmission to information (mean and variance) transmission. By combining the Actor's exploration strategy with dimensional entropy modulation, which changes from a globally uniform approach to a budget-constrained dimensional allocation, the collaborative training process becomes one where the Critic tells the Actor which directions might be high-value, and then dimensional entropy modulation tells the Actor how many steps (i.e., variance) to explore in those directions. Thus, when the Critic is highly uncertain about a region (high variance), it won't give the Actor a strong signal to approach that area (because the gradient is suppressed), thereby protecting the Actor from entering dangerous unexplored regions.

[0046] Furthermore, to avoid quantization errors caused by discretization (such as C51), in addition to using a Gaussian distribution for parameterization, the continuous distribution of Critic can also be replaced with a Laplace distribution or a Gaussian mixture model. Any alternative that maintains the continuity of the distribution, supports accurate modeling of the complete reward distribution, and allows for backpropagation of gradients can be considered. Alternatively, as another alternative to continuous modeling, a diffusion model can be used to parameterize the reward distribution. The diffusion model can accurately fit complex multimodal continuous distributions through a denoising process, and can also solve the accuracy limitations of discrete atom representations, thus mitigating the overestimation of value in high-dimensional spaces.

[0047] In one possible implementation, the scaling factor is determined based on random sampling in a preset uniform distribution.

[0048] In this embodiment, the scaling factor can be assigned randomly and automatically, from a uniform distribution. The sampling is randomly assigned to each parallel environment without the need for manual intervention or pre-setting.

[0049] In one possible implementation, during the update process, the entropy temperature coefficient of the Actor network will be automatically adjusted to meet the target entropy constraint, which is 0.

[0050] In this embodiment, the entropy temperature coefficient of the Actor network can be adaptively adjusted during parameter updates to meet a preset target entropy constraint. As an example, the target entropy constraint... It can be set to 0. This application abandons the heuristic settings commonly used in traditional algorithms (i.e. Instead, the target entropy of the task is uniformly set to... In high-dimensional spaces, traditional negative entropy settings can lead to excessively low target values, forcing the policy to converge prematurely to a low-variance state, thus triggering the "vanishing exploration" problem. Increasing the target entropy to... This approach preserves a sufficient exploration budget, preventing the strategy from prematurely falling into local optima. While this means increased overall randomness, the dimensional entropy modulation mechanism in this application acts as a "structured filter." It precisely guides this increased exploration budget to key dimensions relevant to the task, while effectively suppressing noise interference on redundant dimensions, thus maintaining high exploration activity while ensuring control precision.

[0051] In one possible implementation, the method further includes:

[0052] If the number of action dimensions is below a certain threshold, the normalization module is removed during the prediction and training process of the Actor network.

[0053] In this embodiment, this application aims to prevent actions with extremely high dimensions (e.g., The algorithm is trained and diverged in the HumanoidBench task, while layer normalization modules are introduced in both the Actor and Critic networks for additional normalization processing. This module can be removed for tasks with lower action dimensions (such as MuJoCo Playground and IsaacLab). Experiments show that even without additional regularization, the algorithm still performs excellently, confirming that its performance improvement mainly stems from the core algorithmic advantages of the dimensional entropy modulation mechanism and continuous Critic, rather than relying on specific network architecture techniques.

[0054] Figure 3This diagram illustrates a performance comparison between the algorithm of this application and other existing baseline algorithms in different environments, representing some embodiments of this disclosure. On three major benchmark platforms—HumanoidBench, MuJoCo Playground, and IsaacLab—different environments were selected to compare and evaluate the results obtained by the method of this application (i.e., FastDSAC in the figure) with other state-of-the-art (SOTA) baselines (i.e., FastTD3, FastSAC, DreamerV3, TD-MPC2, SAC, and PPO in the figure). As can be seen from the figure, in high-precision complex coordination tasks, such as basketball and balance hard tasks on HumanoidBench, FastDSAC achieves a final reward 1.8 times and 4 times that of FastTD3, respectively, significantly outperforming other baselines and demonstrating the control advantage of this application in high-dimensional whole-body coordination tasks. In dynamic motion tasks, such as hurdles and running, FastDSAC initially learns slightly slower than FastTD3, but its convergence reward is higher and its training variance is smaller, demonstrating the superiority of this application in the exploration-exploitation balance. Regarding robustness across physics engines, FastDSAC maintains state-of-the-art performance in tasks such as the G1 / T1 joystick task in MuJoCo Playground, rough terrain movement in IsaacLab, and cube placement. It only matches FastSAC in the hand-specific task in IsaacLab, demonstrating the strong generalization ability of this application across different physics engines and task types.

[0055] Figure 4 Schematic diagrams of a high-dimensional robot control device 400 based on dimensional entropy modulation according to some embodiments of this disclosure are shown. The various embodiments in this specification are described in a progressive manner, and similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. Figure 4As shown, the device 400 includes an information acquisition module 401, configured to acquire the robot's current state information and input the current state information into a pre-trained Actor network to obtain the action commands output by the Actor network. The Actor network is used to extract the current state features corresponding to the current state information, and to perform temperature-based hyperparameter scaling and Softmax normalization on the logits corresponding to each action dimension of the current state features. The adjustment weights of each action dimension are obtained by multiplying the probability value output by Softmax by the total number of action dimensions, and the base standard deviation is modulated according to the adjustment weights. The action commands are sampled based on the action distribution constructed according to the modulated target standard deviation. The arithmetic mean of the adjustment weights of all dimensions is 1. The action control module 402 is configured to control the robot based on the action commands. The Actor network is jointly trained with the Critic network; the Actor network is used to guide updates based on the variance of the Critic network's output; during the exploration phase of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment, which is used to modulate the unnormalized output of the Actor network; the Critic network is used to model the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, so as to output the mean and variance that minimize the difference between the predicted value distribution and the target value distribution; the Critic network uses inverse variance-weighted gradient estimation when updating parameters. The gradient estimation includes a mean update term and a variance update term. The mean update term is used to anchor the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

[0056] In one possible implementation, the scaling factor is determined based on random sampling in a preset uniform distribution.

[0057] In one possible implementation, during the update process, the entropy temperature coefficient of the Actor network will be automatically adjusted to meet the target entropy constraint, which is 0.

[0058] In one possible implementation, the device further includes an adaptive adjustment module configured to remove the normalization module during the prediction and training process of the Actor network in response to the number of dimensions of the action dimension being below a number threshold.

[0059] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0060] Figure 5 A block diagram of an electronic device 500 that can implement various embodiments of the present disclosure is shown. For example... Figure 5 As shown, the electronic device 500 includes a processor 510, a disk drive 520, an input / output interface 530, a network interface 540, and a memory 550. The processor 510, disk drive 520, input / output interface 530, network interface 540, and memory 550 can communicate with each other via a communication bus 560.

[0061] The processor 510 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0062] The memory 550 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage devices, etc. The memory 550 can store the operating system 551 used to control the operation of the electronic device 500, and the basic input / output system (BIOS) 552 used to control the low-level operations of the electronic device 500. Additionally, it can store a web browser 553, a data storage management system 554, etc. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 550 and is called and executed by the processor 510.

[0063] Input / output interface 530 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, vibrators, indicator lights, etc.

[0064] Network interface 540 is used to connect a communication module (not shown in the figure) to enable communication and interaction between the device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0065] Bus 560 includes a pathway for transmitting information between various components of the device, such as processor 510, disk drive 520, input / output interface 530, network interface 540, and memory 550.

[0066] It should be noted that although the above-described device only shows the processor 510, disk drive 520, input / output interface 530, network interface 540, memory 550, bus 560, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the method of this application, and does not necessarily include all the components shown in the figures.

[0067] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0069] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A high-dimensional robot control method based on dimensional entropy modulation, characterized in that, The method includes: The robot's current state information is acquired and input into a pre-trained Actor network to obtain action commands output by the Actor network. The Actor network is used to extract current state features corresponding to the current state information. The logits corresponding to each action dimension of the current state features are sequentially scaled based on temperature hyperparameters and normalized using Softmax. The adjustment weights of each action dimension are obtained by multiplying the probability value output by Softmax by the total number of action dimensions. The base standard deviation is modulated based on the adjustment weights. Action commands are sampled based on the action distribution constructed according to the modulated target standard deviation. The arithmetic mean of the adjustment weights of all dimensions is 1. The scaling operation of temperature hyperparameters involves introducing temperature hyperparameters to scale the logits and adjusting the sharpness of the weight allocation. The robot is controlled based on the action commands; The Actor network is jointly trained with the Critic network. The Actor network is updated based on the variance output of the Critic network. During the exploration phase of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment; this scaling factor modulates the unnormalized output of the Actor network. The Critic network models the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, outputting the mean and variance that minimize the difference between the predicted value distribution and the target value distribution. The Critic network uses inverse variance-weighted gradient estimation during parameter updates. The gradient estimation includes a mean update term and a variance update term. The mean update term anchors the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

2. The high-dimensional robot control method based on dimensional entropy modulation according to claim 1, characterized in that, The scaling factor is determined based on random sampling in a preset uniform distribution.

3. The high-dimensional robot control method based on dimensional entropy modulation according to claim 2, characterized in that, During the update process, the entropy temperature coefficient of the Actor network will be automatically adjusted to meet the target entropy constraint, which is 0.

4. The high-dimensional robot control method based on dimensional entropy modulation according to claim 1, characterized in that, The method further includes: If the number of action dimensions is below a certain threshold, the normalization module is removed during the prediction and training process of the Actor network.

5. A high-dimensional robot control device based on dimensional entropy modulation, characterized in that, The device includes: The information acquisition module is configured to acquire the robot's current state information and input the current state information into a pre-trained Actor network to obtain the action commands output by the Actor network. The Actor network is used to extract the current state features corresponding to the current state information, and to perform temperature hyperparameter scaling and Softmax normalization on the logits corresponding to each action dimension of the current state features in sequence. The adjustment weights of each action dimension are obtained by multiplying the probability value output by Softmax by the total number of action dimensions, and the base standard deviation is modulated according to the adjustment weights. The action commands are sampled based on the action distribution constructed according to the modulated target standard deviation. The arithmetic mean of the adjustment weights of all dimensions is 1. The temperature hyperparameter scaling operation involves introducing a temperature hyperparameter to scale the logits and adjusting the sharpness of the weight allocation. A motion control module is configured to control the robot based on the motion commands; The Actor network is jointly trained with the Critic network. The Actor network is updated based on the variance output of the Critic network. During the exploration phase of training, the Actor network randomly samples and assigns a scaling factor to each parallel environment; this scaling factor modulates the unnormalized output of the Actor network. The Critic network models the predicted value distribution of the state-action pairs generated by the Actor network according to the current policy, outputting the mean and variance that minimize the difference between the predicted value distribution and the target value distribution. The Critic network uses inverse variance-weighted gradient estimation during parameter updates. The gradient estimation includes a mean update term and a variance update term. The mean update term anchors the mean update to the target mean, and the variance update term removes the variance shearing boundary. The value update step size corresponding to the gradient estimation is inversely proportional to the predicted variance.

6. An electronic device, characterized in that, include: One or more processors, and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the high-dimensional robot control method based on dimensional entropy modulation according to any one of claims 1-4.

7. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements a high-dimensional robot control method based on dimensional entropy modulation according to any one of claims 1-4.

Images