A conflict learning control method based on state-dependent dynamic weights

Abstract

The present application relates to the technical field of intelligent control, and particularly relates to a conflict learning control method based on state-related dynamic weight, current state information of a controlled object in a running process is acquired, and an end-to-end control strategy input is constructed based on the state information; a risk index is extracted based on the current state information, and a dynamic weight is calculated based on the risk index; online weighted fusion is performed on stable related targets and task related targets based on the dynamic weight, and a fused reward signal is obtained; a control strategy is trained and updated based on the fused reward signal, the method extracts a risk index through current state information, and calculates a dynamic weight online based on the risk index; then, the dynamic weight is used for reward fusion of control targets in a conflict relationship, so that the relative importance of balance targets and task targets is automatically adjusted and balanced by a reinforcement learning model in a training process, thereby realizing adaptive coordination of state-related conflicts.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a conflict learning control method based on state-related dynamic weights. Background Technology

[0002] When performing complex tasks, existing robots or drones typically need to simultaneously satisfy multiple control objectives, such as stabilization control, navigation and propulsion, obstacle avoidance control, trajectory tracking, and task execution. Because these multiple control objectives are often coupled, conflicts can easily arise, especially in highly dynamic, unstable, or complex environments.

[0003] In existing technologies, two main approaches are used to address this type of problem. The first is a modular or hierarchical control approach, where perception, planning, stability control, and execution control are handled by different modules. While this approach facilitates division of labor, the separation of interfaces between modules can easily lead to insufficient coordination, delayed response, and inconsistent overall control. The second approach is a fixed-weight multi-objective optimization approach, where multiple objectives are weighted and summed before unified optimization by pre-setting fixed weights. While this approach is relatively simple to implement, the conflict relationships between different objectives change with the system state. Fixed weights struggle to balance stability requirements in high-risk states with task progression requirements in low-risk states, potentially leading to system instability in high-risk states or overly conservative control in low-risk states.

[0004] Therefore, there is an urgent need for a conflict learning control method based on state-related dynamic weights, which can adjust the relative weights of different objectives online according to the current state and achieve adaptive coordination of multiple conflicting objectives. Summary of the Invention

[0005] The purpose of this invention is to provide a conflict learning control method based on state-related dynamic weights, which aims to adjust the relative weights of different objectives online according to the current state and achieve adaptive coordination of multiple conflicting objectives.

[0006] To achieve the above objectives, this invention provides a conflict learning control method based on state-related dynamic weights, comprising the following steps:

[0007] Obtain the current state information of the controlled object during operation, and construct an end-to-end control strategy input based on the state information;

[0008] Risk indicators are extracted based on the current status information, and dynamic weights are calculated based on the risk indicators.

[0009] Based on the dynamic weights, the stability-related objectives and the task-related objectives are weighted and fused online to obtain a fused reward signal;

[0010] The control strategy is trained and updated based on the fused reward signal.

[0011] The state information can be acquired through one or more of the following: inertial measurement unit, positioning module, environmental perception module, wheel speed encoder, and motor feedback module. The control strategy input includes one or more of the following: vehicle pitch angle, angular velocity, wheel speed, environmental perception information, and target relative orientation information.

[0012] The risk indicator is used to reflect the degree to which the controlled object deviates from its stable state. The risk indicator can be the vehicle body pitch angle of the controlled object. The larger the vehicle body pitch angle, the higher the degree to which the system deviates from its equilibrium state, and the greater the risk of instability.

[0013] The dynamic weights are used to represent the relative importance of stability-related targets and task-related targets in the current state. The stability-related targets are control targets used to maintain the stability and recoverable state of the controlled object, and the task-related targets are control targets used to propel the controlled object to complete a preset task.

[0014] The calculation of dynamic weights based on the aforementioned risk indicators includes:

[0015] The absolute value of the vehicle pitch angle θ(s) is normalized according to a preset range to obtain the stable relevant target weights:

[0016]

[0017] Where θ(s) represents the vehicle pitch angle in the current state, β(s) represents the dynamic weight calculated based on the current state, and 1-β(s) represents the weight of mission-related objectives. This means that the calculation result will be restricted to the range of 0 to 1.

[0018] The fusion reward signal can be expressed as:

[0019]

[0020] in, This indicates the reward items corresponding to the balance objective. This indicates the reward item corresponding to the navigation goal. The aforementioned dynamic weights.

[0021] The control strategy includes a reinforcement learning model with a long short-term memory network. The model includes a state input layer, a long short-term memory network layer, and an action output layer. The state input layer is used to receive a state vector composed of current state information. The long short-term memory network layer is used to extract the temporal features between state information at consecutive time points and form a hidden state representation. The action output layer is used to generate control actions of the controlled object's actuator based on the hidden state representation.

[0022] This invention discloses a conflict learning control method based on state-related dynamic weights. The method acquires the current state information of the controlled object during operation and constructs an end-to-end control strategy input based on this state information. It extracts risk indicators based on the current state information and calculates dynamic weights based on these risk indicators. Based on the dynamic weights, it performs online weighted fusion of stability-related and task-related objectives to obtain a fused reward signal. The control strategy is then trained and updated based on the fused reward signal. This method extracts risk indicators from the current state information and calculates dynamic weights online based on these risk indicators. Furthermore, it utilizes the dynamic weights to perform reward fusion on control objectives with conflicting relationships, enabling the reinforcement learning model to automatically adjust the relative importance of balancing objectives and task objectives according to different states during training, thereby achieving adaptive coordination of state-related conflicts. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a diagram illustrating the overall framework of conflict learning training and application in this invention.

[0025] Figure 2 This is a schematic diagram of the dynamic weight calculation and conflict target fusion of the present invention.

[0026] Figure 3 This is a flowchart of a conflict learning control method based on state-related dynamic weights provided by the present invention. Detailed Implementation

[0027] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0028] Please see Figures 1 to 3 This invention provides a conflict learning control method based on state-related dynamic weights, comprising the following steps:

[0029] S1 acquires the current state information of the controlled object during operation and constructs an end-to-end control strategy input based on the state information;

[0030] In this embodiment of the invention, the state information acquisition section can acquire state information through one or more of the following: an inertial measurement unit, a positioning module, an environmental perception module, a wheel speed encoder, and a motor feedback module. Specifically, the inertial measurement unit acquires the attitude angle, angular velocity, and acceleration information of the controlled object; the positioning module acquires the position coordinates, target distance, or navigation error information of the controlled object within the working area; the environmental perception module acquires the distance to surrounding obstacles, environmental contours, or local environmental information; the wheel speed encoder acquires wheel speed or displacement information; and the motor feedback module acquires motor control quantities, rotational speed, or actuator status.

[0031] The control strategy input does not require separating perception, planning, and control into independent modules. Instead, it combines state variables that reflect the attitude, motion, environmental state, and relative relationship with the target before inputting them into the control strategy. In the two-wheeled inverted pendulum robot embodiment, the control strategy input may include one or more of the following: vehicle pitch angle, angular velocity, wheel speed, environmental perception information, and target relative orientation information. The environmental perception information can be obtained by the environmental perception module and is used to characterize the distance to surrounding obstacles or the local environmental contour. The target relative orientation information may include the relative distance, relative direction, or relative angle between the controlled object and the target endpoint. Before inputting the control strategy, the state information undergoes coordinate transformation processing to unify the information obtained by the positioning module, the environmental perception module, and the controlled object's own state into a coordinate system constructed by the positioning module or the controlled object's own coordinate system, thereby forming a control strategy input under a unified reference system.

[0032] S2 extracts risk indicators based on the current state information and calculates dynamic weights based on the risk indicators;

[0033] In this embodiment of the invention, the risk index is used to reflect the degree to which the controlled object deviates from its stable state. In the two-wheeled inverted pendulum robot embodiment, the vehicle pitch angle can be used as the risk index. The larger the vehicle pitch angle, the higher the degree to which the system deviates from its equilibrium state, and the greater the risk of instability. The dynamic weight is used to represent the relative importance of stability-related targets and task-related targets in the current state. Stability-related targets refer to control targets used to maintain the stability and recoverable state of the controlled object, while task-related targets refer to control targets used to propel the controlled object to complete a preset task. In the two-wheeled inverted pendulum robot embodiment, stability-related targets can correspond to maintaining vehicle balance, and task-related targets can correspond to target point navigation.

[0034] The calculation of dynamic weights based on risk indicators includes:

[0035] The absolute value of the vehicle pitch angle θ(s) is normalized according to a preset range to obtain the stable relevant target weights:

[0036]

[0037] Where θ(s) represents the vehicle pitch angle in the current state, β(s) represents the dynamic weight calculated based on the current state, and 1-β(s) represents the weight of mission-related objectives. This means that the calculation result will be restricted to the range of 0 to 1.

[0038] S3 performs online weighted fusion of stable-related targets and task-related targets based on the dynamic weights to obtain a fused reward signal;

[0039] In this embodiment of the invention, the reward items corresponding to the balancing target and the navigation target are weighted and summed according to dynamic weights. The fused reward signal can be expressed as:

[0040]

[0041] in, This indicates the reward items corresponding to the balance objective. This indicates the reward item corresponding to the navigation goal. The aforementioned dynamic weights.

[0042] When the vehicle pitch angle is large, the dynamic weight increases, and the fused reward signal focuses more on the balance target; when the vehicle pitch angle is small, the dynamic weight decreases, and the fused reward signal focuses more on the navigation target.

[0043] S4 trains and updates the control strategy based on the fused reward signal.

[0044] In this embodiment of the invention, the control strategy can be implemented by a reinforcement learning model including a long short-term memory network. The model includes a state input layer, a long short-term memory network layer, and an action output layer. The state input layer receives a state vector composed of current state information. The long short-term memory network layer extracts temporal features between consecutive state information and forms a hidden state representation. The action output layer generates control actions for the controlled object's actuators based on the hidden state representation. The state vector may include one or more of the following: vehicle pitch angle, angular velocity, wheel speed, environmental perception information, and target relative orientation information. The control actions may correspond to the drive control quantities of the left and right wheel motors.

[0045] During training, the reinforcement learning model parameters are first initialized, and the current state information is acquired in a simulation or actual training environment. Then, the current state information and the historical hidden states of the long short-term memory network layer are input into the model to obtain the current control action. The environment updates the controlled object's state based on the control action and feeds back the state information for the next time step. Simultaneously, a fused reward signal is calculated based on the balance target and the navigation target. Subsequently, the reinforcement learning algorithm updates the model parameters based on the state information, control action, fused reward signal, and the state information for the next time step. This process is repeated until the control strategy meets the preset training conditions.

[0046] To improve the adaptability and robustness of the trained control strategy on actual controlled objects, noise, external disturbances, and random domain settings can be incorporated into the simulation training process. Specifically, measurement noise or delay disturbances can be added to the sensor observation data, execution delays, hold times, or output errors can be added to the actuator control actions, and one or more of the controlled object's mass, inertia, friction parameters, drive response, initial attitude, initial position, obstacle position, and target point position can be randomized. Through these methods, the reinforcement learning model is exposed to different state errors, execution errors, and environmental changes during training, thereby improving the control strategy's adaptability to sensor errors, actuator errors, and environmental uncertainties during actual deployment.

[0047] After training, the reinforcement learning model, which includes a long short-term memory network, is deployed to the actual controlled object, enabling the actual controlled object to directly generate control outputs based on real-time state information and historical hidden states, and to apply the control outputs to the actuators, thereby achieving integrated control of balance and navigation.

[0048] To better understand this technical solution, the following embodiments are provided for further explanation:

[0049] This invention is applied to the integrated control of balance and navigation in a two-wheeled inverted pendulum robot. For example... Figure 1 and Figure 2 As shown, the two-wheeled inverted pendulum robot faces both balance control and navigation objectives during operation. The balance control objective is used to maintain the stability of the robot's body posture, while the navigation objective is used to move the robot towards the target point and reach the target area.

[0050] In this embodiment, the robot acquires operational status data through an IMU, a UWB positioning module, a 2D LiDAR radar, and a wheel speed encoder. The IMU acquires the robot's pitch angle and angular velocity; the wheel speed encoder acquires the wheel speeds of the left and right wheels; the UWB positioning module acquires the robot's position within the work area and the relative distance and orientation between the robot and a target point; and the 2D LiDAR radar acquires distances to surrounding obstacles and local environmental perception data. This data collectively constitutes the control strategy input and is fed into a reinforcement learning model incorporating a long short-term memory network. The model generates drive control quantities for the left and right wheel motors based on the current state input and historical hidden states. These drive control quantities are converted by actuators and applied to the left and right wheels, thereby achieving direct control of the robot's balance and navigation behavior.

[0051] In this embodiment, the vehicle pitch angle is used as a risk indicator, and the state-related dynamic weights are obtained according to the dynamic weight calculation method in the above technical solution. Subsequently, the fusion reward signal calculation method in the above technical solution is used to perform online weighted fusion of the balancing target and the navigation target. Therefore, when the robot's pitch angle is large, the control strategy focuses more on restoring the vehicle's attitude stability; when the robot's pitch angle is small, the control strategy focuses more on moving towards the target point.

[0052] Training data is generated interactively within the simulation environment. Each training round records the robot's state information, left and right wheel control actions, state information at the next moment, fused reward signals, and the round's termination state. Training rounds can be conducted in a simulation scenario containing random initial postures, random initial positions, random target points, and random obstacles. Sensor noise, execution latency, external disturbances, and randomization of dynamic parameters can be incorporated to improve the robustness of the control strategy in actual deployment. Evaluation data includes task success rate, average completion time, average vehicle pitch angle, average pitch rate, fall occurrences, collision occurrences, and timeout occurrences.

[0053] Experimental results show that, on both fixed test maps and high-difficulty random obstacle maps, the PPO+LSTM control strategy using state-dependent dynamic weights achieved a success rate of 1.00%, with average completion times of 21.20±2.50 s and 32.53±2.81 s, respectively. The corresponding average absolute values ​​of vehicle pitch angle were 0.14±0.02 rad and 0.14±0.03 rad, and the average absolute values ​​of pitch angular velocity were 0.13±0.12 rad / s and 0.13±0.11 rad / s, respectively. In contrast, the control method without dynamic weights or using fixed weights was prone to falls or timeouts under the same test conditions, resulting in a success rate of 0%.

[0054] In a practical two-wheeled inverted pendulum robot deployment experiment, the trained control strategy was directly deployed to the actual robot. The task success rate was 95±5%, the average completion time was 17.09±9.04 s, the average path length was 6.52±3.96 m, the average absolute value of the vehicle's pitch angle was 0.05±0.03 rad, and the average absolute value of the pitch angular velocity was 0.49±0.09 rad / s. These results demonstrate that the state-dependent dynamic weight control method can achieve integrated and coordinated control of balance and navigation on the actual robot. Specific Implementation Example 2:

[0056] This invention can also be applied to UAV control scenarios. When a UAV faces both a flight stability target and a mission advancement target simultaneously, the current state information of the UAV can be acquired, and risk indicators can be extracted and dynamic weights calculated based on the state information. These dynamic weights are then used to perform online weighted fusion of the flight stability target and the mission advancement target to generate a fusion reward signal, which is used to update the control strategy. In this embodiment, the flight stability target may correspond to UAV attitude stabilization, altitude maintenance, or angular velocity suppression requirements, while the mission advancement target may correspond to target point navigation, trajectory tracking, or obstacle avoidance requirements. The risk indicators may consist of one or more of the following: UAV attitude deviation, angular velocity, velocity change, obstacle distance, or trajectory deviation. When the UAV is in a high-risk state, the weight of the flight stability target is increased; when the UAV is in a low-risk state, the weight of the mission advancement target is increased, thereby achieving adaptive coordination between flight stability and mission advancement.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1) This invention can adjust the relative weights of different targets online according to the current state of the system, thereby achieving adaptive processing of state-related target conflicts;

[0059] 2) This invention can increase the weight of stability-related targets under high-risk conditions, thereby enhancing the stability and recoverability of the system;

[0060] 3) This invention can increase the weight of mission-related objectives under low-risk conditions, thereby improving navigation propulsion, trajectory tracking, or mission execution efficiency;

[0061] 4) This invention does not require an additional complex hierarchical coordination structure, making the overall method simpler, lighter, and easier to train and deploy;

[0062] 5) This invention can be applied to two-wheeled inverted pendulum robots, as well as to drones and other control systems with multiple target conflicts.

[0063] The above-disclosed embodiments are merely preferred embodiments of a conflict learning control method based on state-related dynamic weights, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A conflict learning control method based on state-related dynamic weights, characterized in that, Includes the following steps: Obtain the current state information of the controlled object during operation, and construct an end-to-end control strategy input based on the state information; Risk indicators are extracted based on the current status information, and dynamic weights are calculated based on the risk indicators. Based on the dynamic weights, the stability-related objectives and the task-related objectives are weighted and fused online to obtain a fused reward signal; The control strategy is trained and updated based on the fused reward signal.

2. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The state information can be obtained through one or more of the following: inertial measurement unit, positioning module, environmental perception module, wheel speed encoder, and motor feedback module. The control strategy input includes one or more of the following: vehicle pitch angle, angular velocity, wheel speed, environmental perception information, and target relative orientation information.

3. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The risk indicator is used to reflect the degree to which the controlled object deviates from its stable state. The risk indicator can be the vehicle body pitch angle of the controlled object. The larger the vehicle body pitch angle, the higher the degree to which the system deviates from its equilibrium state, and the greater the risk of instability.

4. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The dynamic weights are used to represent the relative importance of stability-related targets and task-related targets in the current state. The stability-related targets are control targets used to maintain the stability and recoverable state of the controlled object, and the task-related targets are control targets used to propel the controlled object to complete a preset task.

5. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The risk indicators are calculated with dynamic weights, including: The absolute value of the vehicle pitch angle θ(s) is normalized according to a preset range to obtain the stable relevant target weights: Where θ(s) represents the vehicle pitch angle in the current state, β(s) represents the dynamic weight calculated based on the current state, and 1-β(s) represents the weight of mission-related objectives. This means that the calculation result will be restricted to the range of 0 to 1.

6. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The fusion reward signal can be represented as: in, This indicates the reward items corresponding to the balance objective. This indicates the reward item corresponding to the navigation goal. The aforementioned dynamic weights.

7. The conflict learning control method based on state-related dynamic weights as described in claim 1, characterized in that, The control strategy includes a reinforcement learning model of a long short-term memory network. The model includes a state input layer, a long short-term memory network layer, and an action output layer. The state input layer is used to receive a state vector composed of current state information. The long short-term memory network layer is used to extract the temporal features between state information at consecutive time points and form a hidden state representation. The action output layer is used to generate control actions of the controlled object's actuator based on the hidden state representation.

Images