A self-balancing unicycle control system and method based on multi-modal semantic perception and reinforcement learning

Abstract

The application discloses a self-balancing unicycle control system and method based on multi-modal semantic perception and reinforcement learning, which combines multi-modal semantic perception and reinforcement learning. A sensor module acquires unicycle posture and environmental multi-source information, which is processed by a semantic perception module to form an enhanced state representation containing environmental semantics and vehicle state. A reinforcement learning control module generates unicycle balance control instructions based on the enhanced state through a deep neural network strategy, drives an actuator to realize pitch and roll stability control, and can realize online adaptive adjustment of the strategy through reward feedback. The system constructs a closed-loop control of perception-decision-execution: multi-modal sensors such as vision understand environmental semantics, adaptive strategy iteratively optimizes in real time, and maintains the balance and motion control of the unicycle in a complex dynamic environment. The application improves the environmental understanding ability and the adaptability of the control strategy, and significantly enhances the balance stability and intelligent decision level of the unicycle in unknown road conditions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robots and balance control technology, and in particular to a self-balancing unicycle control system and method based on multimodal semantic perception and reinforcement learning. Background Art

[0002] Self-balancing unicycles have attracted attention for their compact structure and maneuverability. However, with only a single wheel in contact with the ground, unicycles are inherently nonlinear and unstable systems, requiring continuous attitude adjustments to maintain balance. Traditional unicycle control schemes typically employ a cascade PID controller coupled with multi-sensor data fusion. For example, one study designed a three-stage cascade control loop (current loop, velocity loop, and attitude loop) to stabilize motor current, wheel speed, and vehicle body inclination, respectively. In such systems, multiple sensors, such as an IMU, wheel encoders, and an electronic compass (magnetometer), are typically used to acquire attitude and velocity information. Kalman filtering is used to fuse sensor data to estimate the vehicle's attitude angle in real time, thereby achieving longitudinal and axial stability control using an inner and outer dual-loop PID control algorithm. This type of method works well in known models and fixed environments, but has the following limitations: First, parameter tuning is difficult, PID gains require repeated manual debugging, and it is difficult to balance stability and response speed under different loads and road conditions; second, there is a lack of environmental semantic perception. Traditional solutions only use low-level sensor values ​​and cannot understand high-level semantic information about the road conditions or obstacles ahead, resulting in limited decision-making capabilities in complex environments; third, there is insufficient adaptability. Once the environment changes dynamically (for example, the road friction coefficient changes or new interference appears), the fixed rule controller is difficult to adjust in time, which may lead to imbalance.

[0003] In recent years, reinforcement learning (RL) has demonstrated unique advantages in robotic control. RL agents, rather than requiring manual design of control laws, autonomously learn optimal policies through repeated interactions with their environment. Compared to pre-programmed control algorithms, RL methods are able to learn from experience and adapt to changes. For example, in complex and unpredictable situations, RL controllers can gradually optimize their policies through trial and error, maintaining stable performance in dynamic environments. Furthermore, the development of multimodal semantic perception technology has enabled robots to understand high-level semantic information about their environment. Typical applications have demonstrated that robots equipped with deep reinforcement learning can utilize input from multiple sensors, such as cameras, LiDAR, and microphones, to achieve integrated perception and decision-making. Multimodal perception integrates visual, distance, and acoustic information into a unified state representation, giving robots a contextual understanding of their surroundings. This semantic-level perception facilitates more intelligent policy decisions, such as adjusting a unicycle's balance and speed based on the identified road slope and obstacle locations.

[0004] Based on the above background, the existing unicycle control system based on cascade PID is difficult to meet the needs of adaptive balance and intelligent decision-making in complex environments. Therefore, a new type of control system that combines multi-modal semantic perception and reinforcement learning is needed, which can learn the best control action through policy iteration based on obtaining deep semantic information of the environment, and realize the balance and motion control loop of the unicycle in variable scenes. SUMMARY

[0005] Technical purpose: In view of the deficiencies of the prior art in adaptability and intelligent decision-making in complex environments, the present application discloses a self-balancing unicycle control system and method based on multi-modal semantic perception and reinforcement learning. The unicycle system perceives the environment and its own state through multi-modal sensors such as imaging sensors and inertial measurement units, and introduces a semantic perception module to extract high-level semantic information of the environment. Then, using an adaptive reinforcement learning control strategy, the control instructions of the unicycle are calculated in real time according to the fused state and semantic information, realizing intelligent adjustment of the unicycle's attitude and balance.

[0006] Technical scheme: In order to achieve the above technical purpose, the present application adopts the following technical scheme:

[0007] A self-balancing unicycle control system based on multi-modal semantic perception and reinforcement learning, comprising:

[0008] A sensor module, at least one inertial measurement unit for measuring the attitude angular velocity and acceleration of the unicycle; at least one encoder for measuring the wheel speed or angle; at least one imaging sensor for acquiring images or videos of the environment around the unicycle;

[0009] A semantic perception module, including a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the dynamics state of the unicycle and the semantic information of the environment;

[0010] A reinforcement learning control module, including a policy decision unit, which is built-in with a deep reinforcement learning policy network, to calculate the control instructions of the unicycle, including the torque instructions of the drive wheel motor and the control instructions of the lateral balance actuator, using the enhanced state vector as input; a policy optimization unit for training and optimizing the policy network according to a predetermined reward function using reinforcement learning algorithm;

[0011] The actuator includes a drive motor, which receives the torque instruction of the drive wheel motor to drive the wheel of the unicycle to rotate; a balance control actuator, which receives the control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle;

[0012] The modules are connected via a communication link to form a closed-loop control system, enabling the unicycle to achieve self-balancing under the control of the strategy network.

[0013] Preferably, the balance control actuator includes a reaction flywheel mechanism or a dual gyroscope mechanism. When the reaction flywheel mechanism is adopted, the balance motor drives the internal flywheel to rotate to generate a reverse torque to adjust the roll angle of the unicycle; when the dual gyroscope mechanism is adopted, the two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance.

[0014] Preferably, the policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes:

[0015] The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges;

[0016] The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

[0017] Preferably, the semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps, and converting the environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

[0018] Preferably, the components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of the decision, the state or action of several past moments is input into the enhanced state vector as historical features.

[0019] A multimodal semantic perception and reinforcement learning self-balancing unicycle control method is applied to the multimodal semantic perception and reinforcement learning self-balancing unicycle control system as described above, specifically comprising the following steps:

[0020] S1, using imaging sensors, inertial measurement units, and encoder multimodal sensors to collect raw data on the unicycle's own posture and surrounding environment;

[0021] S2. Processing the collected multimodal data includes extracting environmental semantic information from the imaging sensor image through visual semantic analysis, wherein the environmental semantic information includes road type, slope, obstacle location and size, obtaining the attitude angle, angular velocity, position and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information with the unicycle state data, and constructing an enhanced state vector as a representation of the current state of the unicycle;

[0022] S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance;

[0023] S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction;

[0024] S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and repeat steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

[0025] Preferably, the visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time, and the enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

[0026] Preferably, the reinforcement learning strategy model adopts a deep reinforcement learning algorithm with an actor-critic architecture, which consists of a strategy network and a value network. The strategy network acts as an actor to output the unicycle control action, and the value network acts as a critic to estimate the value of the current state-action pair. The strategy network and the value network are adaptively improved through online updating of neural network parameters. During the execution process, the reinforcement learning strategy model is continuously adjusted according to the unicycle's balance performance and task completion status to optimize the long-term cumulative return. ,in is the discount factor, For immediate returns, For control rounds or learning rounds, is the total length of the control round or learning round.

[0027] Preferably, the instant reward is defined as It includes the following items: giving negative rewards for the angle and angular velocity at which the unicycle's posture deviates from the upright position. The larger the angle deviation, the larger the negative value, to encourage reducing tilt; giving positive rewards for the time or distance the unicycle maintains balance to encourage extending the balance time; when there are obstacles, positive rewards are given if the unicycle avoids collision and successfully overcomes the obstacle, and large negative rewards are given if a collision or tipping occurs.

[0028] Preferably, the control action in step S3 includes two components, namely a longitudinal balancing torque and a lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; wherein the longitudinal balancing torque is achieved by adjusting the torque of the unicycle driving wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body, wherein the balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

[0029] Beneficial effects: The self-balancing unicycle control system and method based on multimodal semantic perception and reinforcement learning provided by the present invention has the following beneficial effects:

[0030] 1. Through multimodal semantic perception, the system can identify different terrains, obstacles, and driving environments, and respond in advance to situations such as slopes and uneven surfaces, thereby enhancing the adaptability and stability of the unicycle in unknown and complex environments. By utilizing reinforcement learning strategies, the system does not rely on manual parameter adjustment, but instead continuously optimizes the control strategy through interaction with the environment. It can automatically adjust parameters according to load changes and environmental disturbances, and always maintain optimal balance control performance.

[0031] 2. The present invention integrates multi-source sensor information and obtains the complete system state through state reconstruction. Even if individual sensor signals are unreliable or temporarily invalid, necessary state variables can still be inferred from other information sources, thereby improving the robustness and safety of the system. All links from sensing to control are closely connected, and the semantic understanding module and the reinforcement learning control strategy constitute the decision-making dual core: the former provides environmental understanding, and the latter gives optimized actions, realizing the logical closed loop of the robot's autonomous behavior, and possessing human-like intelligent decision-making capabilities rather than simple rule responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0033] Figure 1 Schematic diagram of the system architecture of the present invention;

[0034] Figure 2 This is a flow chart of the control method of the present invention;

[0035] Figure 3 Schematic diagram of multimodal semantic information fusion of the present invention;

[0036] Figure 4 This is a comparison chart of the simulation performance of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described more clearly and completely below by way of a preferred embodiment in conjunction with the accompanying drawings, but the present invention is not limited to the scope of the embodiment.

[0038] like Figure 1 As shown, a self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning includes:

[0039] The sensor module includes at least one inertial measurement unit for measuring the unicycle's attitude, angular velocity, and acceleration; at least one encoder for measuring wheel speed or rotation angle; and at least one imaging sensor for acquiring images or videos of the unicycle's surroundings. The sensor module acquires multimodal data, including the unicycle's own attitude, angular velocity, and position, as well as images or depth information of the surrounding environment.

[0040] The semantic perception module includes a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; and a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the unicycle's dynamic state and environmental semantic information.

[0041] In one embodiment, the components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of decision-making, the state or action of several past moments is input into the enhanced state vector as historical features.

[0042] The semantic perception module processes multimodal data. For example, it performs object detection and semantic segmentation on camera images, extracting semantic features such as road slope, surface type, and obstacle location. It also filters and fuses the inertial measurement unit and encoder data to estimate state variables such as the unicycle's current posture angle and angular velocity. Through state reconstruction, implicit state variables that cannot be directly measured (such as future balance trends) are incorporated into a unified state representation. This unified state encompasses both the physical state and the semantic description of the environment, enabling control decisions to incorporate global environmental awareness.

[0043] In one embodiment, the semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps and converting environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

[0044] The reinforcement learning control module includes a policy decision unit with a built-in deep reinforcement learning policy network, which uses the enhanced state vector as input to calculate the control instructions of the unicycle, including the torque instructions of the drive wheel motor and the control instructions of the lateral balance actuator; and a policy optimization unit for using a reinforcement learning algorithm to train and optimize the policy network according to a predetermined reward function.

[0045] In one embodiment, the policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes:

[0046] The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges;

[0047] The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

[0048] The reinforcement learning control module adopts a policy network to realize the mapping from the state to the action, and the policy is trained by a reinforcement learning algorithm. Preferably, a deep reinforcement learning algorithm with an actor-critic structure is used, so that the policy network selects a control action according to the current semantic state, and at the same time, the value network evaluates the long-term value of the action. In the training process, the balance of the unicycle and the following of the desired trajectory are taken as the optimization objectives, and a reward function R is designed: for example, negative rewards are given to posture deviation, control energy consumption, and collision risk, and positive rewards are given to the length of the balance time and the success of passing through obstacles. The policy training can be performed offline in a physical simulation environment, and then online adaptation is realized through self-learning in a loop. When the unicycle is driving in the actual environment, the reinforcement learning control module can also realize policy evolution and continuously improve the adaptability of the policy to environmental changes through online updating or policy fine-tuning according to the new situation perceived in real time.

[0049] An actuator, including a driving motor, receives a torque instruction of the driving motor to drive the unicycle wheel to rotate; a balance control actuator receives a control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle.

[0050] In one embodiment, the balance control actuator includes a reaction flywheel mechanism or a double-gyroscope mechanism. When the reaction flywheel mechanism is adopted, a balance motor drives an internal flywheel to rotate to generate a reverse torque to adjust the roll angle of the unicycle; when the double-gyroscope mechanism is adopted, two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance.

[0051] The above modules are connected through a communication link to form a closed-loop control system, so that the unicycle realizes self-balance under the control of the policy network.

[0052] As shown in Figure 2 The application also provides a self-balancing unicycle control method based on multi-modal semantic perception and reinforcement learning, which is applied to the self-balancing unicycle control system based on multi-modal semantic perception and reinforcement learning as described above, and specifically includes the following steps:

[0053] S1, acquiring original data of the posture of the unicycle and the surrounding environment by using imaging sensors, inertial measurement units, and multi-modal sensors such as encoders;

[0054] S2, processing the collected multi-modal data, including extracting environmental semantic information from the imaging sensor images through visual semantic analysis, wherein the environmental semantic information includes road types, slopes, obstacle positions, and sizes, obtaining the attitude angle, angular velocity, position, and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information and the state data of the unicycle, and constructing an enhanced state vector as the current state representation of the unicycle;

[0055] In one embodiment, the visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time, and the enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

[0056] S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance;

[0057] In one embodiment, the control action in step S3 includes two components: a longitudinal balancing torque and a lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; wherein the longitudinal balancing torque is achieved by adjusting the torque of the unicycle drive wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body, wherein the balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

[0058] S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction;

[0059] S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and cyclically execute steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

[0060] Example 1

[0061] This embodiment describes the system architecture of the present invention in detail.

[0062] The self-balancing unicycle control system of the present invention includes a sensor module, a semantic perception module, a reinforcement learning control module and an actuator, as well as a unicycle body. The sensor module includes: an inertial measurement unit, which is used to measure the three-axis angular velocity and acceleration of the unicycle; a wheel encoder, which is used to measure the rotational speed and angular displacement of the unicycle wheel; an imaging sensor, which in this embodiment uses an environmental camera to collect an image sequence of the environment in front of the unicycle; and an optional distance sensor (such as ultrasonic or LiDAR) for detecting the distance to the obstacle in front. Each sensor is connected to the semantic perception module through a data bus. The semantic perception module includes: a posture estimation unit, which receives the inertial measurement unit and encoder data, and calculates the pitch angle of the unicycle through extended Kalman filtering or other data fusion algorithms. , roll angle and its angular velocity 、 The visual semantic understanding unit performs semantic segmentation and target detection on camera images, extracts information such as terrain categories (such as flat land, uphill, downhill, steps), obstacles (position and size), and reference landmarks; the state reconstruction unit synchronizes the time and space of the posture estimation and visual semantic information to generate a unified enhanced state vector The state vector It not only contains the dynamic state of the unicycle itself (angle, angular velocity, position, speed), but also includes key information about the environment (such as the obstacle of height h at d meters ahead, the current road slope In one specific implementation, the enhanced state It can be expressed as:

[0063]

[0064] Where v is the forward speed of the unicycle, is the unicycle linear acceleration, and Respectively represent the distance and height of the detected obstacle, Represents the current slope angle. The above state components are directly measured by sensors or calculated by the visual semantic understanding unit.

[0065] The reinforcement learning control module receives the enhanced state from the semantic perception module As input, the control instructions are calculated and output through the internal policy network. The reinforcement learning control module consists of two parts: the policy decision unit and the policy optimization unit. The policy decision unit has a built-in trained deep neural network (i.e., policy , the parameters are ), the network is in state The control action a of the unicycle is output as input. The control action here includes: the motor torque driving the unicycle wheel and the actuator control quantity for lateral balancing In a specific implementation, the unicycle can use an internal balancing wheel, i.e., a reaction flywheel or a dual gyroscope system to control the lateral balance. For example, the present embodiment uses a reaction flywheel, Represents the reaction flywheel motor torque, used to adjust the roll angle ; Indicates the driving wheel motor torque, used to control the pitch angle and forward / backward motion. Policy network output Acting on the actuator, the drive wheel motor receives To drive the unicycle forward or backward, the reaction flywheel motor receives The reaction flywheel is driven to rotate to generate lateral torque to achieve left-right balance. The action of the actuator changes the posture and displacement of the unicycle body.

[0066] The policy optimization unit is responsible for training and online tuning the policy network using a reinforcement learning algorithm. The training process can be performed in a simulated environment based on a unicycle dynamics model. The longitudinal dynamics of a unicycle can be approximated as an inverted pendulum, while the lateral inertia moment generated by the reaction flywheel allows for a state-space model. For example, based on the Newton-Euler equations, the pitch angular dynamics of a unicycle can be expressed as:

[0067]

[0068] in is the moment of inertia of the unicycle about the horizontal axis, is the pitch angle of the unicycle The second-order time derivative of the pitch angle is the angular acceleration, m is the total mass, h is the height of the center of mass, and g is the acceleration of gravity. is the driving wheel motor torque (positive for forward movement); the roll angle dynamics is formed by the reaction wheel to form an equivalent inverted pendulum:

[0069]

[0070] in is the moment of inertia of the reaction wheel-body system around the longitudinal axis, The strategy optimization unit uses the reinforcement learning algorithm to adjust the strategy parameters. , so that the unicycle can maintain and can all tend to zero, achieving a continuous and stable balance. Specifically, the introduction of cumulative returns ,in is the immediate return at time t, is the discount factor. The immediate return signal is designed according to the balance and target tracking requirements, such as:

[0071]

[0072] in 、 、 、 、 and is a weight constant used to balance the importance of each penalty and reward. 、 The pitch and roll deviations of t, 、 are the angular velocities corresponding to the pitch angle and roll angle, The indicator of the unicycle's continuous uprightness (e.g., a fixed positive reward is given for every second it remains upright without tipping over). and If the value is within the safety threshold, it is 1; otherwise, it is 0. An indicator for avoiding collisions with obstacles. If no collision occurs with an obstacle, it is 1, otherwise it is 0. It is iteratively updated through a policy gradient algorithm (such as proximal policy optimization PPO or deep deterministic policy gradient DDPG). , maximize the cumulative return R, and thus obtain the optimal strategy parameters After simulation training, the policy network is deployed to the policy decision unit for actual unicycle control. When the actual environment deviates from the trained model, the policy optimization unit can perform on-site retraining based on real-world feedback (e.g., fine-tuning the policy using a small amount of actual test data), enabling continuous evolution and adaptation of the policy.

[0073] Example 2

[0074] This embodiment describes the control process of the method in detail.

[0075] S1. When the unicycle is started, the sensor module starts working. The environmental camera collects image frames of the environment in front, the inertial measurement unit measures the angular velocity and acceleration of the vehicle body, the encoder records the wheel rotation information, and other sensors (such as ultrasonic waves) detect the distance to obstacles. Within one cycle, the data of each sensor is sent to the data buffer with a timestamp.

[0076] S2, the semantic perception module reads synchronized multimodal data frames from the buffer, first calls the visual semantic understanding unit to perform convolutional neural network inference on the image, and outputs pixel-level semantic segmentation results and target detection lists, such as identifying the road surface category (flat / pothole / ramp), the location and size of the obstacle in front, etc. Then the posture estimation unit performs Kalman filtering on the inertial measurement unit and encoder data to estimate the current unicycle pitch angle, roll angle and their rate of change. Finally, the state reconstruction unit fuses the visual semantic information with the posture estimation results to form the enhanced state at the current moment. This step may involve converting the obstacle position in the visual coordinate system to the unicycle coordinate system, for example, estimating the distance and height of the obstacle based on the intrinsic parameters of the environment camera and the pitch angle of the inertial measurement unit, and using it as part of the state.

[0077] S3, reinforcement learning control module receiving status After that, the policy decision unit calls the deep neural network to calculate the output action This step is equivalent to the policy function The strategy can be adjusted according to the semantic information of the environment in the state. For example, when an uphill slope with a large slope angle is detected, the strategy network may output a larger forward torque. To prevent backward tilt; when an obstacle is detected on one side that may cause tilt, the strategy network adjusts Correct the balance tendency in advance. The entire decision is completed within milliseconds to ensure the control frequency (typically above 100Hz).

[0078] Figure 3 The figure shows a schematic diagram of the multimodal semantic information fusion of the present invention. The visual semantic understanding unit performs visual semantic analysis on the real-time scene image, extracting environmental labels such as roads, obstacles, and slopes. The posture estimation unit uses the inertial measurement unit and wheel encoders to estimate the state of the unicycle, measuring the numerical states of the vehicle body such as pitch angle, roll angle, angular velocity, and forward speed. The outputs of both are input to the state reconstruction unit for state fusion. The enhanced state vector obtained after fusion combines environmental semantics and high-precision dynamic information. Finally, it is output to the strategy decision unit, providing a comprehensive and robust decision-making basis for other control strategies.

[0079] S4, the calculated control command is sent to the wheelbarrow actuator through the motor driver. Adjust the wheel acceleration or deceleration to control the forward and backward movement and pitch stability of the unicycle; the reaction flywheel motor is based on Changing the speed of the reaction flywheel generates a torque to offset the roll. Through the synergistic effect of these two torques, the unicycle is corrected to the upright balance direction in the current state.

[0080] S5: After the unicycle's controlled movement, the new state is fed back via sensors, entering the next cycle. The entire process is executed continuously, achieving real-time closed-loop control. If an abnormal situation occurs (such as sensor failure or exceeding the stability range), the system can enter a safe mode (such as emergency braking or auxiliary support mechanism intervention) to protect the unicycle and its surroundings.

[0081] Through this cyclical process, the unicycle can maintain self-balancing and complete driving tasks in a variety of complex situations. For example, when encountering complex terrain, the environmental camera detects a step ahead, and the semantic perception module recognizes that the step height exceeds the balancing threshold. The reinforcement learning strategy may then reduce the vehicle's speed and adjust its posture in preparation for parking. When environmental disturbances (such as crosswinds) cause the vehicle to tilt, the change in roll angle fed back by the inertial measurement unit causes the strategy to quickly correct the corresponding reaction wheel torque. Furthermore, through continuous online adjustments through reinforcement learning, system performance is optimized with use, demonstrating smoother control when repeatedly traversing a certain section of road.

[0082] Example 3

[0083] In order to verify the effectiveness of the present invention, a comparative simulation test was carried out. The simulation used a certain dynamic simulation platform to establish a unicycle model, including two degrees of freedom in pitch and roll, including wheel drive and reaction wheel control. The comparison objects were the traditional cascade PID controller and the reinforcement learning controller of the present invention. The PID controller adopts a classic dual-loop structure: the inner loop controls the pitch angle stability, the outer loop controls the forward speed, and adds roll angle correction. The parameters are repeatedly manually debugged to achieve a better result. The reinforcement learning controller uses the Actor-Critic algorithm for pre-training and converges after 1e5 steps of interaction in the simulation.

[0084] In test scenario 1 (maintaining static balance on flat ground), the unicycle initially deviates at a small angle of 1°. Figure 4 The curves of pitch angle variation with time under the action of two controllers are given. It can be seen that the PID controller makes the pitch angle approach 0 in about 2.5 seconds, but there is a The reinforcement learning control strategy of the present invention converges the angle error to 0 in about 1.5 seconds, with an overshoot amplitude less than half of that of PID and no obvious oscillation, showing faster stabilization speed and higher balancing accuracy. This is because the reinforcement learning strategy can output the correction torque in advance according to the state change trend, which is predictable and adaptive, while PID only calculates according to a fixed proportional integral differential based on the current error, and is limited in terms of fast response and oscillation suppression.

[0085] In test scenario two (forward balance with obstacles), a small obstacle 5 cm high was placed 2 meters ahead. The unicycle had to move forward slowly and cross the obstacle while maintaining balance. The results showed that the PID control strategy lacked perception of the obstacle and only suddenly increased the torque to try to maintain balance when the wheel hit the obstacle, resulting in a drastic fluctuation of about 3° in the pitch angle and the vehicle almost tipping over. In contrast, the system of the present invention used vision to perceive the obstacle in advance and slowed down when it approached. At the same time, the policy network had learned the posture adjustment method required to cross the obstacle, so the pitch angle only fluctuated gently by no more than 1°, and the unicycle successfully crossed the obstacle and continued to move forward. Test scenarios also included ramp driving and disturbance recovery, all of which verified the robustness and intelligence of the present invention. This proves that the multimodal semantic perception and reinforcement learning control system of the present invention has significant advantages over traditional methods in terms of stable control and environmental adaptation of the self-balancing unicycle.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning, characterized in that: include: Sensor module, at least one inertial measurement unit, used to measure the angular velocity and acceleration of the unicycle; at least one encoder, used to measure the wheel speed or rotation angle; at least one imaging sensor for acquiring images or videos of the environment surrounding the unicycle; The semantic perception module includes a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; and a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the unicycle's dynamic state and environmental semantic information. The enhanced state vector includes at least: ， in is the pitch angle of the unicycle, is the unicycle roll angle, is the angular velocity corresponding to the pitch angle, is the angular velocity corresponding to the roll angle, v is the forward speed of the unicycle, is the forward acceleration of the unicycle, and Respectively represent the distance and height of the detected obstacle, Represents the current slope angle, and the enhanced state vector 、 、 、 、v、 、 、 and Directly measured by sensors or calculated by visual semantic understanding units; The reinforcement learning control module includes a policy decision unit with a built-in deep reinforcement learning policy network, which uses the enhanced state vector as input to calculate control instructions for the unicycle, including torque instructions for the drive wheel motor and control instructions for the lateral balance actuator; a policy optimization unit, which is used to train and optimize the policy network using a reinforcement learning algorithm according to a predetermined reward function; The reward function is ,in is the discount factor, For immediate returns, For control rounds or learning rounds, The total length of the control or learning rounds, Specifically: ， in 、 、 、 、 and is a weight constant used to balance the importance of each penalty and reward. (t), t) are the pitch angle and roll angle deviations at time t, (t), (t) are the angular velocities corresponding to the pitch angle and roll angle, is an indicator of the unicycle remaining upright. and If the value is within the safety threshold, it is 1; otherwise, it is 0. It is an indicator of avoiding collision with obstacles, which takes 1 if no collision with obstacles occurs, and 0 otherwise; The actuator includes a drive motor, which receives the torque instruction of the drive wheel motor to drive the wheel of the unicycle to rotate; a balance control actuator, which receives the control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle; The balance control actuator includes a reaction flywheel mechanism or a dual gyroscope mechanism. When the reaction flywheel mechanism is adopted, the balance motor drives the internal flywheel to rotate to generate a reverse torque to adjust the roll angle of the unicycle; when the dual gyroscope mechanism is adopted, the two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance. When a reaction flywheel mechanism is used, the pitch angle dynamics of the unicycle can be expressed as: ， in is the moment of inertia of the unicycle about the horizontal axis, is the pitch angular acceleration, m is the total mass, h is the height of the center of mass, g is the acceleration due to gravity, is the motor torque driving the unicycle wheel; the roll angle dynamics is formed by the reaction flywheel to form an equivalent inverted pendulum: ， in is the moment of inertia of the reaction flywheel-body system around the longitudinal axis, is the roll angular acceleration, is the reaction flywheel motor torque; The modules are connected via a communication link to form a closed-loop control system, enabling the unicycle to achieve self-balancing under the control of the strategy network.

2. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes: The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges; The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

3. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps and converting environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

4. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of decision-making, the state or action of several past moments is input into the enhanced state vector as historical features.

5. A self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning, characterized in that: A self-balancing unicycle control system for multimodal semantic perception and reinforcement learning as described in any one of claims 1 to 4 specifically comprises the following steps: S1, using imaging sensors, inertial measurement units, and encoder multimodal sensors to collect raw data on the unicycle's own posture and surrounding environment; S2. Processing the collected multimodal data includes extracting environmental semantic information from the imaging sensor image through visual semantic analysis, wherein the environmental semantic information includes road type, slope, obstacle location and size, obtaining the attitude angle, angular velocity, position and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information with the unicycle state data, and constructing an enhanced state vector as a representation of the current state of the unicycle; S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance; S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction; S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and cyclically execute steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

6. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 5, characterized in that: The visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time. The enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

7. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 5, characterized in that: The reinforcement learning strategy model adopts a deep reinforcement learning algorithm with an actor-critic architecture, consisting of a policy network and a value network. The policy network acts as an actor to output the unicycle control action, and the value network acts as a critic to estimate the value of the current state-action pair. The policy network and the value network achieve adaptive improvement through online updating of neural network parameters. During execution, the reinforcement learning strategy model continuously adjusts according to the unicycle's balance performance and task completion status to optimize the long-term cumulative reward. ,in is the discount factor, For immediate returns, For control rounds or learning rounds, The total length of the control or learning rounds.

8. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 7, characterized in that: Defining the instant return It includes the following items: giving negative rewards for the angle and angular velocity at which the unicycle's posture deviates from the upright position. The larger the angle deviation, the larger the negative value, to encourage reducing tilt; giving positive rewards for the time or distance the unicycle maintains balance to encourage extending the balance time; when there are obstacles, positive rewards are given if the unicycle avoids collision and successfully overcomes the obstacle, and large negative rewards are given if a collision or tipping occurs.

9. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 5, characterized in that: The control action in step S3 includes two components: longitudinal balancing torque and lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; the longitudinal balancing torque is achieved by adjusting the torque of the unicycle drive wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body. The balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

Images