Multi-mode semantic perception and reinforcement learning self-balancing monocycle control system and method

Abstract

The invention discloses a self-balancing monocycle control system and method based on multi-mode semantic perception and reinforcement learning. The multi-mode semantic perception and reinforcement learning are fused. The sensor module obtains wheelbarrow posture and environment multi-source information, and the information is processed by the semantic perception module to form enhanced state representation containing environment semantics and a wheelbarrow body state. The reinforcement learning control module generates a wheelbarrow balance control instruction through a deep neural network strategy based on an enhancement state, drives an execution mechanism to achieve pitching and rolling stable control, and can achieve strategy online self-adaptive adjustment through return feedback. According to the system, perception-decision-execution closed-loop control is constructed, visual sense and other multi-mode sensing understand environment semantics, self-adaptive strategy real-time iteration optimization is carried out, and balance and motion control of the monocycle in a complex dynamic environment are kept. According to the method, the environment understanding capability and the self-adaptability of the control strategy are improved, and the balance stability and the intelligent decision-making level of the monocycle under the unknown road condition are remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robots and balance control technology, and in particular to a self-balancing unicycle control system and method based on multimodal semantic perception and reinforcement learning. Background Art

[0002] Self-balancing unicycles have attracted attention for their compact structure and maneuverability. However, with only a single wheel in contact with the ground, unicycles are inherently nonlinear and unstable systems, requiring continuous attitude adjustments to maintain balance. Traditional unicycle control schemes typically employ a cascade PID controller coupled with multi-sensor data fusion. For example, one study designed a three-stage cascade control loop (current loop, velocity loop, and attitude loop) to stabilize motor current, wheel speed, and vehicle body inclination, respectively. In such systems, multiple sensors, such as an IMU, wheel encoders, and an electronic compass (magnetometer), are typically used to acquire attitude and velocity information. Kalman filtering is used to fuse sensor data to estimate the vehicle's attitude angle in real time, thereby achieving longitudinal and axial stability control using an inner and outer dual-loop PID control algorithm. This type of method works well in known models and fixed environments, but has the following limitations: First, parameter tuning is difficult, PID gains require repeated manual debugging, and it is difficult to balance stability and response speed under different loads and road conditions; second, there is a lack of environmental semantic perception. Traditional solutions only use low-level sensor values and cannot understand high-level semantic information about the road conditions or obstacles ahead, resulting in limited decision-making capabilities in complex environments; third, there is insufficient adaptability. Once the environment changes dynamically (for example, the road friction coefficient changes or new interference appears), the fixed rule controller is difficult to adjust in time, which may lead to imbalance.

[0003] In recent years, reinforcement learning (RL) has demonstrated unique advantages in robotic control. RL agents, rather than requiring manual design of control laws, autonomously learn optimal policies through repeated interactions with their environment. Compared to pre-programmed control algorithms, RL methods are able to learn from experience and adapt to changes. For example, in complex and unpredictable situations, RL controllers can gradually optimize their policies through trial and error, maintaining stable performance in dynamic environments. Furthermore, the development of multimodal semantic perception technology has enabled robots to understand high-level semantic information about their environment. Typical applications have demonstrated that robots equipped with deep reinforcement learning can utilize input from multiple sensors, such as cameras, LiDAR, and microphones, to achieve integrated perception and decision-making. Multimodal perception integrates visual, distance, and acoustic information into a unified state representation, giving robots a contextual understanding of their surroundings. This semantic-level perception facilitates more intelligent policy decisions, such as adjusting a unicycle's balance and speed based on the identified road slope and obstacle locations.

[0004] Given this background, existing unicycle control systems based on cascaded PID controllers struggle to meet the demands for adaptive balance and intelligent decision-making in complex environments. Therefore, a new control system is urgently needed that integrates multimodal semantic perception and reinforcement learning. This control system, based on deep semantic information about the environment, can autonomously learn optimal control actions through policy iteration, achieving closed-loop balance and motion control for the unicycle in diverse scenarios. Summary of the Invention

[0005] Technical purpose: In response to the shortcomings of existing technologies in adaptability to complex environments and intelligent decision-making, the present invention discloses a self-balancing unicycle control system and method based on multimodal semantic perception and reinforcement learning. The unicycle system perceives the environment and its own state through multimodal sensors such as imaging sensors and inertial measurement units, and introduces a semantic perception module to extract high-level semantic information of the environment. Then, an adaptive reinforcement learning control strategy is used to calculate the control instructions of the unicycle in real time based on the fused state and semantic information, thereby realizing intelligent adjustment of the unicycle's posture and balance.

[0006] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] A self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning, comprising:

[0008] Sensor module, at least one inertial measurement unit for measuring the unicycle's attitude angular velocity and acceleration; at least one encoder for measuring wheel speed or rotation angle; at least one imaging sensor for acquiring images or videos of the unicycle's surroundings;

[0009] The semantic perception module includes a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; and a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the unicycle's dynamic state and environmental semantic information.

[0010] The reinforcement learning control module includes a policy decision unit with a built-in deep reinforcement learning policy network, which uses the enhanced state vector as input to calculate control instructions for the unicycle, including torque instructions for the drive wheel motor and control instructions for the lateral balance actuator; a policy optimization unit, which is used to train and optimize the policy network using a reinforcement learning algorithm according to a predetermined reward function;

[0011] The actuator includes a drive motor, which receives the torque instruction of the drive wheel motor to drive the wheel of the unicycle to rotate; a balance control actuator, which receives the control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle;

[0012] The modules are connected via a communication link to form a closed-loop control system, enabling the unicycle to achieve self-balancing under the control of the strategy network.

[0013] Preferably, the balance control actuator includes a reaction flywheel mechanism or a dual gyroscope mechanism. When the reaction flywheel mechanism is adopted, the balance motor drives the internal flywheel to rotate to generate a reverse torque to adjust the roll angle of the unicycle; when the dual gyroscope mechanism is adopted, the two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance.

[0014] Preferably, the policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes:

[0015] The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges;

[0016] The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

[0017] Preferably, the semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps, and converting the environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

[0018] Preferably, the components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of the decision, the state or action of several past moments is input into the enhanced state vector as historical features.

[0019] A multimodal semantic perception and reinforcement learning self-balancing unicycle control method is applied to the multimodal semantic perception and reinforcement learning self-balancing unicycle control system as described above, specifically comprising the following steps:

[0020] S1, using imaging sensors, inertial measurement units, and encoder multimodal sensors to collect raw data on the unicycle's own posture and surrounding environment;

[0021] S2. Processing the collected multimodal data includes extracting environmental semantic information from the imaging sensor image through visual semantic analysis, wherein the environmental semantic information includes road type, slope, obstacle location and size, obtaining the attitude angle, angular velocity, position and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information with the unicycle state data, and constructing an enhanced state vector as a representation of the current state of the unicycle;

[0022] S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance;

[0023] S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction;

[0024] S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and cyclically execute steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

[0025] Preferably, the visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time, and the enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

[0026] Preferably, the reinforcement learning strategy model adopts a deep reinforcement learning algorithm with an actor-critic architecture, which consists of a strategy network and a value network. The strategy network acts as an actor to output the unicycle control action, and the value network acts as a critic to estimate the value of the current state-action pair. The strategy network and the value network are adaptively improved through online updating of neural network parameters. During the execution process, the reinforcement learning strategy model is continuously adjusted according to the unicycle's balance performance and task completion status to optimize the long-term cumulative return. ,in is the discount factor, For immediate returns, For control rounds or learning rounds, is the total length of the control round or learning round.

[0027] Preferably, the instant reward is defined as It includes the following items: giving negative rewards for the angle and angular velocity at which the unicycle's posture deviates from the upright position. The larger the angle deviation, the larger the negative value, to encourage reducing tilt; giving positive rewards for the time or distance the unicycle maintains balance to encourage extending the balance time; when there are obstacles, positive rewards are given if the unicycle avoids collision and successfully overcomes the obstacle, and large negative rewards are given if a collision or tipping occurs.

[0028] Preferably, the control action in step S3 includes two components, namely a longitudinal balancing torque and a lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; wherein the longitudinal balancing torque is achieved by adjusting the torque of the unicycle driving wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body, wherein the balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

[0029] Beneficial effects: The self-balancing unicycle control system and method based on multimodal semantic perception and reinforcement learning provided by the present invention has the following beneficial effects:

[0030] 1. Through multimodal semantic perception, the system can identify different terrains, obstacles, and driving environments, and respond in advance to situations such as slopes and uneven surfaces, thereby enhancing the adaptability and stability of the unicycle in unknown and complex environments. By utilizing reinforcement learning strategies, the system does not rely on manual parameter adjustment, but instead continuously optimizes the control strategy through interaction with the environment. It can automatically adjust parameters according to load changes and environmental disturbances, and always maintain optimal balance control performance.

[0031] 2. The present invention integrates multi-source sensor information and obtains the complete system state through state reconstruction. Even if individual sensor signals are unreliable or temporarily invalid, necessary state variables can still be inferred from other information sources, thereby improving the robustness and safety of the system. All links from sensing to control are closely connected, and the semantic understanding module and the reinforcement learning control strategy constitute the decision-making dual core: the former provides environmental understanding, and the latter gives optimized actions, realizing the logical closed loop of the robot's autonomous behavior, and possessing human-like intelligent decision-making capabilities rather than simple rule responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0033] Figure 1 Schematic diagram of the system architecture of the present invention;

[0034] Figure 2 This is a flow chart of the control method of the present invention;

[0035] Figure 3 Schematic diagram of multimodal semantic information fusion of the present invention;

[0036] Figure 4 This is a comparison chart of the simulation performance of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described more clearly and completely below by way of a preferred embodiment in conjunction with the accompanying drawings, but the present invention is not limited to the scope of the embodiment.

[0038] like Figure 1 As shown, a self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning includes:

[0039] The sensor module includes at least one inertial measurement unit for measuring the unicycle's attitude, angular velocity, and acceleration; at least one encoder for measuring wheel speed or rotation angle; and at least one imaging sensor for acquiring images or videos of the unicycle's surroundings. The sensor module acquires multimodal data, including the unicycle's own attitude, angular velocity, and position, as well as images or depth information of the surrounding environment.

[0040] The semantic perception module includes a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; and a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the unicycle's dynamic state and environmental semantic information.

[0041] In one embodiment, the components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of decision-making, the state or action of several past moments is input into the enhanced state vector as historical features.

[0042] The semantic perception module processes multimodal data. For example, it performs object detection and semantic segmentation on camera images, extracting semantic features such as road slope, surface type, and obstacle location. It also filters and fuses the inertial measurement unit and encoder data to estimate state variables such as the unicycle's current posture angle and angular velocity. Through state reconstruction, implicit state variables that cannot be directly measured (such as future balance trends) are incorporated into a unified state representation. This unified state encompasses both the physical state and the semantic description of the environment, enabling control decisions to incorporate global environmental awareness.

[0043] In one embodiment, the semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps and converting environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

[0044] The reinforcement learning control module includes a policy decision unit with a built-in deep reinforcement learning policy network, which uses the enhanced state vector as input to calculate the control instructions of the unicycle, including the torque instructions of the drive wheel motor and the control instructions of the lateral balance actuator; and a policy optimization unit for using a reinforcement learning algorithm to train and optimize the policy network according to a predetermined reward function.

[0045] In one embodiment, the policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes:

[0046] The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges;

[0047] The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

[0048] The reinforcement learning control module uses a policy network to achieve the mapping from state to action, and its policy is obtained through training of the reinforcement learning algorithm. Preferably, a deep reinforcement learning algorithm with an actor-critic structure is used to enable the policy network to select a control action based on the current semantic state, while the value network evaluates the long-term value of the action. During the training process, the reward function R is designed with the optimization goal of maintaining the balance of the unicycle and following the desired trajectory: for example, negative rewards are given for posture deviation, control energy consumption, collision risk, etc., and positive rewards are given for the length of balance time and successful obstacle passage. Policy training can first be performed offline in a physical simulation environment, and then online adaptation can be achieved through in-loop self-learning. When the unicycle is driving in the actual environment, the reinforcement learning control module can also achieve strategy evolution through online updates or strategy fine-tuning based on new situations perceived in real time, and continuously improve the strategy's adaptability to environmental changes.

[0049] The actuator includes a drive motor, which receives the torque instruction of the drive wheel motor to drive the wheel of the unicycle to rotate; and a balance control actuator, which receives the control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle.

[0050] In one embodiment, the balance control actuator includes a reaction flywheel mechanism or a dual gyroscope mechanism. When the reaction flywheel mechanism is adopted, the balance motor drives the internal flywheel to rotate to generate a reverse torque to adjust the roll angle of the unicycle; when the dual gyroscope mechanism is adopted, the two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance.

[0051] The modules are connected via a communication link to form a closed-loop control system, enabling the unicycle to achieve self-balancing under the control of the strategy network.

[0052] like Figure 2 As shown, the present invention also provides a self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning, which is applied to the self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning as described above, and specifically includes the following steps:

[0053] S1, using imaging sensors, inertial measurement units, and encoder multimodal sensors to collect raw data on the unicycle's own posture and surrounding environment;

[0054] S2. Processing the collected multimodal data includes extracting environmental semantic information from the imaging sensor image through visual semantic analysis, wherein the environmental semantic information includes road type, slope, obstacle location and size, obtaining the attitude angle, angular velocity, position and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information with the unicycle state data, and constructing an enhanced state vector as a representation of the current state of the unicycle;

[0055] In one embodiment, the visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time, and the enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

[0056] S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance;

[0057] In one embodiment, the control action in step S3 includes two components: a longitudinal balancing torque and a lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; wherein the longitudinal balancing torque is achieved by adjusting the torque of the unicycle drive wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body, wherein the balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

[0058] S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction;

[0059] S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and cyclically execute steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

[0060] Example 1

[0061] This embodiment describes the system architecture of the present invention in detail.

[0062] The self-balancing unicycle control system of the present invention includes a sensor module, a semantic perception module, a reinforcement learning control module and an actuator, as well as a unicycle body. The sensor module includes: an inertial measurement unit, which is used to measure the three-axis angular velocity and acceleration of the unicycle; a wheel encoder, which is used to measure the rotational speed and angular displacement of the unicycle wheel; an imaging sensor, which in this embodiment uses an environmental camera to collect an image sequence of the environment in front of the unicycle; and an optional distance sensor (such as ultrasonic or LiDAR) for detecting the distance to the obstacle in front. Each sensor is connected to the semantic perception module through a data bus. The semantic perception module includes: a posture estimation unit, which receives the inertial measurement unit and encoder data, and calculates the pitch angle of the unicycle through extended Kalman filtering or other data fusion algorithms. , roll angle and its angular velocity 、 The visual semantic understanding unit performs semantic segmentation and target detection on camera images, extracts information such as terrain categories (such as flat land, uphill, downhill, steps), obstacles (position and size), and reference landmarks; the state reconstruction unit synchronizes the time and space of the posture estimation and visual semantic information to generate a unified enhanced state vector The state vector It not only contains the dynamic state of the unicycle itself (angle, angular velocity, position, speed), but also includes key information about the environment (such as the obstacle of height h at d meters ahead, the current road slope In one specific implementation, the enhanced state It can be expressed as:

[0063]

[0064] Where v is the forward speed of the unicycle, is the unicycle linear acceleration, and Respectively represent the distance and height of the detected obstacle, Represents the current slope angle. The above state components are directly measured by sensors or calculated by the visual semantic understanding unit.

[0065] The reinforcement learning control module receives the enhanced state from the semantic perception module As input, the control instructions are calculated and output through the internal policy network. The reinforcement learning control module consists of two parts: the policy decision unit and the policy optimization unit. The policy decision unit has a built-in trained deep neural network (i.e., policy , the parameters are ), the network is in state The control action a of the unicycle is output as input. The control action here includes: the motor torque driving the unicycle wheel and the actuator control quantity for lateral balancing In a specific implementation, the unicycle can use an internal balancing wheel, i.e., a reaction flywheel or a dual gyroscope system to control the lateral balance. For example, the present embodiment uses a reaction flywheel, Represents the reaction flywheel motor torque, used to adjust the roll angle ; Indicates the driving wheel motor torque, used to control the pitch angle and forward / backward motion. Policy network output Acting on the actuator, the drive wheel motor receives To drive the unicycle forward or backward, the reaction flywheel motor receives The reaction flywheel is driven to rotate to generate lateral torque to achieve left-right balance. The action of the actuator changes the posture and displacement of the unicycle body.

[0066] The policy optimization unit is responsible for training and online tuning the policy network using a reinforcement learning algorithm. The training process can be performed in a simulated environment based on a unicycle dynamics model. The longitudinal dynamics of a unicycle can be approximated as an inverted pendulum, while the lateral inertia moment generated by the reaction flywheel allows for a state-space model. For example, based on the Newton-Euler equations, the pitch angular dynamics of a unicycle can be expressed as:

[0067]

[0068] in is the moment of inertia of the unicycle about the horizontal axis, is the pitch angle of the unicycle The second-order time derivative of the pitch angle is the angular acceleration, m is the total mass, h is the height of the center of mass, and g is the acceleration of gravity. is the driving wheel motor torque (positive for forward movement); the roll angle dynamics is formed by the reaction wheel to form an equivalent inverted pendulum:

[0069]

[0070] in is the moment of inertia of the reaction wheel-body system around the longitudinal axis, The strategy optimization unit uses the reinforcement learning algorithm to adjust the strategy parameters. , so that the unicycle can maintain and can all tend to zero, achieving a continuous and stable balance. Specifically, the introduction of cumulative returns ,in is the immediate return at time t, is the discount factor. The immediate return signal is designed according to the balance and target tracking requirements, such as:

[0071]

[0072] in 、 、 、 、 and is a weight constant used to balance the importance of each penalty and reward. 、 The pitch and roll deviations of t, 、 are the angular velocities corresponding to the pitch angle and roll angle, The indicator of the unicycle's continuous uprightness (e.g., a fixed positive reward is given for every second it remains upright without tipping over). and If the value is within the safety threshold, it is 1; otherwise, it is 0. An indicator for avoiding collisions with obstacles. If no collision occurs with an obstacle, it is 1, otherwise it is 0. It is iteratively updated through a policy gradient algorithm (such as proximal policy optimization PPO or deep deterministic policy gradient DDPG). , maximize the cumulative return R, and thus obtain the optimal strategy parameters After simulation training, the policy network is deployed to the policy decision unit for actual unicycle control. When the actual environment deviates from the trained model, the policy optimization unit can perform on-site retraining based on real-world feedback (e.g., fine-tuning the policy using a small amount of actual test data), enabling continuous evolution and adaptation of the policy.

[0073] Example 2

[0074] This embodiment describes the control process of the method in detail.

[0075] S1. When the unicycle is started, the sensor module starts working. The environmental camera collects image frames of the environment in front, the inertial measurement unit measures the angular velocity and acceleration of the vehicle body, the encoder records the wheel rotation information, and other sensors (such as ultrasonic waves) detect the distance to obstacles. Within one cycle, the data of each sensor is sent to the data buffer with a timestamp.

[0076] S2, the semantic perception module reads synchronized multimodal data frames from the buffer, first calls the visual semantic understanding unit to perform convolutional neural network inference on the image, and outputs pixel-level semantic segmentation results and target detection lists, such as identifying the road surface category (flat / pothole / ramp), the location and size of the obstacle in front, etc. Then the posture estimation unit performs Kalman filtering on the inertial measurement unit and encoder data to estimate the current unicycle pitch angle, roll angle and their rate of change. Finally, the state reconstruction unit fuses the visual semantic information with the posture estimation results to form the enhanced state at the current moment. This step may involve converting the obstacle position in the visual coordinate system to the unicycle coordinate system, for example, estimating the distance and height of the obstacle based on the intrinsic parameters of the environment camera and the pitch angle of the inertial measurement unit, and using it as part of the state.

[0077] S3, reinforcement learning control module receiving status After that, the policy decision unit calls the deep neural network to calculate the output action This step is equivalent to the policy function The strategy can be adjusted according to the semantic information of the environment in the state. For example, when an uphill slope with a large slope angle is detected, the strategy network may output a larger forward torque. To prevent backward tilt; when an obstacle is detected on one side that may cause tilt, the strategy network adjusts Correct the balance tendency in advance. The entire decision is completed within milliseconds to ensure the control frequency (typically above 100Hz).

[0078] Figure 3 The figure shows a schematic diagram of the multimodal semantic information fusion of the present invention. The visual semantic understanding unit performs visual semantic analysis on the real-time scene image, extracting environmental labels such as roads, obstacles, and slopes. The posture estimation unit uses the inertial measurement unit and wheel encoders to estimate the state of the unicycle, measuring the numerical states of the vehicle body such as pitch angle, roll angle, angular velocity, and forward speed. The outputs of both are input to the state reconstruction unit for state fusion. The enhanced state vector obtained after fusion combines environmental semantics and high-precision dynamic information. Finally, it is output to the strategy decision unit, providing a comprehensive and robust decision-making basis for other control strategies.

[0079] S4, the calculated control command is sent to the wheelbarrow actuator through the motor driver. Adjust the wheel acceleration or deceleration to control the forward and backward movement and pitch stability of the unicycle; the reaction flywheel motor is based on Changing the speed of the reaction flywheel generates a torque to offset the roll. Through the synergistic effect of these two torques, the unicycle is corrected to the upright balance direction in the current state.

[0080] S5: After the unicycle's controlled movement, the new state is fed back via sensors, entering the next cycle. The entire process is executed continuously, achieving real-time closed-loop control. If an abnormal situation occurs (such as sensor failure or exceeding the stability range), the system can enter a safe mode (such as emergency braking or auxiliary support mechanism intervention) to protect the unicycle and its surroundings.

[0081] Through this cyclical process, the unicycle can maintain self-balancing and complete driving tasks in a variety of complex situations. For example, when encountering complex terrain, the environmental camera detects a step ahead, and the semantic perception module recognizes that the step height exceeds the balancing threshold. The reinforcement learning strategy may then reduce the vehicle's speed and adjust its posture in preparation for parking. When environmental disturbances (such as crosswinds) cause the vehicle to tilt, the change in roll angle fed back by the inertial measurement unit causes the strategy to quickly correct the corresponding reaction wheel torque. Furthermore, through continuous online adjustments through reinforcement learning, system performance is optimized with use, demonstrating smoother control when repeatedly traversing a certain section of road.

[0082] Example 3

[0083] In order to verify the effectiveness of the present invention, a comparative simulation test was carried out. The simulation used a certain dynamic simulation platform to establish a unicycle model, including two degrees of freedom in pitch and roll, including wheel drive and reaction wheel control. The comparison objects were the traditional cascade PID controller and the reinforcement learning controller of the present invention. The PID controller adopts a classic dual-loop structure: the inner loop controls the pitch angle stability, the outer loop controls the forward speed, and adds roll angle correction. The parameters are repeatedly manually debugged to achieve a better result. The reinforcement learning controller uses the Actor-Critic algorithm for pre-training and converges after 1e5 steps of interaction in the simulation.

[0084] In test scenario 1 (maintaining static balance on flat ground), the unicycle initially deviates at a small angle of 1°. Figure 4 The curves of pitch angle variation with time under the action of two controllers are given. It can be seen that the PID controller makes the pitch angle approach 0 in about 2.5 seconds, but there is a The reinforcement learning control strategy of the present invention converges the angle error to 0 in about 1.5 seconds, with an overshoot amplitude less than half of that of PID and no obvious oscillation, showing faster stabilization speed and higher balancing accuracy. This is because the reinforcement learning strategy can output the correction torque in advance according to the state change trend, which is predictable and adaptive, while PID only calculates according to a fixed proportional integral differential based on the current error, and is limited in terms of fast response and oscillation suppression.

[0085] In test scenario two (forward balance with obstacles), a small obstacle 5 cm high was placed 2 meters ahead. The unicycle had to move forward slowly and cross the obstacle while maintaining balance. The results showed that the PID control strategy lacked perception of the obstacle and only suddenly increased the torque to try to maintain balance when the wheel hit the obstacle, resulting in a drastic fluctuation of about 3° in the pitch angle and the vehicle almost tipping over. In contrast, the system of the present invention used vision to perceive the obstacle in advance and slowed down when it approached. At the same time, the policy network had learned the posture adjustment method required to cross the obstacle, so the pitch angle only fluctuated gently by no more than 1°, and the unicycle successfully crossed the obstacle and continued to move forward. Test scenarios also included ramp driving and disturbance recovery, all of which verified the robustness and intelligence of the present invention. This proves that the multimodal semantic perception and reinforcement learning control system of the present invention has significant advantages over traditional methods in terms of stable control and environmental adaptation of the self-balancing unicycle.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A self-balancing unicycle control system based on multimodal semantic perception and reinforcement learning, characterized in that: include: Sensor module, at least one inertial measurement unit, used to measure the angular velocity and acceleration of the unicycle; at least one encoder, used to measure the wheel speed or rotation angle; at least one imaging sensor for acquiring images or videos of the environment surrounding the unicycle; The semantic perception module includes a posture estimation unit for fusing inertial measurement unit and encoder data to calculate the pitch angle, roll angle and angular velocity of the unicycle; a visual semantic understanding unit for performing semantic analysis on the images acquired by the imaging sensor to identify road types, obstacles and other semantic features in the environment; and a state reconstruction unit for fusing the posture estimation results with the visual semantic features to generate an enhanced state vector containing the unicycle's dynamic state and environmental semantic information. A reinforcement learning control module includes a strategy decision unit and a built-in deep reinforcement learning strategy network, which uses the enhanced state vector as input to calculate control instructions for the unicycle, including torque instructions for the drive wheel motor and control instructions for the lateral balance actuator; A strategy optimization unit, configured to train and optimize the strategy network using a reinforcement learning algorithm according to a predetermined reward function; The actuator includes a drive motor, which receives the torque instruction of the drive wheel motor to drive the wheel of the unicycle to rotate; a balance control actuator, which receives the control instruction of the lateral balance actuator to generate a lateral torque to adjust the roll angle of the unicycle; The modules are connected via a communication link to form a closed-loop control system, enabling the unicycle to achieve self-balancing under the control of the strategy network.

2. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The balance control actuator includes a reaction flywheel mechanism or a dual gyroscope mechanism. When the reaction flywheel mechanism is adopted, the balance motor drives the internal flywheel to rotate to generate a reverse torque to adjust the unicycle's roll angle; when the dual gyroscope mechanism is adopted, two high-speed rotating gyroscopes generate a precession torque to stabilize the lateral balance.

3. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The policy optimization unit is configured to pre-train the policy network in a simulation environment and perform online learning updates based on feedback data during the actual operation of the unicycle. The policy optimization unit includes: The simulation training module uses a unicycle dynamics model and a virtual environment to perform offline training on the policy network, and uses a reinforcement learning algorithm to iteratively update the policy parameters until the cumulative reward converges; The online adaptation module monitors the unicycle's balance performance during actual operation. When new working conditions or performance degradation occur, it triggers online fine-tuning of the policy network. It further trains or adjusts the policy parameters through a small amount of real-time collected data, thereby maintaining the policy's adaptability to environmental changes.

4. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The semantic perception module further includes a time synchronization and coordinate transformation unit for aligning data from different sensors according to timestamps and converting environmental information output by the imaging sensor from the image coordinate system to the body coordinate system or global coordinate system of the unicycle. Through the time synchronization and coordinate transformation, it is ensured that the visually recognized road type, obstacles and other semantic features are described at the same time and in the same reference system as the unicycle posture state, so as to facilitate fusion into an enhanced state vector input for reinforcement learning control module decision-making.

5. The multimodal semantic perception and reinforcement learning self-balancing unicycle control system according to claim 1, characterized in that: The components in the enhanced state vector can be expanded according to task requirements, including task-related state or historical state features. When the unicycle is used for target tracking or path following, the enhanced state vector further includes target position deviation or path deviation information; when it is necessary to improve the smoothness of decision-making, the state or action of several past moments is input into the enhanced state vector as historical features.

6. A self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning, characterized in that: A self-balancing unicycle control system for multimodal semantic perception and reinforcement learning as described in any one of claims 1 to 5 specifically comprises the following steps: S1, using imaging sensors, inertial measurement units, and encoder multimodal sensors to collect raw data on the unicycle's own posture and surrounding environment; S2. Processing the collected multimodal data includes extracting environmental semantic information from the imaging sensor image through visual semantic analysis, wherein the environmental semantic information includes road type, slope, obstacle location and size, obtaining the attitude angle, angular velocity, position and speed of the unicycle through sensor data fusion and state estimation, fusing the environmental semantic information with the unicycle state data, and constructing an enhanced state vector as a representation of the current state of the unicycle; S3. Input the enhanced state vector into the reinforcement learning policy model, and calculate the control action of the unicycle based on the pre-trained deep reinforcement learning policy network, including the output torque of the drive wheel motor and the control amount of the actuator for lateral balance; S4, sending the control action to the actuator of the unicycle to drive the unicycle forward and backward and adjust the balance so that the unicycle posture is adjusted towards the upright balance direction; S5. Feedback the new state of the unicycle after execution to the semantic perception module through the sensor, update the enhanced state vector, and cyclically execute steps S2 to S4, thereby forming a closed-loop control to achieve self-balancing travel of the unicycle.

7. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 6, characterized in that: The visual semantic analysis in step S2 includes a deep convolutional neural network algorithm, which performs pixel-level semantic segmentation and object detection on the imaging sensor image to obtain ground category, driving slope, obstacle position and size, and target path semantic information; the state estimation uses an extended Kalman filter or an equivalent filter to fuse the gyroscope and accelerometer data of the inertial measurement unit with the encoder data to calculate the pitch angle, roll angle and respective angular velocity of the unicycle in real time. The enhanced state vector includes at least the pitch angle, roll angle and angular velocity of the unicycle, the forward speed, and the current road slope angle, the distance and height of the obstacle ahead obtained based on the visual semantic analysis.

8. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 6, characterized in that: The reinforcement learning strategy model adopts a deep reinforcement learning algorithm with an actor-critic architecture, consisting of a policy network and a value network. The policy network acts as an actor to output the unicycle control action, and the value network acts as a critic to estimate the value of the current state-action pair. The policy network and the value network achieve adaptive improvement through online updating of neural network parameters. During execution, the reinforcement learning strategy model continuously adjusts according to the unicycle's balance performance and task completion status to optimize the long-term cumulative reward. ,in is the discount factor, For immediate returns, For control rounds or learning rounds, is the total length of the control round or learning round.

9. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 8, characterized in that: Defining the instant return It includes the following items: giving negative rewards for the angle and angular velocity of the unicycle's posture deviating from the upright position. The larger the angle deviation, the larger the negative value, to encourage reducing tilt; giving positive rewards for the time or distance the unicycle maintains balance to encourage extending the balance time; when there are obstacles, positive rewards are given if the unicycle avoids collision and successfully overcomes the obstacle, and large negative rewards are given if a collision or tipping occurs.

10. The self-balancing unicycle control method based on multimodal semantic perception and reinforcement learning according to claim 6, characterized in that: The control action in step S3 includes two components: longitudinal balancing torque and lateral balancing torque, which respectively control the front and rear pitch and plane roll of the unicycle; the longitudinal balancing torque is achieved by adjusting the torque of the unicycle drive wheel motor, and the lateral balancing torque is achieved by adjusting the balance control actuator inside the unicycle body. The balance control actuator is a reaction flywheel or a dual gyroscope, which is used to generate a control torque to offset the roll angle.

Images