An unmanned tracked forklift control method based on VLA-hybrid reinforcement learning

Abstract

The application provides a kind of unmanned tracked forklift control method based on VLA-hybrid reinforcement learning, belongs to unmanned forklift control technical field, including by building multi-module collaborative control framework, multi-modal perception and task semantic fusion are realized using VLA, then by hybrid reinforcement learning model generates and corrects action instruction, realizes closed-loop control through PID accurate execution and error correction module, finally through offline training and online fine-tuning iteration optimization model, greatly improves the operation stability and precision under complex terrain.The application adopts the above-mentioned unmanned tracked forklift control method based on VLA-hybrid reinforcement learning, solves the problems of error accumulation, poor adaptation to extreme scenarios, uninterpretable decision, disconnection between perception and execution and low sample efficiency in the prior art.

Description

Technical Field

[0001] This invention relates to the field of unmanned forklift control technology, and in particular to a control method for unmanned tracked forklifts based on VLA-hybrid reinforcement learning. Background Technology

[0002] Unmanned tracked forklifts are widely used in material handling and stacking operations in fields, mines, construction sites and other scenarios. Their control methods are mainly divided into two categories: traditional control and single intelligent control.

[0003] Traditional control methods primarily rely on PID control and fuzzy control, which essentially control forklift movements through preset parameters and fixed logic. Taking traditional PID control as an example, while it can achieve basic control, it cannot adjust parameters in real time according to changes in terrain and load, leading to error accumulation over long-term operation. For instance, during heavy-duty transport, load changes alter the track drive force requirements, and a fixed proportional coefficient causes fluctuations in travel speed, resulting in positioning deviations that accumulate over time. After one hour of operation on uneven terrain, the travel positioning error can exceed the threshold required for industrial operations.

[0004] Single intelligent control methods mainly include pure reinforcement learning control and pure vision recognition control. Pure reinforcement learning control learns strategies through interaction between the agent and the environment, but it suffers from low sample efficiency, susceptibility to errors in the initial exploration phase, and difficulty in handling complex terrain. Its commonly used DQN algorithm relies only on single-state feedback, lacking multimodal environmental information support, resulting in low decision-making accuracy in complex terrain. For example, it can easily generate unreasonable track speed commands when operating on uneven terrain, causing the forklift to slip. Pure vision recognition control relies on a large amount of labeled data and has poor adaptability to extreme scenarios. For example, in dusty scenes, pure CNN vision recognition can cause image features to be occluded, preventing the convolutional layers from extracting effective features, leading to obstacle recognition failure and forklift collisions.

[0005] Some existing solutions attempt to combine reinforcement learning with vision, but most are simply superimposed without deep integration. They cannot simultaneously address adaptive perception, interpretable decision-making, and precise execution in complex terrains, and they do not optimize control strategies for the structural characteristics of tracked forklifts. Summary of the Invention

[0006] The purpose of this invention is to provide a control method for unmanned tracked forklifts based on VLA-hybrid reinforcement learning, which solves the problems of error accumulation, poor adaptation to extreme scenarios, uninterpretable decision-making, disconnect between perception and execution, and low sample efficiency in the existing unmanned tracked forklift control technology.

[0007] To achieve the above objectives, this invention provides a control method for an unmanned tracked forklift based on VLA-hybrid reinforcement learning, comprising: S1. Build the VLA-hybrid reinforcement learning control framework, initialize the parameters, and the control framework relies on the multimodal perception module, hybrid reinforcement learning decision module, precise execution module, and error correction module to work together. S2. Each sensor collects data at a preset frequency. After being processed by a targeted preprocessing algorithm, the VLA visual encoder extracts features from the multimodal data to generate a visual token. The language encoder converts the preset task instructions into semantic representations, fuses them with the visual tokens, and outputs the fused environment-task feature vector. S3. Generate the state space of the hybrid reinforcement learning model, design a multi-dimensional dense reward function, input the output fusion feature vector into the hybrid reinforcement learning model, filter out candidate actions, the VLA policy decoder combines environment-task semantics to correct the candidate actions, finally output the optimal control action instruction, and convert the optimal action instruction into a natural language explanation. S4. The on-board controller converts the optimal control action command into an electrical signal and transmits it to each execution module via the CAN bus. It uses track differential drive PID control and fork height PID control to ensure accurate execution of the action command. S5. The error correction module receives the feedback status data in real time, calculates the error and compares it with the preset threshold, determines the error level and generates targeted correction instructions, and updates the model parameters synchronously. The VLA visual encoder identifies the terrain type and adjusts the action and reward function weights according to different terrains. S6. Model iterative optimization, including offline training and online fine-tuning. Offline training is based on CAD data to build a photorealistic digital environment and input a large number of samples to train the VLA model and the hybrid reinforcement learning model. Online fine-tuning is to periodically extract samples from the experience replay pool to fine-tune the model during the actual operation of the forklift.

[0008] Preferably, the state space of the hybrid reinforcement learning model includes the output environment-task feature vector, forklift posture data, track speed data, load data, and current error data.

[0009] Preferably, the specific method by which the VLA visual encoder extracts features from multimodal data is as follows: The PointNet++ algorithm is used for point cloud feature extraction from LiDAR. Layered feature extraction is performed on 3D point cloud data for forklift operation scenarios. The formula is as follows: ; in, Given the spatial coordinates of a single point cloud, the MLP is a 2-layer fully connected network. Max-pooling uses a 5×5 window to extract global features from the local point cloud, and the final output is... It accurately represents terrain undulations and the spatial location of obstacles; RGB image feature extraction uses a ResNet-50 network, removing the last fully connected layer and retaining the convolutional and pooling layers. The convolutional layer parameters are optimized for forklift operation scenarios to extract visual features from the image. The formula is as follows: ; in, The image is a preprocessed RGB image; ResNet50 is connected via residual connections. , For residual mapping; Deep feature extraction uses a CNN network. For forklift operation scenarios, the input depth image is used to extract depth features using the following formula: ; in, This is the preprocessed depth image. It is a depth feature vector; Multimodal feature fusion employs a Self-Attention mechanism, fusing the three features mentioned above, with the following formula: ; in, , , , , , All are learnable weight matrices. For feature dimensions.

[0010] Preferably, the language encoder formula is: ; in, This refers to the task instruction.

[0011] Preferably, the policy decoder formula is: ; in, It provides context-task fusion features; the Transformer decoder contains 12 attention heads and a Feed-Forward network.

[0012] Preferably, the hybrid reinforcement learning model adopts a hybrid architecture of "PPO policy gradient and DQN value evaluation", including: The PPO-Clip algorithm is used to constrain the policy update magnitude and avoid drastic fluctuations in forklift movements. The objective function is: ; in, , For the current strategy parameters In state Next generation action The probability, The strategy before the update; The dominant function; The Double DQN algorithm is used to avoid overestimation and accurately assess the safety and effectiveness of actions. The core update formula is: ; in, To adapt the parameters to the model's convergence speed, These are the current Q-network parameters. The target Q network parameters; The optimal action for the next state selected by the current Q network; PPO generates initial actions, DQN evaluates their value, VLA outputs the initial actions, and weighted fusion yields candidate actions, using the following formula: ; in, This is the initial action vector for PPO. The initial action vector for VLA, For PPO action weighting coefficients, For VLA action weighting coefficients.

[0013] The preferred formula for the multi-dimensional dense reward function is: ; in, Rewards for completing the task. For accuracy rewards, As a safety reward, Rewards for efficiency This is an error penalty.

[0014] Preferably, in error correction and terrain adaptive adjustment, the error correction module determines the error level and generates correction instructions in the following way: calculate Driving positioning error, Steering angle error The fork height error is compared with a preset threshold to determine the error level: no error, slight error, or severe error. The expression for slight error correction is: ; in, This represents the actual error. For the error threshold, For correction factor, This is the motion correction amount.

[0015] Preferably, in the model iterative optimization, offline training constructs a photorealistic digital environment based on CAD data, simulating various complex terrains and extreme scenarios. A large number of environmental and task samples are input to train the VLA model and the hybrid reinforcement learning model offline, iteratively optimizing the model parameters so that the model converges to the optimal policy. Online fine-tuning is performed during the actual operation of the forklift. The state data, action data, and reward value of each closed-loop control are stored in the experience replay pool. Samples are periodically extracted to fine-tune the model online and update the model weights.

[0016] Therefore, the present invention employs the above-mentioned VLA-hybrid reinforcement learning-based unmanned tracked forklift control method, and the technical effects are as follows: 1. Solve the pain point of error accumulation: The error correction module detects and corrects operational errors in real time, while updating model parameters to reduce error accumulation at the source and meet the high-precision requirements of industrial operations.

[0017] 2. Improved adaptability to extreme scenarios: Based on the VLA model, multimodal perception integrates LiDAR, depth data and visual data to complement and compensate for the shortcomings of a single sensor. In extreme scenarios such as dust, strong light, low temperature and complex terrain, the perception accuracy reaches more than 98%, and the adaptability is significantly better than existing technologies.

[0018] 3. Achieve interpretable decisions: The VLA model transforms the optimal action instructions into natural language explanations, breaking the limitations of the traditional "black box" decision-making in reinforcement learning. This makes it easier for maintenance personnel to troubleshoot problems and improves operational safety and maintainability.

[0019] 4. Improved sample efficiency and operational efficiency: The hybrid reinforcement learning architecture, combined with a multi-dimensional dense reward function, reduces the number of samples required for model convergence, improving sample efficiency by 40%; closed-loop control enables rapid response from perception to decision-making to execution, improving operational efficiency by more than 35% compared to traditional control. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the control framework for the unmanned tracked forklift based on VLA-hybrid reinforcement learning according to the present invention; Figure 2 This is a flowchart illustrating the steps of the unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 like Figures 1-2 As shown, this invention provides a control method for unmanned tracked forklifts based on VLA-hybrid reinforcement learning. It deeply integrates VLA and hybrid reinforcement learning, relying on VLA to achieve multimodal adaptive perception of the environment and interpretability of decisions, and relying on hybrid reinforcement learning to achieve rapid optimization and precise execution of control strategies. At the same time, combined with the structural characteristics of tracked forklifts, a terrain adaptation module and an error correction module are designed to solve the core pain points of traditional systems.

[0024] The control method of this invention relies on four main modules: a multimodal perception module (based on VLA), a hybrid reinforcement learning decision-making module, a precise execution module, and an error correction module. These four modules work together to achieve autonomous, stable, and precise control of the unmanned tracked forklift. The specific steps are as follows: S1. Build the VLA-hybrid reinforcement learning control framework and initialize the parameters.

[0025] An unmanned tracked forklift was constructed, equipped with LiDAR, an RGB camera, a depth camera, an IMU (Inertial Measurement Unit), a track speed sensor, and a load sensor. All sensors are connected to the onboard controller, which communicates with the forklift's track drive module, fork lifting module, and steering module to achieve real-time signal transmission. The sampling frequencies of each sensor are set as follows: LiDAR 10Hz (ensuring real-time capture of obstacles and terrain contours), RGB camera 30Hz (clearly identifying materials and shelves), depth camera 20Hz (accurately acquiring obstacle distances), IMU 100Hz (real-time acquisition of forklift attitude), track speed sensor 50Hz (real-time feedback of track operation status), and load sensor 10Hz (real-time monitoring of fork load), ensuring real-time data acquisition. The onboard controller uses an NVIDIA Jetson AGX Xavier processor with a computing power of ≥21 TOPS, meeting the requirements for real-time multimodal data processing and model inference. Communication uses a CAN bus with a transmission latency of ≤50ms, ensuring rapid transmission of control commands to each execution module and avoiding action lag.

[0026] Based on the OpenVLA open-source model, this invention optimizes the visual encoder and policy decoder to adapt to the multi-sensor data and control requirements of unmanned tracked forklifts. The original OpenVLA visual encoder only supports image data and cannot adapt to the multi-sensor data of forklifts. This invention, after optimization, supports multi-modal fusion of LiDAR 3D point cloud data, RGB image data, and depth data. It adopts a structure of "point cloud feature extraction + image feature extraction + depth feature extraction + feature fusion," with each part designed to adapt to forklift operation scenarios.

[0027] The PointNet++ algorithm is used for lidar point cloud feature extraction. For forklift operation scenarios (detection range 5-10 meters, needing to capture obstacles and terrain contours), hierarchical feature extraction is performed on the 3D point cloud data. The formula is as follows: ; in, The coordinates of a single point cloud correspond to the locations of obstacles and terrain in the forklift's surrounding environment. The MLP is a 2-layer fully connected network with an input dimension of 3 and an output dimension of 128. The ReLU activation function is used to extract local features of a single point cloud. Max-pooling uses a 5×5 window to extract global features from the local point cloud, and the final output is... It accurately represents the terrain undulations and the spatial location of obstacles.

[0028] RGB image feature extraction uses a ResNet-50 network, removing the last fully connected layer and retaining the convolutional and pooling layers. For forklift operation scenarios (requiring material, shelf, and label recognition), the convolutional layer parameters are optimized to extract visual features from the image. The formula is as follows: ; in, The image is a preprocessed RGB image; ResNet50 is connected via residual connections. , The residual mapping includes two convolutional layers and one batch normalization layer to avoid gradient vanishing and ensure effective extraction of deep visual features. It is an image feature vector, adapted to material recognition requirements.

[0029] Depth feature extraction uses a CNN network. For forklift operation scenarios, the input depth image is used to extract depth features, and the formula is as follows: ; in, The image is a preprocessed depth image; the CNN contains 3 convolutional layers and 2 max pooling layers, with ReLU activation function. As a deep feature vector, it accurately represents the distance between the obstacle and the forklift, as well as the difference in terrain height.

[0030] Multimodal feature fusion employs a self-attention mechanism. , , To adapt to the perception needs of forklifts in extreme scenarios, the formula is: ; in, , , , , , All are learnable weight matrices, optimized through offline training; For feature dimensions; To integrate visual tokens, which can simultaneously represent environmental spatial location, visual details, and depth information, the weight of LiDAR and depth features is automatically increased in dusty scenarios to avoid perception failure.

[0031] The language encoder uses the BERT-based model to transform text instructions into semantic representations for forklift operation tasks. The core formula is: ; in, The task instructions (such as "grab materials from shelf number 3 and move them to the area X=10m, Y=5m") are input into the BERT model after WordPiece word segmentation and word embedding. BERT contains a 6-layer Transformer encoder to extract the semantic associations of the task. It is a semantic feature vector that accurately represents the task objectives and action requirements.

[0032] The policy decoder employs a 6-layer Transformer decoder to map visual-language fusion features into continuous forklift action instructions. The core formula is: ; in, This provides environment-task fusion features; the Transformer decoder contains 12 attention heads and a Feed-Forward network, and its output... These correspond to the rotational speeds of the left track wheel. (m / s), right wheel speed (m / s), fork lifting height (m), directly adapted to forklift execution modules.

[0033] The hybrid reinforcement learning model adopts a hybrid architecture of "PPO (Policy Gradient) and DQN (Value Evaluation). PPO generates continuous control actions, and DQN selects the optimal actions to avoid the limitations of a single policy and adapt to the continuous and stable operation requirements of forklifts.

[0034] The PPO-Clip algorithm is used to constrain the policy update magnitude and avoid drastic fluctuations in forklift movements. The objective function is: ; in, , For the current strategy (parameters) In state Next generation action The probability, The strategy before the update; The dominance function is calculated using GAE generalized dominance estimation, and the formula is as follows: , ; in, (Adapt to forklift operations and pay attention to the impact of recent actions). , For the state value function (PPO value network output); The clipping coefficient constrains the update range of the strategy, preventing sudden changes in track speed and steering angle from causing material to fall.

[0035] The Double DQN algorithm is used to avoid overestimation and accurately assess the safety and effectiveness of actions. The core update formula is: ; in, (Adapt to the convergence speed of the model). These are the current Q-network parameters. The target Q network parameters; The optimal action for the next state is selected from the current Q network to avoid overestimating the value of dangerous actions.

[0036] PPO generates initial actions DQN assesses its value VLA outputs initial action The candidate actions are obtained by weighted fusion, and the formula is: ; in, This is the initial action vector for PPO. The initial action vector for VLA, The PPO action weighting coefficient (PPO is the dominant exploratory action, adaptable to complex terrain and unknown situations). The weights are VLA action weights (VLA dominates stability, while multimodal perception ensures safety). The weights are optimized through offline training to ensure that the actions balance exploration and stability.

[0037] S2. Multimodal environment perception and preprocessing based on VLA visual encoder.

[0038] Each sensor collects data at a preset frequency. The lidar captures obstacles and terrain contours within 5-10 meters (1000 points / frame); the RGB camera collects environmental images and identifies materials and shelves; the depth camera obtains obstacle distances and terrain height differences; the IMU collects forklift tilt angles and accelerations; the track speed sensor provides feedback on the left and right wheel speeds; and the load sensor monitors the load on the forks. All data is transmitted to the vehicle controller via the CAN bus with a transmission delay of ≤50ms to ensure real-time data performance and provide support for perception and decision-making.

[0039] To address the issues of sensor noise and data heterogeneity in forklift operation scenarios, a targeted preprocessing algorithm is employed to ensure effective data consistency.

[0040] The lidar point cloud preprocessing uses pass-through filtering and voxel filtering to remove noise and downsample, making it suitable for forklift operation scenarios.

[0041] The expression for a pass-through filter is: ; in, The Z-axis height of the point cloud is used to filter out ground and high-altitude noise. The expression for voxel filtering is: ; in, The average number of point clouds within a voxel is used as the representative point of the voxel, reducing the number of point clouds from 1000 points / frame to 200 points / frame, thereby improving processing speed.

[0042] RGB image preprocessing employs Gaussian filtering for noise reduction and normalization, adapting to scenes with dust and strong light interference.

[0043] The Gaussian filter expression is: ; in, It is a Gaussian function. Standard deviation, These are the original image pixel values. These are the filtered pixel values; The normalization expression is: ; The image is scaled to 224×224 and mapped to the [0,1] interval to adapt to ResNet-50 input.

[0044] The depth data preprocessing uses median filtering to remove outliers and normalization to adapt to depth misdetection scenarios.

[0045] The median filter expression is: ; in, This is the original depth value. This is the filtered depth value. This is a median operation, which can effectively remove isolated outliers in depth data.

[0046] Then normalization is performed, and the normalization expression is: ; Mapped to the [0,1] interval to adapt to deep feature extraction.

[0047] IMU data preprocessing employs Kalman filtering to remove vibration noise, adapting to forklift driving vibration scenarios. The state equation and observation equation are as follows: The state equation is expressed as follows: The expression for the observation equation is: ; in, For the voxel in row i and column j, This represents the number of point clouds within a voxel. Given the coordinates of a single point cloud within a voxel, the average value of all point clouds within the voxel is taken as the representative point of the voxel to achieve downsampling and ensure the efficiency of subsequent feature extraction. For the state vector ( The tilt angle, (angular velocity) The state transition matrix ( (IMU sampling period) For the control matrix, Angular acceleration input (acquired by IMU) Process noise (variance) (Adapted to the noise generated by forklift vibration) The observed value (tilt angle acquired by the IMU). For the observation matrix, For observation noise (variance) ).

[0048] The update steps of Kalman filtering include: , ; , , ; in, To predict the state, To predict the covariance matrix, For Kalman gain, This is the updated state. The updated covariance matrix, This is the identity matrix. Kalman filtering can reduce noise in the IMU tilt angle by more than 80%, improving the accuracy of attitude data and ensuring the precision of subsequent terrain adaptation and error correction.

[0049] Since the sampling frequencies of each sensor are different, linear interpolation is used to align all preprocessed data to the same timestamp (based on the sampling timestamp of the IMU, with one timestamp every 0.01s) to ensure time synchronization of multimodal data and avoid perception deviations caused by time differences.

[0050] The VLA visual encoder extracts features from multimodal data to generate visual tokens; the language encoder transforms the pre-set task instructions (in text form) into semantic representations, merges them with the visual tokens, and outputs a fused environment-task feature vector, achieving simultaneous completion of "environmental perception and task understanding". This solves the problem that single visual perception cannot adapt to extreme scenarios (such as dust scenarios, where LiDAR and depth data complement each other to avoid visual recognition failure).

[0051] Visual token generation involves inputting preprocessed LiDAR point cloud data, RGB image data, and depth data into the optimized visual encoder's point cloud feature extraction module, image feature extraction module, and depth feature extraction module, respectively, to obtain... (Spatial features representing terrain and obstacles) (Visual characteristics representing materials and shelves) (Representing obstacle distance and terrain height difference), obtained through attention mechanism fusion. This refers to visual tokens, where an attention mechanism automatically assigns weights. For example, in a dusty scene, the weight of image features is reduced, while the weight of LiDAR and depth features is increased, ensuring that the visual token can accurately represent the environmental state.

[0052] Semantic representation generation involves inputting the task instruction (e.g., "grab the material from shelf number 3 and move it to the designated area (coordinates X=10m, Y=5m)") into the BERT-based language encoder. This process involves word segmentation (using WordPiece to extract words such as "grab," "shelf number 3," "material," "move," "designated area," "X=10m," and "Y=5m"), embedding (converting each word into a 768-dimensional word vector, where coordinate information is converted into numerical embeddings), and bidirectional Transformer encoding (a 6-layer Transformer encoder extracts semantic relationships between words, such as the relationship between "shelf number 3" and "material"), resulting in a semantic feature vector. This semantic vector can accurately represent the target (such as material location, transport destination) and action requirements (such as grasping, transport) of the task.

[0053] Environment-task feature fusion will integrate visual tokens With semantic representation Concat the vectors to obtain the fused environment-task feature vector. The formula is: ; This vector contains both multimodal features of the environment (terrain, obstacles, material locations) and task semantics (operation objectives, action requirements), providing comprehensive input information for the subsequent decision-making module and ensuring that the decision-making module can generate precise control actions based on the environment and task.

[0054] S3, Hybrid reinforcement learning decision-making that integrates VLA policy decoders.

[0055] The output environment-task feature vector, forklift attitude data, track speed data, load data, and current error data together constitute the state space S of the hybrid reinforcement learning model, where S = [environment-task feature vector, attitude data, track speed data, load data, error data], i.e. ,in, For tilt angle, X-axis acceleration, Y-axis acceleration , All are track speeds. For load, This is the driving positioning error. The value represents the steering angle error; the action space A represents the forklift's executable actions. in, All speeds are forklift travel speeds. The stacking height is denoted as A; the motion space A includes the left / right wheel speed of the track, the steering angle, and the lifting height of the forks, all of which are continuous values.

[0056] To avoid the low sample efficiency problem caused by sparse rewards, a multi-dimensional dense reward function is designed. The reward function formula is as follows: ; in, Rewards for completing the task. For accuracy rewards, As a safety reward, Rewards for efficiency This is an error penalty.

[0057] Task completion reward Segmented incentives adapt to the phased needs of material handling operations: ; Precision reward Encourage high-precision operations and reduce error accumulation: ; Safety Rewards To ensure operational safety and prevent collisions and abnormal postures: ; Efficiency Rewards Adapt to different loads and terrains to optimize operational efficiency: ; in, The average speed of the forklift. This is the optimal speed.

[0058] Error penalty To suppress error accumulation, the formula is: .

[0059] The output fused feature vector is input into the hybrid reinforcement learning model. PPO generates an initial continuous action A based on the current state S. DQN evaluates the value of the initial action A and outputs the action value Q. The action with the highest Q value is selected as the candidate action. The VLA policy decoder combines the environment-task semantics to modify the candidate actions (such as modifying the track speed in complex terrain to avoid slippage), and finally outputs the optimal control action command.

[0060] Initial action generation: Input the hybrid reinforcement learning model, PPO according to the current state Initial continuous actions are generated through a policy network. It follows a Gaussian distribution. This ensures the continuity of the actions.

[0061] Motion Value Assessment: DQN Value assessment is performed using linear interpolation. ,filter The action with the highest value is selected as the candidate action. Simultaneously, the VLA policy decoder outputs the initial action. as a candidate action .

[0062] Motion fusion and correction: according to the formula: By fusing candidate actions, the VLA policy decoder combines context-task semantics (such as "undulating terrain" and "overload") to further refine the actions. Uneven terrain (height difference > 5cm): Correction ,formula: ， ; in, Due to the terrain elevation difference, reduce the engine speed to avoid slippage; Heavy load ( tons): Correction The formula is: ; Lower the fork height to improve stability; The final output is the optimal control action. .

[0063] The VLA model translates the optimal action commands into natural language explanations (such as "The current terrain is uneven, reduce the track speed to 0.5m / s, adjust the steering angle to 5° to avoid slippage"), and transmits them synchronously to the display module of the vehicle controller, making the decision-making process interpretable and facilitating maintenance and troubleshooting.

[0064] S4. Precise execution and real-time feedback.

[0065] The vehicle controller will The signals are converted into electrical signals and transmitted to each execution module via the CAN bus. The differential speed drive PID control and fork height PID control are used to ensure the precise execution of the action commands.

[0066] The track drive control uses PID control, with a target speed of [missing value]. The control formula is: ; ; in, , , All are adjustment coefficients. Through PID control, the track speed control error is ≤0.01m / s, ensuring smooth travel.

[0067] The forklift lifting control uses PID control, with a target height of [missing information]. The control formula is: ; in, , , With a lifting accuracy of ≤2mm, it is suitable for material grabbing and stacking needs.

[0068] Steering control is achieved through the speed difference between the left and right wheels, which is controlled by PID control to ensure that the steering angle error is ≤2°. The formula is: PID control This makes the actual steering angle closer to... .

[0069] During execution, each sensor continuously collects data at a preset frequency. After preprocessing, the data is fed back in real time to the multimodal perception module (updating environmental features) and the error correction module (detecting errors), forming a closed loop of "perception-decision-execution-feedback". The feedback delay is ≤50ms, ensuring that the action can be adjusted in real time to adapt to dynamic environmental changes (such as sudden obstacles).

[0070] S5. Error correction and terrain adaptive adjustment.

[0071] The error correction module receives real-time feedback status data and calculates... (Driving positioning error) (Steering angle error) (Fork height error), compared with the preset threshold ( , , By comparing the results, the error level is determined (no error: ≤ threshold; slight error: threshold < error ≤ 2 times threshold; severe error: > 2 times threshold).

[0072] in, This refers to the fork height error, indicating the actual height. Altitude of the target The difference is calculated using the formula: .

[0073] Based on the error level, generate targeted correction instructions and update model parameters synchronously to avoid error accumulation, as detailed below: No error: Maintain the current control action without correction; Minor error: Generate motion correction instructions, the formula is: ; in, This represents the actual error. For the error threshold, For correction factor, This is the motion correction amount; Severe error: Generate an emergency correction instruction to stop the current action, re-collect environmental data, and regenerate the optimal action; at the same time, update the VLA visual encoder weights and the parameters of the hybrid reinforcement learning model (add an error penalty term to the PPO objective function and prioritize sampling error samples in the DQN experience replay pool) to avoid the recurrence of similar errors.

[0074] VLA visual encoder through The formula for identifying terrain type is: ; The MLP outputs three types of terrain probabilities (flat, uneven, and sloping), with the highest probability representing the current terrain. Difference in elevation between uneven terrain ， To adapt to uneven terrain; slope angle ( (for lidar detection range) It needs to be adapted to sloping terrain; Uneven terrain can be corrected by adjusting the weights of the reward function. , Reduce speed and improve stability; Sloping terrain can be modified by adjusting the weights of the reward function to correct the action. , Increase track traction (by adjusting track driving force using PID control, formula) , (For flat ground drive), fork angle correction To prevent materials from slipping; In extreme scenarios (dust, low temperature), by increasing the attention weight of LiDAR and depth features, the motion speed reduction is corrected by 20%, ensuring stable perception and execution.

[0075] S6. Model Iterative Optimization.

[0076] Offline training constructs a photorealistic digital environment based on CAD data, simulating various complex terrains and extreme scenarios. It inputs a large number of environmental and task samples to train the VLA model and the hybrid reinforcement learning model offline, iteratively optimizing the model parameters so that the model converges to the optimal policy, reducing errors in the online exploration phase.

[0077] Online fine-tuning involves storing the state data, action data, and reward values ​​of each closed-loop control operation in the experience playback pool during actual forklift operation. Samples are periodically extracted (every 100 closed-loop controls) to fine-tune the model online, update the model weights, and make the model adapt to subtle changes in the actual operation scenario, thereby further improving control accuracy and scenario adaptability.

[0078] Applying the above control methods to warehouse operations with complex terrain includes: The unmanned tracked forklift in this embodiment has a load capacity of 1.5 tons and is equipped with the following hardware: a 16-line LiDAR (detection range 5-10 meters, accuracy ±1 cm), a high-definition RGB camera (resolution 1920×1080), a depth camera (measurement range 0.5-5 meters), an IMU inertial measurement unit (measurement accuracy ±0.01°), a track speed sensor (measurement range 0-1 m / s), and a load sensor (measurement range 0-2 tons). The on-board controller uses NVIDIA Jetson AGX Xavier, which supports real-time processing of multimodal data. The operating environment is a warehouse, which includes flat passages, uneven ground (height difference 3-8 cm), and slopes (angle 8-12°). There are temporarily piled materials (obstacles), and some areas are subject to dust interference.

[0079] A VLA-hybrid reinforcement learning control framework was built. The VLA model adopted OpenVLA-7B, with an optimized visual encoder to adapt to multi-sensor data. The hybrid reinforcement learning model adopted the PPO+DQN architecture, with the following parameter settings: learning rate 0.001, discount factor γ=0.95, initial exploration rate ε 0.9, batch size 64, and experience replay pool capacity 100,000. The error thresholds were set as driving positioning error ±3mm and steering angle error ±2°. The terrain adaptation thresholds were 5cm for uneven terrain height difference and 15° for slope angle.

[0080] The LiDAR system collects 3D point cloud data of the warehouse environment, removes ground noise, and captures obstacles (temporary materials) and shelf positions. The RGB camera collects environmental images and identifies material pallets. The depth camera collects depth data to obtain the distance to obstacles (e.g., material is 2.5 meters away from the forklift). The IMU collects forklift attitude data (tilt angle 3°). The load sensor collects the initial load (0 tons, no material is being grabbed). After all data is preprocessed, it is input into the VLA vision encoder. The language encoder embeds the task instruction "Grab the material on shelf number 3 and move it to the designated area (coordinates X=10m, Y=5m)", and outputs the fused environment-task feature vector.

[0081] State space S = [Environment-Task Feature Vector, Tilt Angle 3°, Track Speed ​​0m / s, Load 0 tons, Error 0mm]; PPO generates initial action instructions: Track speed 0.6m / s, Turning angle 0° (straight ahead), Fork lifting height 0.5m; DQN evaluates the action's Q value as 7.8 (out of 10), filtering it as a candidate action; VLA policy decoder combines environmental features (flat aisle, no obstacles), corrects the action instructions (track speed maintained at 0.6m / s, fork lifting height adjusted to 0.6m, adapting to pallet height), outputs the optimal action instructions, and generates an explanation: "Currently a flat aisle, no obstacles, proceed straight to rack 3, forks adjusted to 0.6m height, ready to grab materials."

[0082] The on-board controller converts the action commands into electrical signals, controlling the track drive module to move straight at a speed of 0.6 m / s, and the fork lifting module to raise the forks to a height of 0.6 m. During the execution, the sensors provide real-time data feedback, and the forklift travels smoothly without any deviation.

[0083] When the forklift travels over uneven ground (6cm height difference), the VLA visual encoder identifies the terrain change, and the IMU's tilt angle changes to 6°; the terrain adaptation module generates adaptation instructions, adjusting the reward function. and The weights of the hybrid reinforcement learning model are used to correct the action commands: the track speed is reduced to 0.4m / s, the steering angle is maintained at 0°, and the fork height remains unchanged; the error correction module detects that the driving positioning error is 2mm (not exceeding the threshold), and no additional correction is required; the forklift passes smoothly over uneven ground without slipping or tilting.

[0084] During the operation, the data (state, action, reward) of each closed-loop control is stored in the experience replay pool. Every 100 closed-loop controls, 500 samples are extracted for online fine-tuning to update the model weights. In the offline training phase, 1,000 complex terrain scenarios are simulated in the simulation environment, and the model is iterated 10,000 times to converge to the optimal policy.

[0085] Using this method, the unmanned tracked forklift successfully completed material grabbing and handling tasks. During operation, the success rate of adapting to complex terrain (uneven ground, slope) was 100%, and there were no perception failures in dusty environments. The driving positioning error was ≤2.5mm, the steering angle error was ≤1.8°, and there was no error accumulation. The decision-making process can be explained by natural language, which is convenient for operation and maintenance. The operation efficiency is 35% higher than that of traditional PID control and 20% higher than that of pure reinforcement learning control. The sample efficiency is improved by 40%, and there is no equipment loss or material damage in the early exploration stage.

[0086] Therefore, this invention employs a VLA-hybrid reinforcement learning-based unmanned tracked forklift control method. It enhances the sample efficiency of reinforcement learning through the multimodal representations provided by VLA, overcomes the limitations of VLA's imitation learning through hybrid reinforcement learning, and achieves policy optimization. Simultaneously, an error correction mechanism is added to avoid error accumulation and adapt to complex and extreme scenarios. Specifically, the VLA model is responsible for transforming multimodal environmental data and task instructions into interpretable semantic-visual representations. The hybrid reinforcement learning model generates precise control actions based on these representations. The error correction module corrects action deviations in real time, and the terrain adaptation module optimizes action outputs based on the transmission characteristics of the tracked forklift, forming a complete closed loop of "perception-decision-execution-correction-optimization".

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A control method for an unmanned tracked forklift based on VLA-hybrid reinforcement learning, characterized in that, include: S1. Build the VLA-hybrid reinforcement learning control framework, initialize the parameters, and the control framework relies on the multimodal perception module, hybrid reinforcement learning decision module, precise execution module, and error correction module to work together. S2. Each sensor collects data at a preset frequency. After being processed by a targeted preprocessing algorithm, the VLA visual encoder extracts features from the multimodal data to generate a visual token. The language encoder converts the preset task instructions into semantic representations, fuses them with the visual tokens, and outputs the fused environment-task feature vector. S3. Generate the state space of the hybrid reinforcement learning model, design a multi-dimensional dense reward function, input the output fusion feature vector into the hybrid reinforcement learning model, filter out candidate actions, the VLA policy decoder combines environment-task semantics to correct the candidate actions, finally output the optimal control action instruction, and convert the optimal action instruction into a natural language explanation. S4. The on-board controller converts the optimal control action command into an electrical signal and transmits it to each execution module via the CAN bus. It uses track differential drive PID control and fork height PID control to ensure accurate execution of the action command. S5. The error correction module receives the feedback status data in real time, calculates the error and compares it with the preset threshold, determines the error level and generates targeted correction instructions, and updates the model parameters synchronously. The VLA visual encoder identifies the terrain type and adjusts the action and reward function weights according to different terrains. S6. Model iterative optimization, including offline training and online fine-tuning. Offline training is based on CAD data to build a photorealistic digital environment and input a large number of samples to train the VLA model and the hybrid reinforcement learning model. Online fine-tuning is to periodically extract samples from the experience replay pool to fine-tune the model during the actual operation of the forklift.

2. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The state space of the hybrid reinforcement learning model includes the output environment-task feature vector, forklift posture data, track speed data, load data, and current error data.

3. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The VLA visual encoder extracts features from multimodal data in the following specific way: The PointNet++ algorithm is used for point cloud feature extraction from LiDAR. Layered feature extraction is performed on 3D point cloud data for forklift operation scenarios. The formula is as follows: ； in, Given the spatial coordinates of a single point cloud, the MLP is a 2-layer fully connected network. Max-pooling uses a 5×5 window to extract global features from the local point cloud, and the final output is... It accurately represents terrain undulations and the spatial location of obstacles; RGB image feature extraction uses a ResNet-50 network, removing the last fully connected layer and retaining the convolutional and pooling layers. The convolutional layer parameters are optimized for forklift operation scenarios to extract visual features from the image. The formula is as follows: ； in, The image is a preprocessed RGB image; ResNet50 is connected via residual connections. , For residual mapping; Deep feature extraction uses a CNN network. For forklift operation scenarios, the input depth image is used to extract depth features using the following formula: ； in, This is the preprocessed depth image. It is a depth feature vector; Multimodal feature fusion employs a Self-Attention mechanism, fusing the three features mentioned above, with the following formula: ； in, , , , , , All are learnable weight matrices. For feature dimensions.

4. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The formula for the language encoder is: ； in, This refers to the task instruction.

5. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The formula for the policy decoder is: ； in, It provides context-task fusion features; the Transformer decoder contains 12 attention heads and a Feed-Forward network.

6. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The hybrid reinforcement learning model adopts a hybrid architecture of "PPO policy gradient and DQN value evaluation", including: The PPO-Clip algorithm is used to constrain the policy update magnitude and avoid drastic fluctuations in forklift movements. The objective function is: ； in, , For the current strategy parameters In state Next generation action The probability, The strategy before the update; The dominant function; The Double DQN algorithm is used to avoid overestimation and accurately assess the safety and effectiveness of actions. The core update formula is: ； in, To adapt the parameters to the model's convergence speed, These are the current Q-network parameters. The target Q network parameters; The optimal action for the next state selected by the current Q network; PPO generates initial actions, DQN evaluates their value, VLA outputs the initial actions, and weighted fusion yields candidate actions, using the following formula: ； in, This is the initial action vector for PPO. The initial action vector for VLA, For PPO action weighting coefficients, For VLA action weighting coefficients.

7. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, The formula for the multidimensional dense reward function is: ； in, Rewards for completing the task. For accuracy rewards, As a safety reward, Rewards for efficiency This is an error penalty.

8. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, In error correction and terrain adaptive adjustment, the error correction module determines the error level and generates correction commands in the following way: calculate Driving positioning error, Steering angle error The fork height error is compared with a preset threshold to determine the error level: no error, slight error, or severe error. The expression for slight error correction is: ; in, This represents the actual error. For the error threshold, For correction factor, This is the motion correction amount.

9. The unmanned tracked forklift control method based on VLA-hybrid reinforcement learning according to claim 1, characterized in that, In the model iterative optimization, offline training constructs a photorealistic digital environment based on CAD data to simulate various complex terrains and extreme scenarios. A large number of environmental and task samples are input to train the VLA model and the hybrid reinforcement learning model offline, iteratively optimizing the model parameters so that the model converges to the optimal policy. Online fine-tuning is performed during the actual operation of the forklift. The state data, action data, and reward values ​​of each closed-loop control are stored in the experience replay pool. Samples are periodically extracted to fine-tune the model online and update the model weights.

Images