A model car speed control system and method based on reinforcement learning

Abstract

The application relates to the technical field of model vehicle speed hierarchical feedback and double-time-lag adaptive control based on reinforcement learning, and discloses a model vehicle speed control system and method based on reinforcement learning. A friction perception is arranged below a vehicle, tangential force and normal force are obtained by sampling, a normal force benchmark and an abnormal protection are established, friction is calculated and a road surface is coded; the maximum safe speed is taken as a dry condition, and a target is set according to a threshold proportion and clipped friction in a wet and slippery condition. Invalid actions are eliminated, and valid speed measurement is obtained, response steps are confirmed according to significant speed difference and direction consistency, and hierarchical statistical average time lags are obtained; the current speed measurement, the lag speed and the target speed are normalized, and the state is formed by combining the time lag steps and the road surface coding. A discrete speed increment is constructed, and the next period instruction is limited in amplitude and output; the target deviation and the adjacent action difference are used to form a reward; the state is quantized, a hierarchical value table is established, and the value table is updated by using a decreasing learning rate and a fixed discount; and the optimal action with a small increment or zero increment is selected in each period.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning-based hierarchical feedback and dual-time-delay adaptive control technology for model vehicle speed, specifically to a reinforcement learning-based model vehicle speed control system and method. Background Technology

[0002] With the rapid development of intelligent vehicles, autonomous driving model vehicles, and related intelligent transportation systems, intelligent vehicle speed control has become one of the key technologies for ensuring driving safety and system efficiency. In complex traffic scenarios, model vehicles often face variable road conditions, such as frequent switching between dry and wet surfaces, and large fluctuations in the ground friction coefficient, which places higher demands on vehicle speed regulation and driving stability. Traditional vehicle speed control methods mainly rely on model-based PID control, fuzzy control, or classical state-space regulation strategies. These methods mostly assume that the road surface condition can be perceived in advance, that time delay effects are negligible, or that simplified compensation is achieved through fixed parameters. In practical applications, because the friction coefficient between the model vehicle and the real road surface is uncontrollable and changes constantly, existing technologies often struggle to perceive the ground friction condition in real time and accurately, cannot dynamically adjust the speed target under typical scenarios such as dry and wet conditions, and lack the ability to hierarchically model and identify action-response lag effects.

[0003] Furthermore, conventional control algorithms are generally designed based on the assumptions of "no delay" or "constant delay," making it difficult to adapt to time-varying delays caused by mechanical, signal processing, and control execution factors. While some adaptive and intelligent control methods introduce hysteresis compensation based on historical data, they often employ global averaging or fixed windows, failing to dynamically identify and compensate for hysteresis in different road friction states (e.g., dry / wet). More significantly, most control methods based on existing models or frameworks fail to fully integrate the microscopic feedback of real-time sensor data on the vehicle's physical response, and are unable to provide robust handling and safety assurance for complex operating conditions (e.g., sudden switching between wet and slippery conditions, extreme data disturbances, boundary saturation, etc.). In recent years, some research has attempted to introduce intelligent algorithms such as reinforcement learning into the field of vehicle control, achieving adaptive decision-making through online learning. However, existing related methods are mostly reinforcement learning strategies based on single-layer value tables and traditional state spaces, failing to innovate control parameters and state representations by incorporating vehicle physical constraints, time delay diversity, and layered road condition characteristics. Furthermore, existing methods lack fine-grained detection and recognition mechanisms for the dynamic lag relationship between actions and responses, making it difficult to achieve multi-level, real-time lag identification and compensation under different road conditions, and also difficult to avoid system response anomalies caused by extreme scenarios such as boundary saturation.

[0004] Therefore, this case aims to propose a model vehicle speed control system and method based on reinforcement learning. First, it directly senses the interaction between the tire and the ground at the lowest level, constructing robust friction state recognition through tangential and normal forces, and setting safety protections under abnormal and extreme data conditions. Then, it dynamically maps the target speed according to the road surface condition, treating typical working conditions of wet and dry conditions in a stratified manner. Fine-grained detection is performed on the time-varying delay between actions and the vehicle's actual response, establishing a lag sample set and average time delay divided according to road conditions, thus introducing mechanically reasonable lag speeds during state construction. At the decision-making level, it employs discrete actions protected by physical boundaries and deterministic amplitude-limited outputs, combined with real-time target tracking and action smoothing reward shaping, using a stratified value table for online learning and updating. Summary of the Invention

[0005] This invention provides a model vehicle speed control system and method based on reinforcement learning, which helps to solve the problems mentioned in the background art.

[0006] This invention provides the following technical solution: a model vehicle speed control method based on reinforcement learning, comprising:

[0007] Friction sensors are installed under the front of the vehicle to collect tangential and normal forces at sampling intervals, establish normal force benchmarks and anomaly protection, calculate the friction coefficient, and generate discrete road surface state codes based on friction thresholds.

[0008] In dry conditions, the maximum safe speed is used; in wet conditions, the target speed for the current period is generated by scaling the ratio of the trimmed friction coefficient to the friction threshold.

[0009] The measured speed is processed to be effective and invalid actions are determined. Within the future window, a confirmation step is searched based on the significant speed difference threshold and direction consistency. The average time delay is statistically calculated in the dry layer and the wet and slippery layer respectively.

[0010] The measured speed, lag speed and target speed are normalized to the maximum safe speed and combined with the time delay step number and road surface state code to form a continuous state vector, and the corresponding vector for the next cycle is generated at the same time.

[0011] Construct a set of velocity increments with the step distance as the interval, limit the amplitude within the upper and lower bounds of the physical velocity, and form the velocity command for the next cycle.

[0012] The target speed deviation and the difference between adjacent actions constitute the scalar reward for the current period.

[0013] The continuous components of the state are quantized with a fixed number of quantization layers. Discrete value tables for the dry and slippery layers are established and incrementally updated with a learning rate that decreases with the period and a given discount factor.

[0014] At the beginning of each cycle, based on the current discrete state and road surface state, the speed increment with the highest value is selected from the corresponding layered value table. If there is a tie, the minimum absolute value is prioritized. If there is still a tie, the zero increment is selected. The speed command is output and the reward calculation and value table update are completed in the next cycle.

[0015] Optionally, the step of setting a friction sensor below the front end of the vehicle, collecting tangential and normal forces at sampling intervals, establishing a normal force benchmark and anomaly protection, calculating the friction coefficient, and generating a discrete road surface state code based on the friction threshold specifically includes:

[0016] After power-on, establish a normal force reference window and minimum boundary: take a statistical period of one second as the reference, take ten samples to form a set, calculate the average value of the normal force within the window, and set the minimum normal force boundary as a certain proportion of this average value; the sampling interval is 0.1 seconds, and the control period index is counted according to a discrete sequence;

[0017] In each control cycle, a safety denominator is constructed: the larger value between the current cycle normal force and the minimum normal force boundary is used as the denominator, and the ratio of the tangential force to this safety denominator is used as the instantaneous friction coefficient; when the average normal force is equal to zero, protection is triggered, the current friction coefficient is set to zero, the road surface condition code is set to slippery, and the speed setpoint for the next cycle is set to zero, until a non-zero average value is re-established;

[0018] Perform non-negative clipping and hierarchical coding: Perform non-negative clipping on the friction coefficient; when the friction coefficient before clipping is not lower than the friction threshold of 0.60, it is coded as dry; when it is lower than the threshold, it is coded as wet and slippery; write the code into the structured state data.

[0019] Optionally, the step of using the maximum safe speed in the dry state and scaling the target speed in the wet state according to the ratio of the trimmed friction coefficient to the friction threshold specifically includes:

[0020] In dry conditions, the maximum safe speed is set to be equal to the physical speed limit of 3.0 m / s;

[0021] The maximum safe speed of the dry layer is scaled by the ratio of the friction coefficient after non-negative cutting to 0.60 under wet and slippery conditions to generate the target speed for the current period.

[0022] The target speed for road surface condition switching is determined based on the current cycle state and cached for calculation in the next cycle.

[0023] Optionally, the step of validating the measured speed and determining invalid actions, searching for confirmation steps within a future window based on a significant speed difference threshold and directional consistency, and statistically averaging the time delay in both the dry and slippery layers, specifically includes:

[0024] The measured speed is limited by an interval to obtain the effective speed;

[0025] When the effective speed is at the physical upper limit and the action is acceleration, or at the physical lower limit and the action is deceleration, it is judged as an invalid action and the delay steps are recorded as the maximum delay detection steps;

[0026] When the speed increment is not zero and no invalid action is triggered, the confirmation step that first meets the three conditions of significant speed difference threshold, direction consistency and previous step not reaching the threshold is searched step by step in the subsequent window.

[0027] The time delay of each action is assigned to the dry set and the wet set respectively, and the average time delay is calculated in each set; when the set is empty, the average time delay is set to a sampling interval.

[0028] Optionally, the step of normalizing the measured speed, lag speed, and target speed according to the maximum safe speed, and combining them with the time delay step number and road surface state code to form a continuous state vector, while simultaneously generating the corresponding vector for the next cycle, specifically includes:

[0029] The average time delays of the dry layer and the wet slip layer are respectively converted into hysteresis steps;

[0030] In the current cycle, if the number of historical steps is not less than the number of lag steps in the corresponding layer, the effective speed of the corresponding historical cycle is read as the lag speed; otherwise, the effective speed of the first cycle is used as the lag speed.

[0031] The measured speed, the lag speed and the target speed are normalized according to the maximum safe speed, and the average time delay is represented by the number of steps. Combined with the road surface state code, they form the current period continuous state vector.

[0032] The number of lag steps for the next cycle is determined based on the road surface condition for the next cycle. The lag velocity and target velocity for the next cycle are generated to form the continuous state vector for the next cycle.

[0033] Optionally, the constructed set of velocity increments at intervals of distance walking distance is limited within the upper and lower bounds of the physical velocity to form the velocity command for the next cycle, specifically including:

[0034] Establish a discrete set of velocity increments, and set the discrete step size and value range;

[0035] After selecting the current speed increment, add it to the current effective speed, and limit the speed according to the upper and lower limits of the physical speed to obtain the speed command for the next cycle;

[0036] Write the speed command for the next cycle into the controller.

[0037] Optionally, the step of constructing the current period's scalar reward from the target speed deviation term and the difference term between adjacent actions specifically includes:

[0038] At the end of each control cycle, calculate the squared term of the target speed deviation;

[0039] Calculate the square of the difference between the velocity increment of the current cycle and the previous cycle, with the increment of the previous cycle considered as zero for the first cycle;

[0040] The sum of the two aforementioned items is taken as the negative value and cached in the value table update process.

[0041] Optionally, the step of quantizing the continuous state components with a fixed number of quantization layers, establishing discrete value tables for the dry and slippery layers, and incrementally updating them using a learning rate that decreases with the period and a given discount factor, specifically includes:

[0042] The first three normalized components of the state vector are binned and quantized according to a fixed number of quantization layers to obtain integer indices;

[0043] The time delay component is rounded down to an integer range from one to the maximum delay detection step and then clamped to obtain the time delay index;

[0044] Use the road surface condition code as a binary index;

[0045] Using a five-element index and candidate speed increments as keys, corresponding entries are created in the dry table and the wet / slippery table, respectively.

[0046] Using a learning rate that decreases with each cycle and a given discount factor, the maximum value of the entries in the next state across all candidate actions is used as the estimation target, and incremental updates are performed on the entries of the current key.

[0047] Optionally, at the beginning of each cycle, based on the current discrete state and road surface state, the step of selecting the speed increment with the highest value from the corresponding hierarchical value table, prioritizing the smallest absolute value when there are ties, and taking the zero increment when still tied, outputting a speed command, and completing the reward calculation and value table update in the next cycle, specifically includes:

[0048] At the beginning of each control cycle, read the current discrete state index and road surface state;

[0049] Select the speed increment with the highest value from the value table of the corresponding layer;

[0050] For tandem maximum values, select the speed increment with the smallest absolute value; if they are still tandem, select zero increment.

[0051] Call the amplitude limiting operation to generate the speed command for the next cycle and output it;

[0052] In the next sampling period, the measured velocity is obtained, the reward calculation is completed, and the hierarchical value table is updated.

[0053] A system for implementing the reinforcement learning-based model vehicle speed control method includes:

[0054] The friction sensing module is installed under the front of the vehicle body and collects tangential and normal forces at sampling intervals.

[0055] The speed measurement module outputs the linear velocity of the wheels.

[0056] The vehicle control unit consists of a processor and a memory. It stores discrete value tables for the dry and wet skid layers and performs status coding, target speed calculation, action selection, value table updating, and speed command generation.

[0057] The communication and execution interface sends speed commands to the driver and receives speed measurement data and friction sensing data, and then physically clamps the commands.

[0058] The power supply and timing unit provides stable power to each module and provides periodic timing trigger signals.

[0059] The present invention has the following beneficial effects:

[0060] 1. A high-frequency friction sensing module is installed at the lower front end of the model vehicle. Through real-time acquisition and benchmark statistics of tangential and normal forces, it achieves sensitive capture and anomaly protection for minute fluctuations in normal force. Traditional vehicle control often relies on experience or low-frequency signals to judge road conditions, making it difficult to detect changes in friction between the tires and the road surface in a timely manner. This solution, however, combines multi-cycle statistics in the first second with minimum boundary settings. This avoids calculation errors caused by division by zero or non-physical amplification, and directly maps the continuous friction coefficient to two states, "dry" and "slippery," through simple threshold comparison. This discrete state encoding not only facilitates the design of the state dimension of the subsequent reinforcement learning module, but also effectively compresses the continuous state space, reduces computational complexity, and improves online update speed and stability. At the same time, through protection logic in the case of zero normal force, this solution ensures the safe shutdown of the system under extreme failure conditions, providing a reliable fault isolation and protection mechanism for the testing and application of the model vehicle.

[0061] 2. To address the control requirements under different friction conditions, this solution proposes a target speed setting strategy of "fixed upper limit in dry conditions and scaling proportionally to friction in wet conditions." Unlike traditional one-size-fits-all or linear empirical maximum speed settings, the target speed in wet conditions is strictly scaled proportionally based on the real-time measured friction coefficient and a preset threshold. This maintains strict constraints on wet-slip risks while maximizing the usable safe speed under current friction conditions. This adaptive speed limiting strategy based on real-time perception ensures that the model vehicle always operates at the optimal balance between safety and efficiency under different road conditions. Compared to fixed safe speeds or simple mode switching, it has higher speed utilization and smaller energy consumption fluctuations. Furthermore, this method relies only on simple trimming and scaling calculations, without the need for complex models or large amounts of historical data, lowering the implementation threshold and facilitating the deployment of embedded real-time controllers.

[0062] 3. An online detection and hierarchical modeling of action execution lag is introduced. By searching for significant differences in speed increment actions within a finite window, the physical delay between each control output and the actual speed response is accurately quantified. This process not only automatically identifies invalid actions (such as saturation states at speed limits), avoiding accidental inclusion in statistics, but also collects delay samples in both dry and wet / slippery environments hierarchically, calculating the average lag steps separately. Traditional methods often treat lag as constant or ignore its variation. This scheme, through sample-driven dynamic estimation, can reflect the impact of road conditions, actuator temperature, or battery voltage changes on response speed in real time, providing more accurate delay compensation information for subsequent state vector construction. This adaptive lag modeling improves the matching degree of control command execution, reduces the risk of overshoot or oscillation caused by lag mismatch, thereby improving vehicle ride smoothness and safety.

[0063] 4. Regarding state representation, this scheme normalizes the measured speed, delayed speed, and target speed according to the maximum safe speed, and combines the time delay steps and road surface state to form a multi-dimensional continuous state vector, providing a concise yet comprehensive environmental representation for reinforcement learning. Traditional reinforcement learning often uses raw sensor values ​​or features extracted from experience, while the completion and normalization of speed history and the synchronous generation of future state vectors here enable the state encoding to have both historical backtracking and feedforward prediction capabilities, enhancing the agent's ability to perceive dynamic changes in the environment. Simultaneously, this scheme quantizes the average time delay into discrete step counts using a "half-round up" method, naturally matching the discrete state space, simplifying the table update and indexing process, and ensuring real-time performance and stable convergence of the algorithm during online operation.

[0064] 5. To balance motion accuracy and computational complexity, this scheme establishes a discrete set of velocity increments and clamps the motion results according to a uniform step size and physical limits to form the velocity command for the next cycle. Compared to a continuous motion space, a discrete motion space is easier to store and update the value function in tabular reinforcement learning; and compared to a single maximum increment or fixed gear, a fine-grained set of increments balances control smoothness and adjustment flexibility, helping to achieve a smooth transition between fine-tuning and large-amplitude adjustments. Furthermore, the limited velocity command is directly written into the controller during motion execution, eliminating the need for additional filtering or secondary calculations, thus reducing system latency and software complexity. This design satisfies both the need for precise control over the dynamic speed response of the model vehicle and is feasible for engineering implementation.

[0065] 6. The reward function proposed in this scheme consists of a dual penalty: a target speed deviation term and an adjacent action difference term. This encourages the agent to control the speed closer to the desired value to improve tracking accuracy, while suppressing excessively drastic speed changes to maintain driving smoothness. Traditional reinforcement learning often focuses only on tracking error or reward, which can easily lead to drastic action fluctuations or delayed responses. However, by adding an action smoothness penalty, the agent can achieve high-precision tracking while actively avoiding unnecessary control jitter, thereby reducing energy consumption and mechanical wear. More importantly, this composite reward function only relies on the measured and set values ​​at the end of the control cycle, requiring no additional models or historical data support. It is simple, easy to implement, and has good interpretability, facilitating maintenance and parameter tuning.

[0066] 7. For both "dry" and "slippery" road surfaces, the scheme constructs discrete value tables separately, and uses multi-layer quantization and discrete indexing of state components to map the five-dimensional continuous state to a storable table space. Unlike general deep Q-learning or linear function approximation, discrete tables not only converge faster in the small-scale state space of the model vehicle, but also do not require high computing power. The designed quantization function and indexing rules ensure the continuity and noise resistance of the state mapping, while the hierarchical table partitioning further reduces mutual interference under different road surface conditions. The update rule uses a periodically decreasing learning rate and a fixed discount factor, which not only ensures the asymptotic stability of online learning, but also dynamically balances exploration and utilization according to the training progress, ultimately achieving efficient convergence of the value table.

[0067] 8. At the beginning of each control cycle, this scheme directly queries the corresponding value table based on the current discrete state and pavement layer code, employing a greedy strategy to select the velocity increment with the highest value. To handle decision ambiguity caused by coinciding maximum values, a hierarchical rule of "prioritizing the smallest absolute value, then taking the zero increment" is designed, balancing control smoothness and decision stability. Compared with ε-greedy and other exploration strategies, this pure greedy output is more deterministic and consistent during online testing; simultaneously, the "zero increment backoff" mechanism provides natural suppression in coinciding situations, avoiding unnecessary velocity oscillations. The command output uses a predefined physical limiting function, ensuring that the final issued velocity command always falls within a safe and executable range, greatly enhancing the system's robustness and safety. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example, refer to Figure 1 A speed control method for a model vehicle based on reinforcement learning, comprising:

[0071] Friction sensors are installed under the front of the vehicle to collect tangential and normal forces at sampling intervals, establish normal force benchmarks and anomaly protection, calculate the friction coefficient, and generate discrete road surface state codes based on friction thresholds.

[0072] In dry conditions, the maximum safe speed is used; in wet conditions, the target speed for the current period is generated by scaling the ratio of the trimmed friction coefficient to the friction threshold.

[0073] The measured speed is processed to be effective and invalid actions are determined. Within the future window, a confirmation step is searched based on the significant speed difference threshold and direction consistency. The average time delay is statistically calculated in the dry layer and the wet and slippery layer respectively.

[0074] The measured speed, lag speed and target speed are normalized to the maximum safe speed and combined with the time delay step number and road surface state code to form a continuous state vector, and the corresponding vector for the next cycle is generated at the same time.

[0075] Construct a set of velocity increments with the step distance as the interval, limit the amplitude within the upper and lower bounds of the physical velocity, and form the velocity command for the next cycle.

[0076] The target speed deviation and the difference between adjacent actions constitute the scalar reward for the current period.

[0077] The continuous components of the state are quantized with a fixed number of quantization layers. Discrete value tables for the dry and slippery layers are established and incrementally updated with a learning rate that decreases with the period and a given discount factor.

[0078] At the beginning of each cycle, based on the current discrete state and road surface state, the speed increment with the highest value is selected from the corresponding layered value table. If there is a tie, the minimum absolute value is prioritized. If there is still a tie, the zero increment is selected. The speed command is output and the reward calculation and value table update are completed in the next cycle.

[0079] First, by setting a friction sensing unit below the front end and continuously acquiring tangential and normal forces within a fixed sampling interval, a real-time friction coefficient calculation process and anomaly protection mechanism were established, effectively avoiding calculation anomalies and safety hazards caused by zero normal force. Next, after classifying friction states into dry and wet / slippery categories, target speeds were set using fixed upper limits and proportional scaling methods respectively. This ensured the highest usable speed on dry surfaces while intelligently adjusting to a safe range on wet / slippery surfaces, thus resolving the contradiction between safety and efficiency that traditional control strategies cannot balance. In the action response delay detection and modeling step, the physical delay between each speed adjustment and the actual response was accurately quantified through significant speed difference search within a future window and direction consistency determination. The average time delay under both road conditions was statistically analyzed, providing reliable time delay compensation data for subsequent state construction. Subsequently, by normalizing the currently measured speed, the completed lag speed, and the target speed, and combining the time delay steps with road state encoding, a continuous state vector that considers both historical information and feedforward prediction was constructed, allowing the agent to obtain richer and more stable inputs in environmental representation. Discretization and amplitude limiting of the velocity-motion space ensure sufficient adjustment accuracy while maintaining smoothness. The dual reward structure based on target velocity deviation and motion difference balances tracking accuracy and operational stability. Finally, the hierarchical value table is updated separately in dry and wet / slippery environments, and online rapid convergence is achieved using a time-decreasing learning rate and a fixed discount factor. The entire process directly selects the optimal velocity increment at the start of each cycle using the current discrete state, and suppresses unnecessary jitter by prioritizing small or zero increments, thus achieving real-time, adaptive, and safe velocity output. Unlike existing methods based on classical control or single machine learning models, this method does not rely on a precise environment model and can continuously optimize online during operation. It solves the problems of variable road surfaces, uncertain actuator time delays, and the difficulty of simultaneously achieving motion smoothness and tracking accuracy, improving the driving safety and control efficiency of the model vehicle.

[0080] The process of installing a friction sensor below the front of the vehicle, collecting tangential and normal forces at sampling intervals, establishing a normal force benchmark and anomaly protection, calculating the friction coefficient, and generating discrete road surface state codes based on friction thresholds specifically includes:

[0081] After power-on, establish a normal force reference window and minimum boundary: take a statistical period of one second as the reference, take ten samples to form a set, calculate the average value of the normal force within the window, and set the minimum normal force boundary as a certain proportion of this average value; the sampling interval is 0.1 seconds, and the control period index is counted according to a discrete sequence;

[0082] In each control cycle, a safety denominator is constructed: the larger value between the current cycle normal force and the minimum normal force boundary is used as the denominator, and the ratio of the tangential force to this safety denominator is used as the instantaneous friction coefficient; when the average normal force is equal to zero, protection is triggered, the current friction coefficient is set to zero, the road surface condition code is set to slippery, and the speed setpoint for the next cycle is set to zero, until a non-zero average value is re-established;

[0083] Perform non-negative clipping and hierarchical coding: Perform non-negative clipping on the friction coefficient; when the friction coefficient before clipping is not lower than the friction threshold of 0.60, it is coded as dry; when it is lower than the threshold, it is coded as wet and slippery; write the code into the structured state data.

[0084] Further specific implementation steps include:

[0085] A friction sensing module is installed under the front of the model car. The module detects the tangential force on the rigid micro-slip segments of the wheels. With normal force Calculate the instantaneous friction coefficient at the current moment, specifically as follows:

[0086] To avoid Divide-by-zero or non-physical amplification caused by proximity to zero, first after power-on. Each sample constitutes a set Evaluation of normal force benchmark:

[0087] , ;in, Index of the discrete control cycle; For the first Tangential force during periodic sampling; For the first Normal force during periodic sampling; This is the number of samples used for baseline statistics 1 second after power-on; A second is the physical time interval between two consecutive discrete periods; It is a set of discrete periodic indices containing the first second; This refers to the discrete sampling index within the baseline window; The average normal force; This is the boundary with minimum normal force.

[0088] Then construct a safe denominator function. And calculate: ;in, , Formal parameter; For the first Periodic instantaneous friction coefficient;

[0089] like Then immediately enter protection mode: Order , , And enter the next cycle, setting the speed setpoint to zero. Until it is restarted with a new get Only then can normal calculations be resumed; among them, It is a non-negative shear friction coefficient; For the first Periodic pavement layer condition coding, with 0 representing dry and 1 representing wet and slippery; Set the speed for the next cycle;

[0090] Perform conditional judgment on the above friction coefficient and complete the state coding:

[0091] like Assume the current road surface is dry, and let the status code be... ;

[0092] like Assume the current road surface is slippery, and let the state code be... ;

[0093] Encode the status Store it in a structured state vector.

[0094] First, the friction sensing module installed at the lower front end acquires detailed friction characteristic data by detecting the tangential and normal forces generated during the slight slippage between the wheel and the road surface. This fundamentally overcomes the limitations of relying solely on wheel speed or inertial sensing, resulting in more accurate road condition perception. Second, upon power-up, the minimum normal force boundary is clearly defined through continuous multi-cycle average statistics of normal force during the initial period. This provides a robust denominator benchmark for subsequent friction coefficient calculations and avoids numerical instability or division-to-zero errors caused by sudden changes in normal force. This step, by specifying a fixed time length and number of samplings, ensures that a reliable baseline can be quickly acquired even when the environment changes abruptly, thereby improving the fault tolerance of the entire sensing module. The third step involves constructing a safety denominator function that keeps the denominator within a reasonable range, further ensuring the continuity and stability of friction coefficient calculation. Simultaneously, when the average normal force is zero, a protection mode is immediately entered, setting both the friction coefficient and speed setpoint for the current cycle to zero and marking it as a slippery state. Normal operation only resumes after a non-zero normal force is regained. This mechanism effectively prevents the risk of control failure in extreme road conditions or sensor malfunctions. Finally, by mapping the continuous friction coefficient to either dry or slippery codes based on preset thresholds and storing this data in structured state data, continuous and clear input of the environmental state to subsequent reinforcement learning modules is achieved. These four steps, operating in tandem, not only solve the inaccurate judgment problems caused by high signal noise or computational instability in traditional friction sensing but also provide the agent with high-quality discrete environmental information, providing a reliable basis for subsequent target speed setting and control decisions, thereby improving the overall reliability and safety redundancy of the control system.

[0095] The process of using the maximum safe speed in a dry state and scaling the calculated target speed in a wet state based on the ratio of the trimmed friction coefficient to the friction threshold specifically includes:

[0096] In dry conditions, the maximum safe speed is set to be equal to the physical speed limit of 3.0 m / s;

[0097] The maximum safe speed of the dry layer is scaled by the ratio of the friction coefficient after non-negative cutting to 0.60 under wet and slippery conditions to generate the target speed for the current period.

[0098] The target speed for road surface condition switching is determined based on the current cycle state and cached for calculation in the next cycle.

[0099] Further specific implementation steps include:

[0100] Set the maximum safe speed on dry roads to the physical limit: ;in, This is the maximum safe speed for the drying layer; This is the upper limit of physical speed;

[0101] For slippery conditions Based on the friction coefficient after nonnegative cutting Calculate the corresponding target velocity: ;in, For the slip layer in the cycle The target safe speed;

[0102] Construct the overall target velocity function: ;in, For the first The target speed of the cycle.

[0103] An adaptive speed limit strategy for different road conditions is proposed for the target speed setting process. This strategy effectively solves the safety or efficiency trade-offs caused by traditional fixed speed limits or experience-based switching through a three-step process. The first step, in dry conditions, directly uses the predetermined maximum safe speed as the target speed limit, fully utilizing the maximum load-bearing capacity of the road surface. This approach better leverages the vehicle's performance potential than a simple conservative speed limit, while maintaining simple and intuitive control logic. The second step, in wet conditions, firstly, performs non-negative clipping on the friction coefficient to prevent negative spikes caused by sensor noise or algorithm jitter. Then, it dynamically maps the clipped friction coefficient to a preset threshold to calculate a safe speed that matches the dry speed limit. This dynamic scaling based on friction changes is more real-time and accurate than relying solely on experience-based wet speed selection. It can promptly increase the speed when road conditions improve and quickly tighten the speed limit when conditions worsen, achieving an intelligent balance between safety and efficiency. The third step, by unifying the use of these two types of speeds in discrete states, forms a branch selection logic for the overall target speed, making the speed setting process simple and easy to implement. Overall, this step enables safe and efficient speed planning based solely on real-time friction sensing without relying on external maps or complex environment models. It fills the gap between single fixed speed limits and complex conditional reasoning in existing technologies, and provides a beneficial innovation for the lightweight implementation of embedded controllers.

[0104] The process of validating the measured speed and determining invalid actions, searching for confirmation steps within a future window based on a significant speed difference threshold and directional consistency, and statistically averaging the time delay in both the dry and slippery layers specifically includes:

[0105] The measured speed is limited by an interval to obtain the effective speed;

[0106] When the effective speed is at the physical upper limit and the action is acceleration, or at the physical lower limit and the action is deceleration, it is judged as an invalid action and the delay steps are recorded as the maximum delay detection steps;

[0107] When the speed increment is not zero and no invalid action is triggered, the confirmation step that first meets the three conditions of significant speed difference threshold, direction consistency and previous step not reaching the threshold is searched step by step in the subsequent window.

[0108] The time delay of each action is assigned to the dry set and the wet set respectively, and the average time delay is calculated in each set; when the set is empty, the average time delay is set to a sampling interval.

[0109] Further specific implementation steps include:

[0110] Construct a speed measurement validation function: ;in, For input speed Perform range limiting to ;

[0111] And order: , ;in, For the first The speed is obtained by maximizing the period;

[0112] The validity criterion for an action is as follows:

[0113] or If the action is invalid, it will be recorded as no action detected, and the action will be directly set to null. And these samples will not be included in any lagged sample set during subsequent sample aggregation; among them, No delayed detection steps; This represents the maximum number of delayed detection steps.

[0114] in, This is a constant representing the lower limit of speed. For the first Cyclic speed increment action; For symbolic functions, Time return , Time return , Time return , As the independent variable;

[0115] when If no invalid action is triggered, the smallest integer that satisfies the triple conditions S301 to S303 is searched step by step within the future window. :

[0116] S301, ;in, For the future The measured speed of the step; This is a significant velocity difference threshold constant;

[0117] S302, ;

[0118] S303, ;in, For intermediate counting index;

[0119] If there exist conditions that satisfy S301 to S303. Then in the first The sampling period is completed and confirmed and recorded. If not found, then in the... The sampling period is denoted as no significant response, and set to ;

[0120] like Similarly, it was also determined as no action detection, and recorded. ;

[0121] Calculate the response delay of the action at the confirmation time: ;in, For the first Physical time delay caused by periodic actions;

[0122] Only if S304 and S305 are satisfied will... Included in the corresponding set:

[0123] S304, And no invalid action was triggered;

[0124] S305, Confirmed ;

[0125] Press at this time Incorporate samples into the set: , ;in, , Time-delay samples of the dry layer and the wet / slippery layer are stored separately; For periodic indexes; For the first Periodic time delay;

[0126] When no action is detected, do not Include in any set;

[0127] Calculate the average time delay for each state separately:

[0128] ,

[0129] ;in, , Sets , The cardinality of the set; , The average time delays are for the dry and wet layers, respectively.

[0130] This paper proposes a process for detecting action response delay and hierarchical lag modeling. Through five sub-steps, it accurately quantifies actuator delay, solving the control instability and overshoot problems caused by time delay uncertainty. First, by physically limiting the measured speed, the signal range is fixed within a predetermined safe range, filtering out noise data exceeding the feasible range and providing stable input for subsequent time delay detection. Second, two typical invalid action scenarios are identified—requiring acceleration even when the speed reaches the upper limit or deceleration even when the speed reaches the lower limit—and these are directly marked as invalid actions, avoiding contamination of time delay statistics under actuator saturation conditions. This mechanism improves the validity and representativeness of the sample data. Third, for actions not judged as invalid, the earliest response time satisfying the speed difference threshold, direction consistency, and previous threshold not being reached is progressively searched within a future window. Multi-condition joint judgment ensures the accuracy of time delay confirmation. Fourth, the confirmed delay samples are stored in dry or wet / slippery sets according to road surface conditions, ensuring that time delay statistics under the two environments are independent and avoiding mutual interference under different conditions. The final step involves averaging the values ​​of each set and using substitute values ​​when the sample size is insufficient, ensuring that a reasonable average time delay estimate can be obtained regardless of the operating state.

[0131] The process of normalizing the measured speed, lag speed, and target speed according to the maximum safe speed, and combining them with the time delay step number and road surface state code to form a continuous state vector, while simultaneously generating the corresponding vector for the next cycle, specifically includes:

[0132] The average time delays of the dry layer and the wet slip layer are respectively converted into hysteresis steps;

[0133] In the current cycle, if the number of historical steps is not less than the number of lag steps in the corresponding layer, the effective speed of the corresponding historical cycle is read as the lag speed; otherwise, the effective speed of the first cycle is used as the lag speed.

[0134] The measured speed, the lag speed and the target speed are normalized according to the maximum safe speed, and the average time delay is represented by the number of steps. Combined with the road surface state code, they form the current period continuous state vector.

[0135] The number of lag steps for the next cycle is determined based on the road surface condition for the next cycle. The lag velocity and target velocity for the next cycle are generated to form the continuous state vector for the next cycle.

[0136] Further specific implementation steps include:

[0137] Calculate the number of lag steps for each substate:

[0138] , ;in, , These represent the hysteresis steps corresponding to the dry and wet layers, respectively. This is a function that rounds up to the nearest integer when the decimal part is strictly less than 0.5, and rounds up when the decimal part is exactly 0.5.

[0139] like ,but ;in, For the first Periodic lag velocity;

[0140] like Then, the first sample is used to complete the sample: ;

[0141] Construct the dimensionless current control cycle state vector:

[0142] ;in, For the first Periodic continuous state vector; Normalize the velocity measurement; To normalize the lag velocity; Normalize the target velocity; The dimensionless ratio for converting seconds to steps;

[0143] To construct the state vector for the next cycle, first consider the road surface state for the next cycle. Select the corresponding number of lag steps ;

[0144] like Then let ;in, The lag speed for the following week;

[0145] like Then let ;

[0146] The friction coefficient of the next cycle With non-negative clipping calculate: ;in, The target speed for the next cycle;

[0147] After normalization, we get:

[0148] ;in, This is the continuous state vector for the next cycle.

[0149] A complete scheme for constructing continuous state vectors and completing lag velocity is proposed. Through four sub-steps, it addresses the agent's need for simultaneous perception of historical and future information while ensuring the integrity of state input. The first step converts the average time delay of the dry and wet / slippery layers into lag steps and quantizes it using a half-rounding method, maintaining the operability of time delay compensation and avoiding systematic errors caused by rounding. The second step determines whether the speed of the corresponding historical period can be directly read based on the length of historical data within the current cycle; otherwise, it completes the state using the first sample, effectively solving the problem of missing states during system cold starts or when historical data is insufficient. The third step normalizes the measured speed, the compensated lag velocity, and the target speed according to the maximum safe speed, and combines the time delay ratio and road state code to form a five-dimensional vector, providing a multi-dimensional, uniform-scale environmental representation for reinforcement learning. This design not only connects historical and prospective information in the same vector but also reduces the impact of numerical scale differences on the learning process through normalization. The fourth step involves simultaneously constructing the state vector for the next cycle, allowing the agent to anticipate the input state for the next control cycle while performing the current action, further enhancing the consistency and predictability of decision-making.

[0150] The constructed set of velocity increments with intervals of distance step size is constrained within the upper and lower bounds of the physical velocity to form the velocity command for the next cycle, specifically including:

[0151] Establish a discrete set of velocity increments, and set the discrete step size and value range;

[0152] After selecting the current speed increment, add it to the current effective speed, and limit the speed according to the upper and lower limits of the physical speed to obtain the speed command for the next cycle;

[0153] Write the speed command for the next cycle into the controller.

[0154] Further specific implementation steps include:

[0155] Constructing Discrete Action Sets :

[0156] ;in, For the first One candidate velocity increment action; Integer index;

[0157] Let the currently selected action be Using a velocity-based physical clamping function Take the upper and lower limits of physical speed , The speed setting for the next cycle is obtained: ;in, Set the speed for the next cycle; , It is the formal parameter of the independent variable.

[0158] This paper proposes a discrete velocity action space construction and command limiting execution approach. Through a three-step process, it addresses the challenge of efficiently storing and updating continuous actions in tabular reinforcement learning, while maintaining both action flexibility and smoothness. First, the velocity increment is quantitatively divided into uniform step sizes, forming a discrete action set with multiple levels including negative, zero, and positive directions. This provides an enumerable and finite action space for reinforcement learning, avoiding storage explosion caused by high-dimensional continuous spaces. Second, after selecting candidate increments, physical limiting is applied based on the current effective velocity, ensuring that the final issued velocity command always remains within a safe and feasible range, effectively preventing control failure or overspeed risks caused by extreme commands. Third, the limited command is directly written to the controller, achieving seamless integration from action selection to command execution, reducing secondary calculations and software latency.

[0159] The scalar reward for the current period, which is composed of the target speed deviation term and the difference term of adjacent actions, specifically includes:

[0160] At the end of each control cycle, calculate the squared term of the target speed deviation;

[0161] Calculate the square of the difference between the velocity increment of the current cycle and the previous cycle, with the increment of the previous cycle considered as zero for the first cycle;

[0162] The sum of the two aforementioned items is taken as the negative value and cached in the value table update process.

[0163] Further specific implementation steps include:

[0164] Constructing a reward function A combined penalty for target speed deviation and motion smoothness:

[0165] Among them, the first period boundary is based on deal with; For the first Periodic scalar reward.

[0166] A dual penalty structure for bias and smoothness is designed for the reward function. This structure addresses the control jitter or lag issues caused by focusing solely on tracking error or smoothness in reinforcement learning through a three-term combination operation. The first step uses the square of the difference between the target speed and the measured speed as a tracking accuracy penalty, guiding the agent to approach the target speed limit as closely as possible with the output speed. The second step uses the square of the change in amplitude between the current action and the previous action as a smoothness penalty, suppressing operational jitter and mechanical wear caused by drastic single adjustments. The third step adds these two terms and takes the negative as the scalar reward, achieving a dynamic trade-off between accurate tracking and smooth control. This composite reward relies only on two speed measurements and two action histories, requiring no additional models or long-term historical data, thus maintaining the algorithm's simplicity and interpretability. Furthermore, treating the previous action in the first cycle as zero as a boundary condition ensures that the reward calculation during the cold start phase remains reasonable.

[0167] The process of quantizing continuous state components using a fixed number of quantization layers, establishing discrete value tables for dry and slippery layers, and incrementally updating them using a learning rate that decreases with the period and a given discount factor specifically includes:

[0168] The first three normalized components of the state vector are binned and quantized according to a fixed number of quantization layers to obtain integer indices;

[0169] The time delay component is rounded down to an integer range from one to the maximum delay detection step and then clamped to obtain the time delay index;

[0170] Use the road surface condition code as a binary index;

[0171] Using a five-element index and candidate speed increments as keys, corresponding entries are created in the dry table and the wet / slippery table, respectively.

[0172] Using a learning rate that decreases with each cycle and a given discount factor, the maximum value of the entries in the next state across all candidate actions is used as the estimation target, and incremental updates are performed on the entries of the current key.

[0173] Further specific implementation steps include:

[0174] First, deterministic quantization is performed on the continuous components of the state vector to construct the quantization function:

[0175] Among them, the quantization function Will Inner continuous quantity Binning to integers ; This is a parameter for quantizing the number of layers;

[0176] Take a fixed number of quantization layers ;

[0177] And on Calculate the discrete state index:

[0178] , , ,

[0179] , ;in, for The five components; For the first Periodic discrete state index components;

[0180] Based on this, the first... Periodic discrete state tuple ;

[0181] And this is used as the state key of the hierarchical value table, so the discrete state space is:

[0182] ;in, It is the set of Cartesian products in discrete state space;

[0183] Dry road surface: Slippery road surface: ;in, , Dry and wet layers respectively surface; It is the set of real numbers;

[0184] Construct a hierarchical value table update function:

[0185] ;in, The learning rate; This is the discount factor; For all candidate actions Take the maximum value; , These represent the layer and discrete state of the next week and the next cycle, respectively.

[0186] This paper proposes a scheme to construct discrete value tables for two different road surface conditions and implement online incremental updates. Through six steps, it addresses the challenges of tabular reinforcement learning in multi-environment switching, state dimension discretization, and online learning efficiency. First, by performing fixed-level equidistant binning on the first three components of the continuous state vector and rounding the time delay component to a reasonable range, the five-dimensional continuous state is mapped to a discrete index space. The road surface code is then loaded as a binary index, thus constructing a state key that highly distinguishes the effects of different speed differences and time delays. Second, different value tables are assigned to the dry and wet / slippery layers, ensuring that the same state index has independent value estimates under different environments, avoiding data confusion and erroneous migration. Third, a learning rate rule that decreases over time is set to balance the need for rapid early exploration and fine-grained later convergence. Fourth, a fixed discount coefficient is used to ensure a reasonable trade-off for future rewards. In the fifth step, the maximum value of all candidate actions in the next cycle is used as the estimation target in each update, which ensures the stability and convergence of temporal difference learning; the sixth step only performs incremental updates on the current key, avoiding the computational overhead of full table scan.

[0187] At the beginning of each cycle, based on the current discrete state and road surface state, the speed increment with the highest value is selected from the corresponding hierarchical value table. In case of a tie, the smallest absolute value is prioritized; if still tied, the zero increment is selected. A speed command is output, and reward calculation and value table updates are completed in the next cycle. Specifically, this includes:

[0188] At the beginning of each control cycle, read the current discrete state index and road surface state;

[0189] Select the speed increment with the highest value from the value table of the corresponding layer;

[0190] For tandem maximum values, select the speed increment with the smallest absolute value; if they are still tandem, select zero increment.

[0191] Call the amplitude limiting operation to generate the speed command for the next cycle and output it;

[0192] In the next sampling period, the measured velocity is obtained, the reward calculation is completed, and the hierarchical value table is updated.

[0193] Further specific implementation steps include:

[0194] At the beginning of each control cycle, based on the current discrete state... Road surface condition Select an action from the corresponding hierarchical value table: ;in, For the first The selected speed increment action in the cycle;

[0195] If there are tied maximum values, choose If the smallest are still tied, then choose... ;

[0196] The speed command for the next cycle is directly calculated by the speed physical clamping function and output to the controller: Then take in the next sampling period Used for reward calculation and tiered value table updates.

[0197] Focusing on action selection strategies and command output processes, this approach ensures the determinism and safety of decisions through five operations. First, at the start of each cycle, a table is consulted based on the current discrete state key and road layer code, achieving extremely low-latency decision invocation. Compared to online computation or random exploration, this scheme's pure greedy strategy provides highly stable and consistent control output. Second, to resolve ambiguity caused by parallel maximum values, a decision hierarchy of "prioritizing the smallest absolute value, then selecting zero increment" is introduced. This automatically reverts to a smoother action in parallel situations by suppressing large increments, avoiding speed jitter caused by indiscriminate increment selection. Third, the previously defined physical limiting function is directly applied to generate the speed command for the next cycle, ensuring operation within absolute safety boundaries. Fourth, the command is output to the controller, achieving seamless connection between command initiation and execution. Fifth, the measured speed is acquired in the next sampling cycle, and reward calculation and value table updates are completed, achieving a closed loop of decision-making, execution, and learning.

[0198] This embodiment also provides a system for a model vehicle speed control method based on reinforcement learning, comprising:

[0199] The friction sensing module is installed under the front of the vehicle body and collects tangential and normal forces at sampling intervals.

[0200] The speed measurement module outputs the linear velocity of the wheels.

[0201] The vehicle control unit consists of a processor and a memory. It stores discrete value tables for the dry and wet skid layers and performs status coding, target speed calculation, action selection, value table updating, and speed command generation.

[0202] The communication and execution interface sends speed commands to the driver and receives speed measurement data and friction sensing data, and then physically clamps the commands.

[0203] The power supply and timing unit provides stable power to each module and provides periodic timing trigger signals.

[0204] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0205] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speed control method for a model vehicle based on reinforcement learning, characterized in that, include: Friction sensors are installed under the front of the vehicle to collect tangential and normal forces at sampling intervals, establish normal force benchmarks and anomaly protection, calculate the friction coefficient, and generate discrete road surface state codes based on friction thresholds. In dry conditions, the maximum safe speed is used; in wet conditions, the target speed for the current period is generated by scaling the ratio of the trimmed friction coefficient to the friction threshold. The measured speed is processed to be effective and invalid actions are determined. Within the future window, a confirmation step is searched based on the significant speed difference threshold and direction consistency. The average time delay is statistically calculated in the dry layer and the wet and slippery layer respectively. The measured speed, lag speed and target speed are normalized to the maximum safe speed and combined with the time delay step number and road surface state code to form a continuous state vector, and the corresponding vector for the next cycle is generated at the same time. Construct a set of velocity increments with the step distance as the interval, limit the amplitude within the upper and lower bounds of the physical velocity, and form the velocity command for the next cycle. The target speed deviation and the difference between adjacent actions constitute the scalar reward for the current period. The continuous components of the state are quantized with a fixed number of quantization layers. Discrete value tables for the dry and slippery layers are established and incrementally updated with a learning rate that decreases with the period and a given discount factor. At the beginning of each cycle, based on the current discrete state and road surface state, the speed increment with the highest value is selected from the corresponding layered value table. If there is a tie, the minimum absolute value is prioritized. If there is still a tie, the zero increment is selected. The speed command is output and the reward calculation and value table update are completed in the next cycle.

2. The speed control method for a model vehicle based on reinforcement learning according to claim 1, characterized in that, The process of installing a friction sensor below the front of the vehicle, collecting tangential and normal forces at sampling intervals, establishing a normal force benchmark and anomaly protection, calculating the friction coefficient, and generating discrete road surface state codes based on friction thresholds specifically includes: After power-on, establish a normal force reference window and minimum boundary: take a statistical period of one second as the reference, take ten samples to form a set, calculate the average value of the normal force within the window, and set the minimum normal force boundary as a certain proportion of this average value; the sampling interval is 0.1 seconds, and the control period index is counted according to a discrete sequence; In each control cycle, a safety denominator is constructed: the larger value between the current cycle normal force and the minimum normal force boundary is used as the denominator, and the ratio of the tangential force to this safety denominator is used as the instantaneous friction coefficient; when the average normal force is equal to zero, protection is triggered, the current friction coefficient is set to zero, the road surface condition code is set to slippery, and the speed setpoint for the next cycle is set to zero, until a non-zero average value is re-established; Perform non-negative clipping and hierarchical coding: Perform non-negative clipping on the friction coefficient; when the friction coefficient before clipping is not lower than the friction threshold of 0.60, it is coded as dry; when it is lower than the threshold, it is coded as wet and slippery; write the code into the structured state data.

3. The method for controlling the speed of a model vehicle based on reinforcement learning according to claim 2, characterized in that, The process of using the maximum safe speed in a dry state and scaling the calculated target speed in a wet state based on the ratio of the trimmed friction coefficient to the friction threshold specifically includes: In dry conditions, the maximum safe speed is set to be equal to the physical speed limit of 3.0 m / s; The maximum safe speed of the dry layer is scaled by the ratio of the friction coefficient after non-negative cutting to 0.60 under wet and slippery conditions to generate the target speed for the current period. The target speed for road surface state switching is determined based on the current cycle state and cached for calculation in the next cycle.

4. The speed control method for a model vehicle based on reinforcement learning according to claim 3, characterized in that, The process of validating the measured speed and determining invalid actions, searching for confirmation steps within a future window based on a significant speed difference threshold and directional consistency, and statistically averaging the time delay in both the dry and slippery layers specifically includes: The measured speed is limited by an interval to obtain the effective speed; When the effective speed is at the physical upper limit and the action is acceleration, or at the physical lower limit and the action is deceleration, it is judged as an invalid action and the delay steps are recorded as the maximum delay detection steps; When the speed increment is not zero and no invalid action is triggered, the confirmation step that first meets the three conditions of significant speed difference threshold, direction consistency and previous step not reaching the threshold is searched step by step in the subsequent window. The time delay of each action is assigned to the dry set and the wet set respectively, and the average time delay is calculated in each set; when the set is empty, the average time delay is set to a sampling interval.

5. The method for controlling the speed of a model vehicle based on reinforcement learning according to claim 4, characterized in that, The process of normalizing the measured speed, lag speed, and target speed according to the maximum safe speed, and combining them with the time delay step number and road surface state code to form a continuous state vector, while simultaneously generating the corresponding vector for the next cycle, specifically includes: The average time delays of the dry layer and the wet slip layer are respectively converted into hysteresis steps; In the current cycle, if the number of historical steps is not less than the number of lag steps in the corresponding layer, the effective speed of the corresponding historical cycle is read as the lag speed; otherwise, the effective speed of the first cycle is used as the lag speed. The measured speed, the lag speed and the target speed are normalized according to the maximum safe speed, and the average time delay is represented by the number of steps. Combined with the road surface state code, they form the current period continuous state vector. The number of lag steps for the next cycle is determined based on the road surface condition for the next cycle. The lag velocity and target velocity for the next cycle are generated to form the continuous state vector for the next cycle.

6. The method for speed control of a model vehicle based on reinforcement learning according to claim 5, characterized in that, The constructed set of velocity increments with intervals of distance step size is constrained within the upper and lower bounds of the physical velocity to form the velocity command for the next cycle, specifically including: Establish a discrete set of velocity increments, and set the discrete step size and value range; After selecting the current speed increment, add it to the current effective speed, and limit the speed according to the upper and lower limits of the physical speed to obtain the speed command for the next cycle; Write the speed command for the next cycle into the controller.

7. The speed control method for a model vehicle based on reinforcement learning according to claim 6, characterized in that, The scalar reward for the current period, which is composed of the target speed deviation term and the difference term of adjacent actions, specifically includes: At the end of each control cycle, calculate the squared term of the target speed deviation; Calculate the square of the difference between the velocity increment of the current cycle and the previous cycle, with the increment of the previous cycle considered zero for the first cycle; The sum of the two aforementioned items is taken as the negative value and cached in the value table update process.

8. The method for speed control of a model vehicle based on reinforcement learning according to claim 7, characterized in that, The process of quantizing continuous state components using a fixed number of quantization layers, establishing discrete value tables for dry and slippery layers, and incrementally updating them using a learning rate that decreases with the period and a given discount factor specifically includes: The first three normalized components of the state vector are binned and quantized according to a fixed number of quantization layers to obtain integer indices; The time delay component is rounded down to an integer range from one to the maximum delay detection step and then clamped to obtain the time delay index; Use the road surface condition code as a binary index; Using a five-element index and candidate speed increments as keys, corresponding entries are created in the dry table and the wet / slippery table, respectively. Using a learning rate that decreases with each cycle and a given discount factor, the maximum value of the entries in the next state across all candidate actions is used as the estimation target, and incremental updates are performed on the entries of the current key.

9. The speed control method for a model vehicle based on reinforcement learning according to claim 8, characterized in that, At the beginning of each cycle, based on the current discrete state and road surface state, the speed increment with the highest value is selected from the corresponding hierarchical value table. In case of a tie, the smallest absolute value is prioritized; if still tied, the zero increment is selected. A speed command is output, and reward calculation and value table updates are completed in the next cycle. Specifically, this includes: At the beginning of each control cycle, read the current discrete state index and road surface state; Select the speed increment with the highest value from the value table of the corresponding layer; For tandem maximum values, select the speed increment with the smallest absolute value; if they are still tandem, select zero increment. Call the amplitude limiting operation to generate the speed command for the next cycle and output it; In the next sampling period, the measured velocity is obtained, the reward calculation is completed, and the hierarchical value table is updated.

10. A system employing the reinforcement learning-based model vehicle speed control method as described in claim 9, characterized in that, include: The friction sensing module is installed under the front of the vehicle body and collects tangential and normal forces at sampling intervals. The speed measurement module outputs the linear velocity of the wheels. The vehicle control unit consists of a processor and a memory. It stores discrete value tables for the dry and wet skid layers and performs status coding, target speed calculation, action selection, value table updating, and speed command generation. The communication and execution interface sends speed commands to the driver and receives speed measurement data and friction sensing data, and then physically clamps the commands. The power supply and timing unit provides stable power to each module and provides periodic timing trigger signals.

Images