Autonomous decision-making and escape method, device, medium and unmanned vehicle for trapped scene of unmanned vehicle based on reinforcement learning

Abstract

The application provides a method, device, medium and unmanned vehicle for autonomous decision-making and escaping of an unmanned vehicle in a trapped scenario based on reinforcement learning. The method comprises: after detecting that the unmanned vehicle is in a trapped state, acquiring environment state information of an environment in which the unmanned vehicle is located and self-running information; constructing a reinforcement learning state space feature at a current time; inputting the feature into a pre-trained decision-making model for escaping based on a deep reinforcement learning algorithm to determine an escaping action instruction for the unmanned vehicle; and controlling the unmanned vehicle to escape according to the action instruction. The application solves the problem of poor scenario adaptability of a traditional fixed logic scheme and has a high autonomous escaping success rate.

Description

Technical Field

[0001] This application relates to the field of autonomous driving and unmanned vehicle control technology, and more specifically, to a method, device, computer-readable storage medium, and unmanned vehicle for autonomous decision-making and escape from a trapped scenario based on reinforcement learning. Background Technology

[0002] With the rapid development of autonomous driving technology, unmanned vehicles are increasingly being used in complex and unstructured environments such as logistics delivery, field exploration, and special operations. In these complex scenarios, unmanned vehicles often face extreme road conditions such as rugged terrain, dense obstacles, or slippery surfaces, making them prone to getting stuck, such as tire slippage, chassis bottoming out, or being jammed by obstacles, which seriously affects the working efficiency and mission continuity of unmanned vehicles.

[0003] Existing solutions for unmanned vehicles to escape difficulties typically employ a rule-based fixed logic control architecture. This approach first uses onboard sensors to collect static point cloud data around the vehicle to identify obstacle boundaries. Then, the system's internal fixed state machine matches a pre-programmed library of fixed escape actions (such as performing simple reverse maneuvers or tentative turns at a fixed angle) based on the current obstacle distance. Finally, the system outputs power to the drive motor according to a pre-set single-step length to attempt to leave the trapped area.

[0004] However, this rule-based fixed logic control scheme has obvious technical flaws. Because its actual escape action commands rely heavily on the exhaustive enumeration of a pre-set rule base by the developers, the fixed state machine often cannot cover all combinations of environmental states when faced with complex and unknown non-standard trapped scenarios. At the same time, the scheme cannot dynamically combine environmental feedback and the vehicle's underlying motion mechanism to evolve the strategy, resulting in extremely poor scenario adaptability of its escape actions under changing working conditions, making it difficult to effectively escape trapped situations. Summary of the Invention

[0005] This application provides a method, apparatus, computer-readable storage medium, and unmanned vehicle for autonomous decision-making and escape from distress scenarios based on reinforcement learning, so as to at least alleviate the above-mentioned technical problems.

[0006] A method for autonomous decision-making and escape from distress scenarios for unmanned vehicles based on reinforcement learning, the method comprising: When the unmanned vehicle is detected to be in a distressed state, the environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information are obtained. Based on the environmental state information and its own operational information, construct the reinforcement learning state space features at the current moment; The reinforcement learning state space features are input into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The unmanned vehicle is controlled to perform an escape operation according to the escape action command.

[0007] Optionally, the unmanned vehicle is detected to be in a distressed state through the following steps: Obtain the actual moving speed and power output parameters of the unmanned vehicle; Determine whether the power output parameter is greater than a preset power threshold, and determine whether the actual moving speed is continuously less than a preset speed threshold within a preset time window; When it is determined that the power output parameter is greater than the preset power threshold, and the actual moving speed is continuously less than the preset speed threshold within the preset time window, it is determined that the unmanned vehicle is in a trapped state.

[0008] Optionally, constructing the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information includes: Extract the relative distance and relative angle features of the target obstacle relative to the unmanned vehicle from the environmental state information; The wheel speed characteristics and chassis attitude characteristics of the unmanned vehicle are extracted from its own operating information; The relative distance feature, the relative angle feature, the wheel speed feature, and the chassis attitude feature are fused to obtain the reinforcement learning state space feature.

[0009] Optionally, the escape decision model is trained through the following steps: An initial network model is constructed based on the deep reinforcement learning algorithm, and the network parameters of the initial network model are initialized. Control the virtual unmanned vehicle to make tentative movements in a simulated trapped scenario and collect the tentative state sequence; The trial state sequence is input into the initial network model for action value evaluation processing to obtain the corresponding evaluation result sequence; The exploratory movement is fed back and calculated according to a preset reward function to obtain a reward score sequence; The network parameters are updated using the reward score sequence and the evaluation result sequence until the network parameters meet the convergence condition. The converged initial network model is then determined as the escape decision model.

[0010] Optionally, the step of performing feedback calculations on the exploratory movements according to a preset reward function to obtain a reward score sequence includes: Obtain the position offset and obstacle distance of the virtual unmanned vehicle after it performs an exploratory movement; Determine whether the distance to the obstacle is less than a preset safety threshold; When it is determined that the distance to the obstacle is less than the preset safety threshold, the negative incentive value corresponding to the collision penalty item is calculated based on the distance to the obstacle; When the position offset is determined to represent that the virtual unmanned vehicle is far away from the initial trapped point, the positive incentive value corresponding to the escape reward item is calculated according to the magnitude of the position offset. The negative incentive values ​​and the positive incentive values ​​are weighted and summed to obtain the reward scores in the reward score sequence.

[0011] Optionally, controlling the unmanned vehicle to perform an escape operation according to the escape action command includes: Analyze the escape command and extract the target steering angle and longitudinal acceleration parameters; Based on the target steering angle parameters, the steering actuator of the unmanned vehicle is controlled to adjust the steering. Based on the target longitudinal acceleration parameters, the power actuator of the unmanned vehicle is controlled to output power.

[0012] Optionally, after controlling the unmanned vehicle to execute the escaping operation according to the escaping action command, the autonomous decision-making escaping method further includes: Obtain the current position coordinates and the starting position coordinates of the stranded vehicle; Calculate the straight-line distance between the current position coordinates and the starting position coordinates of the trapped location; If the straight-line distance does not reach the preset escape distance threshold, return to the step of obtaining the environmental state information of the environment where the unmanned vehicle is located and the unmanned vehicle's own operation information.

[0013] An autonomous decision-making and escape device for unmanned vehicles in distress scenarios based on reinforcement learning, the autonomous decision-making and escape device comprising: The information acquisition module is used to acquire environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information when it is detected that the unmanned vehicle is in a distressed state. The feature construction module is used to construct the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information. An action decision module is used to input the reinforcement learning state space features into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The execution control module is used to control the unmanned vehicle to perform the traction removal operation according to the traction removal action command.

[0014] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the above-described autonomous decision-making escape method.

[0015] An unmanned vehicle includes a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to implement the steps of the above-described autonomous decision-making and escape method.

[0016] Technical advantages of the technical solution provided in this application This application presents a reinforcement learning-based autonomous decision-making and escape method for unmanned vehicles in distress scenarios. Addressing the poor adaptability of traditional rule-based fixed logic control schemes to different escape scenarios, this method acquires environmental state information and the vehicle's own operational information upon detecting a distressed state. This solves the perception blind spot problem of traditional solutions relying solely on point cloud recognition and ignoring underlying vehicle motion feedback. Compared to traditional solutions, this method offers higher comprehensiveness in perceiving distress boundary conditions. Based on the environmental state information and the vehicle's own operational information, a reinforcement learning state space feature is constructed for the current moment, and this reinforcement learning state space... The inter-feature input is fed into a pre-trained escaping decision model to determine the escaping action command of the unmanned vehicle. This solves the technical bottleneck of traditional solutions that rely on exhaustive search of preset rules, which cannot cover unknown scenarios. Since the escaping decision model is trained based on a deep reinforcement learning algorithm, it can make inference decisions based on a continuous feature space. Compared with traditional fixed action library matching, its output escaping action command has higher scene adaptability and flexibility. Finally, the unmanned vehicle is controlled to execute the escaping operation according to the escaping action command, so that the unmanned vehicle can get rid of its dependence on prior logic and a single state machine, and ultimately has a high success rate of autonomous escaping in complex and unknown trapped environments. Attached Figure Description

[0017] Figure 1 This application provides an embodiment of an autonomous decision-making and escape method for unmanned vehicles in distress scenarios based on reinforcement learning.

[0018] Figure 2 This application provides an embodiment of an autonomous decision-making and escape device for unmanned vehicles in distress scenarios based on reinforcement learning.

[0019] Figure 3 This is a computer-readable storage medium according to an embodiment of the present application.

[0020] Figure 4 This application describes an unmanned vehicle. Detailed Implementation

[0021] like Figure 1 As shown in the figure, this application provides an embodiment of an autonomous decision-making and escape method for unmanned vehicles in distress scenarios based on reinforcement learning. The autonomous decision-making and escape method includes: When the unmanned vehicle is detected to be in a distressed state, the environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information are obtained. Based on the environmental state information and its own operational information, construct the reinforcement learning state space features at the current moment; The reinforcement learning state space features are input into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The unmanned vehicle is controlled to perform an escape operation according to the escape action command.

[0022] Optionally, the unmanned vehicle is detected to be in a distressed state through the following steps: Obtain the actual moving speed and power output parameters of the unmanned vehicle; Determine whether the power output parameter is greater than a preset power threshold, and determine whether the actual moving speed is continuously less than a preset speed threshold within a preset time window; When it is determined that the power output parameter is greater than the preset power threshold, and the actual moving speed is continuously less than the preset speed threshold within the preset time window, it is determined that the unmanned vehicle is in a trapped state.

[0023] Preferably, in determining whether an autonomous vehicle is trapped, it is necessary to comprehensively process information from the vehicle dynamics feedback level and information from the time accumulation level to distinguish between one-off, occasional mismatches between power output and motion performance and the characteristics exhibited by persistent entrapment. This application divides the detection of entrapment into three closely connected analysis stages: a quantitative acquisition stage of power propulsion intention, a continuous observation stage of motion response within a time window, and a entrapment state fusion decision stage based on dual-dimensional criteria.

[0024] Preferably, in the first stage, during the specific implementation of the quantitative acquisition stage of propulsion intention, power output parameters characterizing the working intensity of the power system are continuously acquired from the unmanned vehicle chassis control bus, and the actual moving speed of the vehicle relative to the ground is acquired from wheel speed sensors or inertial measurement units. The power output parameters can be the real-time torque value of the drive motor, the drive current value, or the power percentage value corresponding to the throttle control command. Essentially, they reflect the propulsion force actively applied by the unmanned vehicle control system to overcome current driving resistance. For example, when the unmanned vehicle cannot move forward due to tire slippage in soft sand or muddy, potholed terrain, the control system will continuously increase the output torque of the drive motor, and the power output parameters will be maintained at a high level. The actual moving speed refers to the linear velocity scalar value of the unmanned vehicle's center of gravity along the tangential direction of the ground. This velocity scalar value most intuitively reflects the macroscopic motion state of the unmanned vehicle relative to the external environment, rather than representing the rotational speed of the wheels spinning in place.

[0025] Preferably, after obtaining the two different types of feedback quantities—power output parameters and actual moving speed—the second stage introduces observation and analysis with a time accumulation dimension to construct a continuous motion response observation mechanism within a time window. This stage is designed because when a vehicle normally passes over a speed bump, briefly runs over a small protruding stone, or overcomes a minor road surface indentation, there may be a momentary surge in power output and a brief decrease in vehicle speed. If judgment is based solely on data from a single sampling moment, such normal, brief fluctuations in operating conditions are easily misinterpreted as the vehicle being stuck, leading to unnecessary intervention to extricate itself from the predicament and disrupting the continuous driving task. Therefore, this application sets a preset time window. The length of this preset time window is selected based on the sampling frequency of the control cycle and the response delay characteristics of the vehicle's transmission system. For example, it can be set to fifty consecutive control cycles (corresponding to a time span of approximately 0.5 seconds when the sensor sampling frequency is 100 Hz, or it can be increased to one second depending on the vehicle's inertia). Within this preset time window, the states where the power output parameters are greater than a preset power threshold and the states where the actual moving speed is consistently lower than a preset speed threshold need to be monitored simultaneously.

[0026] Preferably, during the time window monitoring process, each sampled value of the power output parameter is compared with a preset power threshold. This preset power threshold is set based on the maximum power output required for the autonomous vehicle to overcome internal rolling resistance while crawling at low speed on a common, smooth, hard road surface, with a certain margin above it. For example, if the drive motor torque required to maintain low-speed driving on a flat asphalt road is 15% of the motor's rated torque, then the preset power threshold can be set to 35% of the motor's rated torque. The purpose of this setting is that only when the system-recorded value of the power output parameter exceeds this preset power threshold does it indicate that the autonomous vehicle is in a strong, forceful state attempting to break free from the current driving resistance constraints. This strong force-generating state is not common during normal driving, but it will be continuously recorded when the tires slip, the chassis is stuck on rocks, or the vehicle is immobilized by mud.

[0027] Preferably, in sync with the power output parameters, continuous low-speed stability analysis is performed on the actual moving speed within the same preset time window. Specifically, the scalar value of the actual moving speed acquired at each moment is compared with a preset speed threshold, which is close to zero, for example, 0.05 meters per second, representing a state where the vehicle is almost stationary or in an extremely slow creeping state. When it is detected that among all the sampled values ​​of the actual moving speed recorded within the preset time window, a preset proportion (e.g., 90% of the consecutive sampled points within the window) of the sampled values ​​fails to exceed the preset speed threshold, the actual moving speed is considered to meet the judgment condition of being continuously less than the preset speed threshold within the preset time window. This design considers the instantaneous speed jumps that may be caused by sensor measurement noise, avoids erroneous interruption of the identification of the continuous low-speed state due to individual abnormal peak speed data, and ensures the anti-interference capability of the trapped feature identification.

[0028] Preferably, after comparing the power output parameters with the preset power threshold and determining the continuous state of the actual movement speed within the preset speed threshold within the preset time window, the third stage initiates a dual-dimensional criterion-based entrapment decision. This entrapment decision simultaneously verifies two parallel judgment results: first, confirming that the strong propulsion state (power output parameters greater than the preset power threshold) is established within the preset time window; and second, confirming that the extremely weak movement response state (actual movement speed continuously less than the preset speed threshold) is established simultaneously within the preset time window. An entrapment confirmation signal is generated only when these two judgment results, based on different physical signal sources and with contradictory attributes, are simultaneously confirmed, thus declaring that the autonomous vehicle is currently in a substantial entrapment situation. This dual-dimensional fusion discrimination architecture can fundamentally exclude scenarios where the autonomous vehicle is parking and accumulating power on a slope, actively braking and waiting—where the power output is high but the wheels are not expected to move—or from standby scenarios where the speed is zero but the drive system is not applying propulsion, from entrapment states, ensuring that the entrapment detection conclusion has high scenario specificity and working condition identification specificity.

[0029] Preferably, after generating the distress status confirmation signal, this signal will trigger the subsequent distress status flag setting logic. Specifically, a flag indicating the current driving condition of the autonomous vehicle is set as "distressed state." Setting this flag will directly drive the environmental state perception module and the self-operation information perception module to start the reinforcement learning state space feature construction process for the distress scenario. Unlike the state information collected in the normal driving mode, the reinforcement learning state space features constructed in the distressed state will be subject to an information filtering focus shift, that is, the collection weight of feature components that directly reflect the stuck state and passable geometric boundaries, such as the relative angle of nearby obstacles, the change in chassis ground clearance attitude, and the speed difference of each drive wheel, will be increased. This will enable the subsequently activated escape decision model to receive highly recognizable features that reflect the current distress mechanism, achieving targeted reasoning of escape action commands and sufficient environmental basis, forming a coherent closed-loop technical processing flow from distress status detection, distress flag establishment to distress feature system switching.

[0030] Optionally, constructing the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information includes: Extract the relative distance and relative angle features of the target obstacle relative to the unmanned vehicle from the environmental state information; The wheel speed characteristics and chassis attitude characteristics of the unmanned vehicle are extracted from its own operating information; The relative distance feature, the relative angle feature, the wheel speed feature, and the chassis attitude feature are fused to obtain the reinforcement learning state space feature.

[0031] Preferably, after detecting that the autonomous vehicle is in a trapped state and generating a trapped state confirmation signal, the construction process of reinforcement learning state space features is immediately initiated. This construction process is not a simple concatenation of raw sensor data, but follows a hierarchical feature extraction and fusion path from local to global, and from single representation to composite representation. First, obstacle spatial distribution features reflecting external geometric constraints are extracted from environmental state information. Then, dynamic features reflecting internal motion and attitude are extracted from the vehicle's own operational information. Finally, these two types of features, which have different sources and physical properties but are causally related to the same escape goal, are systematically integrated in the numerical dimension. This approach differs from the traditional approach of treating environmental perception and vehicle motion feedback separately. Instead, it unifies external constraints and internal responses within the same reinforcement learning state space feature framework, enabling the subsequent escape decision model to simultaneously read the correspondence between the current trapped boundary conditions and the vehicle's mobility potential from this reinforcement learning state space feature.

[0032] Preferably, the specific processing steps for extracting the spatial distribution features of obstacles from environmental state information focus on transforming the raw point cloud, raster map, or depth map output by the environmental perception module into relative distance and relative angle features with clear physical orientation. In this process, obstacle instance segmentation or connected component clustering is first performed on the environmental state information to distinguish multiple independent obstacle units around the autonomous vehicle. Then, for each obstacle unit, a local polar coordinate system is constructed, with the autonomous vehicle's centroid as the reference origin and the vehicle's current orientation as the reference polar axis, moving in real time with the vehicle. In this local polar coordinate system, the shortest Euclidean distance between the outer contour of the obstacle unit and the autonomous vehicle's centroid is calculated, and this distance value is determined as the relative distance feature corresponding to that obstacle unit. This relative distance feature directly reflects the amount of space the autonomous vehicle can pass in a certain direction; the shorter the distance, the tighter the local constraint. Simultaneously, in the same local polar coordinate system, the azimuth angle of the obstacle unit's centroid or the nearest contour point relative to the reference polar axis is calculated, and this azimuth angle is determined as the relative angle feature corresponding to that obstacle unit. This relative angle feature can reveal the offset of obstacles relative to the direction of travel of the autonomous vehicle, providing a basis for choosing the direction of escape maneuvers. In this way, environmental state information is converted into a series of obstacle spatial distribution feature sequences composed of one-to-one pairing of relative distance features and relative angle features, replacing the original high-dimensional disordered point cloud data and reducing data noise and redundancy in subsequent feature fusion stages.

[0033] Preferably, when organizing the obstacle spatial distribution feature sequence, the sequence can be sorted according to the numerical value of the relative angle features, and the sorted obstacle spatial distribution feature sequence can be mapped to a fixed-dimensional environmental feature vector. The dimension of this environmental feature vector corresponds to the number of divided angle intervals. For example, if the 360-degree range around the autonomous vehicle is divided into 72 angle intervals (each interval corresponds to an azimuth sector of approximately 5 degrees), then the dimension of the environmental feature vector is 72. When multiple individual obstacles exist within a certain angle interval, the relative distance feature corresponding to the obstacle with the smallest relative distance feature value is filled into the vector component position corresponding to that angle interval, and the remaining component positions are filled with the default maximum distance value representing the obstacle-free state. The environmental feature vector formed in this way retains a complete directional description of the obstacle distribution density and eliminates the influence of changes in the number of obstacles on the state space structure through the fixed-dimensional design, enabling the input interface of the subsequent escape decision model to receive environmental information from different obstacle scenarios in a unified data format.

[0034] Preferably, while extracting the spatial distribution characteristics of obstacles from environmental state information, dynamic intrinsic features are extracted in parallel from the vehicle's own operational information. The operational information originates from onboard sensing devices such as the unmanned vehicle chassis controller local area network bus, wheel speed sensors, and inertial measurement units. It includes two fundamental physical quantities that directly reflect the interaction between the vehicle and the ground: wheel speed characteristics and chassis attitude characteristics. For wheel speed characteristic extraction, the wheel speed sensor signals corresponding to each of the four or six driving and driven wheels are synchronously sampled. The collected wheel speed values ​​are sorted according to the physical installation position of the wheels on the vehicle body to form a wheel speed sequence. Subsequently, differential speed analysis is performed on the wheel speed sequence, that is, the speed difference between the average speed of the left wheel and the average speed of the right wheel, and the speed difference between the average speed of the front axle and the average speed of the rear axle are calculated. These speed differences, along with the original speed values ​​of each wheel, are included in the wheel speed characteristics. The wheel speed characteristics can not only reflect whether the vehicle is slipping in place (manifested as the driving wheel speed is high while the driven wheel speed is close to zero), but also indicate the yaw trend of the vehicle in unstructured terrain due to one side being suspended or one side having abnormal resistance through the difference in left and right speeds, providing fine-grained clues about the mechanism of loss of mobility for the escape decision model.

[0035] Preferably, the extraction of chassis attitude features involves acquiring time-series data of the autonomous vehicle's pitch angle, roll angle, and vertical acceleration in three-dimensional space from the inertial measurement unit (IMU). In trapped scenarios, chassis bottoming out or the vehicle being lifted by large boulders often leads to abnormal steady-state shifts in pitch or roll angles, while transient changes in vertical acceleration reflect the energy transfer path of the suspension system attempting to break free. Therefore, the chassis attitude features are constructed as a multi-dimensional array consisting of the pitch angle and roll angle at the current sampling moment, and the peak and root mean square values ​​of vertical acceleration within a short backtracking period (e.g., within 0.2 seconds). By concatenating the wheel speed features and chassis attitude features along the feature dimensions, the internal dynamics feature vector is obtained. This internal dynamics feature vector is physically independent of the aforementioned environmental feature vector, but they complement each other in technical function: the environmental feature vector describes the static geometric constraints of the external obstacle field, while the internal dynamics feature vector characterizes the vehicle's mechanical response state under the current constraints. Together, they constitute a complete state description of the trapped scenario.

[0036] Preferably, after obtaining the environmental feature vector and the internal dynamics feature vector, the feature fusion stage synthesizes these two heterogeneous feature vectors in a unified numerical space to form reinforcement learning state space features. This synthesis process is not a simple vector stacking; instead, it first performs dimensionless normalization on the environmental and internal dynamics feature vectors respectively, mapping each component value to the same numerical range (e.g., the range from zero to one) to eliminate statistical biases that may be caused by scale differences between different dimensions such as relative distance, angle, rotational speed, and acceleration in subsequent numerical calculations. After normalization, the normalized environmental feature vector and the normalized internal dynamics feature vector are concatenated end-to-end according to a preset feature concatenation order to form a single high-dimensional feature vector, which is the reinforcement learning state space feature at the current moment. Since the dimensions of both the environmental and internal dynamics feature vectors are fixed, the total dimension of the reinforcement learning state space features is also fixed. This design allows the number of input layer neurons in the escape decision model to be determined once without dynamic adjustment as the scene changes, reducing the engineering complexity of model deployment.

[0037] Preferably, once the reinforcement learning state space features are constructed, they serve as the link between the perception module and the decision-making module. These reinforcement learning state space features are fed into a pre-trained escape decision-making model for forward inference. The escape decision-making model, based on a policy network trained using deep reinforcement learning algorithms, extracts higher-level interaction features layer by layer from each component of the reinforcement learning state space features, ultimately generating escape action commands in the action space. Because the reinforcement learning state space features have already refined and structured the core features of the trapped scenario during the construction phase through the fusion of relative distance features, relative angle features, wheel speed features, and chassis attitude features, the escape decision-making model no longer needs to learn feature extraction methods from massive amounts of raw sensor data. This reduces the number of sample interactions required for the policy network to converge during training and improves the inference response speed of escape action commands in practical applications. In extremely challenging terrains such as mud-adhesive areas and jagged, gravelly terrain, the roll anomaly component in the chassis attitude features and the minimum relative distance feature value directly in front in the environmental side feature vector will form a correlation pattern in the reinforcement learning state space features. Based on this, the escape decision model can infer that continuing forward with high thrust to escape is not feasible, and instead outputs a composite escape action command of first turning to avoid the obstacle and then crawling backward, demonstrating the scene adaptive decision-making capability supported by the finely constructed reinforcement learning state space features.

[0038] Optionally, the escape decision model is trained through the following steps: An initial network model is constructed based on the deep reinforcement learning algorithm, and the network parameters of the initial network model are initialized. Control the virtual unmanned vehicle to make tentative movements in a simulated trapped scenario and collect the tentative state sequence; The trial state sequence is input into the initial network model for action value evaluation processing to obtain the corresponding evaluation result sequence; The exploratory movement is fed back and calculated according to a preset reward function to obtain a reward score sequence; The network parameters are updated using the reward score sequence and the evaluation result sequence until the network parameters meet the convergence condition. The converged initial network model is then determined as the escape decision model.

[0039] Preferably, the training process of the escape decision-making model relies on a designed four-layer structured deep reinforcement learning network architecture. This architecture, from input to output, sequentially includes a state embedding layer, a policy convolutional layer, a dual evaluation competition layer, and a fusion update layer. These four layers do not operate independently but rather organically cooperate in each training iteration driven by tentative movements, following a bidirectional data flow of forward evaluation and backward parameter updates. This allows the model to gradually learn a policy from the simulated trapped scenario that maps the current trapped state to effective escape action commands. The following sections, in conjunction with the training steps, explain the technical role of each structural layer and the inter-layer cooperation relationships.

[0040] Preferably, the state embedding layer serves as the entry structural layer of the escape decision model, directly receiving the trial state sequence generated by each round of tentative movement. During the initial network model building phase, after determining the network parameters based on the deep reinforcement learning algorithm, the state embedding layer is tasked with compactifying and recoding the high-dimensional, sparse trial state sequence. In the trapped scenario, the trial state sequence contains the aforementioned reinforcement learning state space features. These features include both obstacle spatial distribution constraints represented by relative distance and relative angle features, and the vehicle-ground interaction state reflected by wheel speed and chassis attitude features. The numerical ranges of these feature components vary considerably. The state embedding layer uses a set of trainable linear embedding matrices to group, scale, and offset the reinforcement learning state space features according to component type, mapping them to an embedded state vector with a fixed dimension and values ​​within a similar dynamic range for each dimension. For example, in a simulated trapped scenario, the original values ​​of the relative distance feature might range from zero to tens of meters, while the root mean square value of the vertical acceleration might only fluctuate to a few decimal places. The embedding matrix of the state embedding layer adjusts these two types of heterogeneous components to a numerical range with zero mean and controllable variance through feature-wise scaling and translation factors. This embedded state vector provides subsequent structural layers with a numerically stable processing object, avoiding oscillations in network parameter updates caused by significant differences in input scale.

[0041] Preferably, the policy convolutional layer receives the embedded state vector from the state embedding layer and is responsible for extracting spatial interaction patterns and temporal dependency patterns related to the escape action decision. In terms of structural design, the policy convolutional layer does not use traditional two-dimensional image convolution kernels, but instead constructs a hybrid extraction structure that combines a set of one-dimensional temporal convolution channels and a set of fully connected mapping branches in parallel. The one-dimensional temporal convolution channels unfold the embedded state vector in the time dimension, performing sliding window convolution operations along the time axis using multiple convolution kernels of different scales to capture the dynamic changes in wheel speed features and chassis attitude features during continuous trial movement of the vehicle, such as the implicit representation of the typical slippage pattern where the drive wheel speed continuously increases while the actual moving speed remains extremely low. The fully connected mapping branches perform nonlinear spatial compression on the embedded state vector at the current moment. Through two layers of fully connected computation with rectified activation functions, the environmental constraint information encoded by relative distance features and relative angle features is transformed into a compact environmental constraint feature vector. The environmental constraint feature vector and the dynamic trend feature vector output by the one-dimensional temporal convolution channel are concatenated in the channel dimension to obtain the policy hidden layer feature, which carries both the spatial feasibility of the current trapped site and the temporal evolution clues of the vehicle's motion state.

[0042] Preferably, the dual-evaluation competition layer receives the policy hidden layer features output from the policy convolutional layer and performs action value evaluation processing on them based on a dual-evaluation head competition architecture to generate an evaluation result sequence. Unlike traditional single Q-value evaluation networks, the dual-evaluation competition layer internally sets up two sets of structurally identical but independently initialized first and second evaluation subnets. The policy hidden layer features are simultaneously fed into the first and second evaluation subnets. Each evaluation subnet performs computation through several fully connected layers, outputting a first action evaluation value and a second action evaluation value, respectively. The two sets of action evaluation values ​​together constitute the evaluation result sequence. In each round of exploratory movement, the dual-evaluation competition layer also independently generates a candidate escape action command for the current embedded state vector based on the policy hidden layer features via a lightweight action generation branch network. This candidate escape action command consists of a numerical pair of target steering angle parameters and target longitudinal acceleration parameters. The candidate escape action command, along with the first and second action evaluation values, is fed into the backend fusion update layer. The purpose of the dual evaluation competition layer is to alleviate the overestimation bias that single evaluation networks are prone to in scenarios where sparse rewards are a problem. When two evaluation subnetworks give significantly different evaluation values ​​for the same state-action combination, the fusion update layer can select the more conservative evaluation value as the basis for updating, thereby avoiding the policy being misled by a few overly excited actions that accidentally obtain positive rewards in the early stages of training.

[0043] Preferably, the fusion update layer is a pivotal structure layer connecting the forward evaluation path and the backward update path during training, performing synchronous calibration of network parameters using the reward score sequence and the evaluation result sequence. After the virtual autonomous vehicle completes a trial movement, a preset reward function calculates the reward score corresponding to the trial movement based on the position offset and obstacle distance, and incorporates this reward score into the reward score sequence. The fusion update layer extracts the reward score for the current step from the reward score sequence, extracts the first action evaluation value and the second action evaluation value from the evaluation result sequence, and then calculates two independent temporal difference error values ​​for the first evaluation subnet and the second evaluation subnet, respectively, according to the designed temporal difference error construction rules. When calculating the temporal difference error value, the fusion update layer introduces a target embedding state vector, which is generated by the embedding state vector of the next time step through a delayed-updated target embedding layer, to reduce the fluctuations introduced by drastic changes in the embedding representation when calculating the target value. For the two temporal difference error values, the fusion update layer takes the smaller one as the base update error, and then performs gradient descent updates on the network parameters of the first and second evaluation sub-networks respectively using the base update error. Subsequently, to reduce the sensitivity of action evaluation values ​​to action selection, the fusion update layer correspondingly delays the update of the network parameters of the policy convolutional layer. That is, after the evaluation sub-network is updated several times, a single gradient ascent adjustment is performed on the network parameters of the policy convolutional layer along the direction that maximizes the output of the first evaluation sub-network, using the current policy hidden layer features as input. The network parameters of the state embedding layer are updated via backpropagation along with the policy convolutional layer during the above parameter update process. Through the above alternating parameter calibration, the converged initial network model is the trained escape decision model.

[0044] Preferably, the coordination process of the above four structural layers throughout the training cycle can be summarized as follows: In a simulated trapped scenario, the virtual unmanned vehicle makes tentative movements based on the escape action commands output by the escape decision model. After executing the escape action commands, a preset reward function calculates the feedback reward, and the transition relationship between the new reinforcement learning state space features and the old reinforcement learning state space features is stored as an attempt state sequence. Subsequently, the state embedding layer compresses the reinforcement learning state space features in the attempt state sequence into embedded state vectors. The policy convolutional layer extracts policy hidden layer features from the embedded state vectors. The dual evaluation competition layer generates candidate escape action commands and evaluation result sequences based on the policy hidden layer features. The fusion update layer then calculates the temporal difference error between the reward score sequence fed back by the preset reward function and the evaluation result sequence, and updates the network parameters layer by layer in reverse. This process is repeated in multiple rounds of simulated trapped scenarios until the average reward score of the policy network corresponding to the network parameters no longer increases significantly in multiple consecutive evaluation cycles, at which point the convergence condition is satisfied.

[0045] Preferably, under the condition that the simulated entrapment scenarios cover multiple entrapment prototypes such as tire slippage, chassis bottoming out, and being stuck by obstacles, the entrapment decision model trained by the above four-layer structure has a one-dimensional temporal convolutional channel in its policy convolutional layer that spontaneously forms a strong activation response to the physical causal chain of increased speed difference—no vehicle speed response. The dual evaluation competition layer, through a conservative smaller value evaluation mechanism, suppresses the exaggerated prediction of action value due to accidental success in an extremely narrow entrapment window. In practical applications, after this entrapment decision model is loaded into the onboard computing unit of an autonomous vehicle, when the entrapment state is detected and reinforcement learning state space features are constructed, the state embedding layer can complete the embedding mapping within a few milliseconds. Subsequently, the policy convolutional layer and the dual evaluation competition layer can output the entrapment action command in a single forward inference without the need for online network parameter updates, thus keeping the entrapment decision latency in unstructured terrain at a low level.

[0046] Optionally, the step of performing feedback calculations on the exploratory movements according to a preset reward function to obtain a reward score sequence includes: Obtain the position offset and obstacle distance of the virtual unmanned vehicle after it performs an exploratory movement; Determine whether the distance to the obstacle is less than a preset safety threshold; When it is determined that the distance to the obstacle is less than the preset safety threshold, the negative incentive value corresponding to the collision penalty item is calculated based on the distance to the obstacle; When the position offset is determined to represent that the virtual unmanned vehicle is far away from the initial trapped point, the positive incentive value corresponding to the escape reward item is calculated according to the magnitude of the position offset. The negative incentive values ​​and the positive incentive values ​​are weighted and summed to obtain the reward scores in the reward score sequence.

[0047] Preferably, the process of using a preset reward function to calculate the reward score sequence for exploratory movements is directly embedded before the fusion update layer of the four-layer structured deep reinforcement learning network architecture used to train the aforementioned escape decision model. This serves as a signal conversion link connecting the execution results of exploratory movements with the parameter calibration of the fusion update layer. In one round of exploratory movement, after the virtual unmanned vehicle completes an action based on the candidate escape action instructions output by the dual evaluation competition layer, the simulation environment will provide a new environmental state after the action. The preset reward function then uses the execution result of this exploratory movement as a basis, and through a dual-channel parallel incentive structure of geometric space measurement and safety boundary constraints, quantifies the safety cost and the benefit in terms of the escape objective of this exploratory movement, thereby providing the fusion update layer with dense reward scores that can stably guide the policy gradient.

[0048] Preferably, in the first channel of the feedback calculation, the position offset and obstacle distance of the virtual unmanned vehicle after performing a trial movement are acquired. Specifically, the position of the virtual unmanned vehicle in the simulated trapped scenario is represented by two-dimensional Cartesian coordinates. Before performing the trial movement, the position of the virtual unmanned vehicle is recorded as the pre-movement position, and after performing the trial movement, the position of the virtual unmanned vehicle is recorded as the post-movement position. The Euclidean distance between the post-movement position and the initial trapped point coordinates is defined as the original value of the position offset, and the shortest geometric gap between the outer contour of the virtual unmanned vehicle at the post-movement position and the nearest obstacle is determined as the obstacle distance. The position offset and obstacle distance point to two contradictory sub-objectives in the escape mission: an increase in the position offset means that the virtual unmanned vehicle is leaving the physical center of the trapped area, reflecting the progress of escape; while a decrease in the obstacle distance indicates that the collision risk is increasing, reflecting the tightness of safety constraints. The simultaneous acquisition of these two physical quantities constitutes the data basis for subsequent positive and negative bidirectional excitation calculations.

[0049] Preferably, after obtaining the obstacle distance, it is compared with a preset safety threshold to determine whether the current exploratory movement has excessively approached the obstacle. The preset safety threshold is determined based on factors such as the autonomous vehicle's body size, maximum speed, and sensor blind spots. For example, for an autonomous vehicle with a maximum outer diameter of 0.6 meters, the preset safety threshold can be set to 1.2 meters to ensure that there is at least a buffer distance of one vehicle width between the vehicle's outer contour and the obstacle. When the obstacle distance is less than the preset safety threshold, the exploratory movement is determined to have entered a high-collision-risk zone. At this time, a negative incentive value corresponding to the collision penalty is calculated based on the obstacle distance. When calculating the negative incentive value, a nonlinear mapping function based on the reciprocal of the distance is used to convert the obstacle distance into a negative penalty scalar. The closer the distance is to zero, the more rapidly the absolute value of the negative incentive value increases, thereby applying a strong gradient suppression signal to exploratory actions that may lead to collisions. When the distance to the obstacle is greater than or equal to the preset safety threshold, the negative incentive value corresponding to the collision penalty is directly set to zero, indicating that the exploratory movement is within an acceptable safety range.

[0050] Preferably, in parallel with the calculation path of the collision penalty term, the position offset is used to determine the intention to escape, in order to determine whether the positive incentive value corresponding to the escape reward term is triggered. In the design, not all tentative movements that generate position offsets necessarily indicate that the virtual unmanned vehicle is moving away from the initial trapped point, because the virtual unmanned vehicle may perform detours, U-turns, or lateral pathfinding operations in the obstacle-dense trapped area. The position offsets generated by these operations may be numerically perpendicular to or even opposite to the escape direction. Therefore, before calculating the positive incentive value, a reference escape vector is introduced. The direction of this reference escape vector points from the initial trapped point to the current maximum passable direction of the virtual unmanned vehicle. This maximum passable direction is obtained by analyzing the environmental feature vectors in the reinforcement learning state space features, specifically taking the orientation corresponding to the angle interval with the largest relative distance feature value in the environmental feature vectors. The displacement vector between the moved position and the original position is projected along the direction of this reference escape vector. If the projection value is positive and exceeds a starting threshold, the position offset of this tentative movement is determined to indicate that the virtual unmanned vehicle is moving away from the initial trapped point. In this scenario, a positive incentive value corresponding to the escape reward is calculated based on the magnitude of the projected value. This positive incentive value adopts an increasing function form with an upper bound, such as a logarithmic function. This ensures that even small initial escape progress yields substantial incentives, but the marginal contribution of simply increasing distance to the reward gradually decreases in the later stages, guiding the strategy to balance escape efficiency and path smoothness. If the projected value does not exceed the initial threshold or is negative, the positive incentive value is set to zero, indicating that the exploratory movement has not yet constituted meaningful escape progress.

[0051] Preferably, after the negative incentive value corresponding to the collision penalty and the positive incentive value corresponding to the escape reward are calculated, these two incentive values ​​are weighted and summed to obtain the reward score corresponding to the exploratory movement. The weight coefficients for the weighted summation can be configured differently according to the type of trapped scenario to adapt to the safety and escape priorities under different trapped conditions. For example, in a chassis bottoming-out trapped scenario, the bottom of the virtual unmanned vehicle is supported by rocks or embankments, and the wheels are partially suspended in the air. The drive wheels are prone to high-speed free-spinning in the suspended state. If a large steering or acceleration action is applied at this time, there is a high probability that the vehicle will sideslip and collide with the obstacle. Therefore, in this type of scenario, the weight coefficient of the collision penalty can be increased to enhance the sensitivity to the safety boundary. In a tire slipping trapped scenario, the obstacle layout is relatively sparse and the collision risk is low. The weight coefficient of the escape reward can be increased to encourage the virtual unmanned vehicle to try different escape directions more quickly. The reward score obtained by weighted summation is stored in the reward score sequence in real time for extraction by the fusion update layer.

[0052] Preferably, the reward score sequence works bidirectionally with the evaluation result sequence during the training of the escape decision model, providing a quantitative basis for updating network parameters in the fusion update layer. When the fusion update layer performs temporal difference error calculation, the reward score for the current step is extracted from the reward score sequence and used together with the first and second action evaluation values ​​obtained from the dual evaluation competition layer to construct the temporal difference error. When a tentative movement obtains a large negative incentive value due to approaching an obstacle, the value of the temporal difference error changes accordingly and is then transmitted to the gradient descent update of the first and second evaluation subnets via the basic update error. This allows the dual evaluation competition layer to lower the evaluation value pointing to the dangerous action when facing similar embedded state vectors in the future. Conversely, when positive incentive values ​​are continuously injected into the reward score sequence, it effectively alleviates the policy learning lag caused by the sparse number of successful escape samples in the trapped scenario, enabling the one-dimensional temporal convolutional channel and fully connected mapping branch in the policy convolutional layer to establish state associations for effective escape directions within fewer training rounds.

[0053] Preferably, through the aforementioned feedback calculation architecture of parallel incentives via collision penalty and escape reward, followed by weighted summation and fusion, the reward score sequence generated by the preset reward function can provide a high-density and clearly directional training signal for parameter update throughout the entire training cycle of the escape decision model. Compared to the traditional approach that uses the final success of escape as the basis for sparse reward distribution, this preset reward function treats each tentative movement as an evaluable intermediate decision node. By comparing the obstacle distance with the preset safety threshold and judging the directionality of the position offset with the reference escape vector, it decomposes safety constraints and escape progress into superimposed instantaneous incentive components. This allows the escape decision model to quickly eliminate invalid action spaces with obvious collision risks through the penalty feedback of negative incentive values, even when no successful escape experience has been accumulated in the early stages of the entrapment. It can also gradually converge to a strategy of progressive escape along the sparse direction of obstacles through positive incentive values, ultimately coalescing into an escape strategy with adaptive capabilities for complex and unknown entrapment scenarios in the converged initial network model.

[0054] Preferably, the process of inputting reinforcement learning state space features into a pre-trained escape decision model to determine the autonomous vehicle's escape action commands relies on a single forward inference mode after deployment of a trained four-layer structured deep reinforcement learning network architecture. In this inference mode, the fusion update layer no longer participates in the reverse update iteration of network parameters. Instead, the state embedding layer, policy convolutional layer, and dual evaluation competition layer collaboratively complete the end-to-end mapping from high-dimensional state features to specific escape action commands through sequential data flow. After the autonomous vehicle detects the trapped state and constructs the reinforcement learning state space features at the current moment, these features, as the state input to be processed, are first injected into the state embedding layer of the escape decision model, thereby initiating the forward inference pipeline.

[0055] Preferably, after receiving the reinforcement learning state space features, the state embedding layer immediately performs compactification recoding to generate an embedded state vector. Given that the escape decision model has converged, the values ​​of each element of the linear embedding matrix in the state embedding layer have been trained and locked to a set of optimal parameter combinations. This linear embedding matrix performs feature-wise scaling and offset transformations on the components with different physical dimensions within the reinforcement learning state space features. For example, the relative distance feature component from the environmental feature vector in the reinforcement learning state space features may have original values ​​between one and eight meters in a tire-skidding entrapment scenario, while the root mean square component of the vertical acceleration from the internal dynamic feature vector may be distributed between 0.01 and 0.3 times the gravitational acceleration. The state embedding layer maps the former to a normalized numerical range with zero mean and controllable variance, and the latter to a normalized numerical range of the same magnitude, respectively, using scaling and offset factors corresponding to these two components in the embedding matrix. All components after this processing are concatenated to form the embedded state vector. This embedded state vector is both the output of the state embedding layer and the only information input data for the subsequent policy convolutional layer.

[0056] Preferably, the policy convolutional layer processes the embedded state vector, enabling its internal one-dimensional temporal convolutional channel and fully connected mapping branch to extract dynamic trend feature vectors and environmental constraint feature vectors in parallel, and then fuse them to generate policy hidden layer features. During the inference phase, the one-dimensional temporal convolutional channel does not require additional historical state caching. Instead, it utilizes the temporal unfolding information already retained in the feature dimension of the embedded state vector, performing sliding window convolution operations on the embedded state vector along the time axis using multiple one-dimensional convolutional kernels of different scales. These convolutional kernels, during training, have formed strong activation response patterns based on the slippage patterns repeatedly occurring in simulated trapped scenarios, such as the increase in drive wheel speed and the lack of vehicle speed response. This allows them to extract dynamic trend feature vectors reflecting the changes in the mechanical state between the vehicle and the ground from the current embedded state vector. Almost simultaneously, the fully connected mapping branch performs two layers of fully connected nonlinear spatial compression calculations with rectified activation functions on the embedded state vector, mapping the obstacle spatial distribution constraint components—i.e., the encoded information derived from relative distance and relative angle features—into a low-dimensional and compact environmental constraint feature vector. By concatenating and merging the dynamic trend feature vector and the environmental constraint feature vector along the channel dimension, the policy hidden layer feature is obtained. This policy hidden layer feature is then passed to the dual evaluation competition layer to provide the complete contextual representation required for action value assessment and action generation under the current trapped state.

[0057] Preferably, after the dual evaluation competition layer reads the hidden features of the strategy, it independently generates candidate escape action commands through its internal action generation branch network. The first and second evaluation subnetworks each provide corresponding action evaluation values ​​for selection. The action generation branch network performs several fully connected calculations on the hidden features of the strategy, directly mapping the output to a numerical pair consisting of the target steering angle parameter and the target longitudinal acceleration parameter. This numerical pair is the numerical expression of the candidate escape action command. The target steering angle parameter, in radians or degrees, specifies the expected deflection angle of the autonomous vehicle's steering actuator; the target longitudinal acceleration parameter, in meters per second squared, specifies the expected acceleration / deceleration intensity of the autonomous vehicle's power actuator. Synchronously with the candidate escape action commands, the first and second evaluation subnetworks perform their own independent multi-layer fully connected nonlinear transformations on the hidden features of the strategy, each outputting a first action evaluation value and a second action evaluation value. These two action evaluation values ​​represent the escape decision model's estimate of the long-term cumulative reward that may be obtained by executing the candidate escape action command in the current trapped state. Since the first and second evaluation subnets have been calibrated by the fusion update layer using a conservative lower-value mechanism of dual evaluation competition during the training phase, the lower value of the two action evaluation values ​​can often suppress overly optimistic value estimates. Thus, even when faced with unfamiliar and trapped terrain that was not accurately covered during training, a relatively prudent value judgment benchmark can still be maintained during inference.

[0058] Preferably, after obtaining candidate escape action commands, first action evaluation values, and second action evaluation values ​​in the dual evaluation competition layer, the final output escape action command can be determined based on the comparison result of these two action evaluation values. One selection rule is that when the difference between the first action evaluation value and the second action evaluation value is within a small threshold range, it indicates that the two evaluation subnets have reached a high degree of consensus on the value of the current candidate escape action command. In this case, the candidate escape action command output by the action generation branch network is directly output as the escape action command to the downstream execution control link. When the difference between the evaluation values ​​of the first and second actions exceeds a certain threshold, it indicates a significant discrepancy between the two evaluation subnets in their value judgments of the current state-action combination. In this case, a more robust strategy can be adopted: select the evaluation subnet corresponding to the larger of the first and second action evaluation values, using its corresponding action evaluation value as a value reference. However, instead of directly adopting the candidate escape action command, trigger the action generation branch network to perform a limited number of sampling evaluations in a small perturbation space near the target steering angle and longitudinal acceleration parameters. Each sampling generates a fine-tuned candidate action, and the two evaluation subnets recalculate the evaluation value. The fine-tuned action with the highest evaluation value is then selected as the final escape action command. This mechanism, combining selection and verification, achieves a balance between inference speed and decision quality, avoiding overly aggressive escape actions output due to value estimation spikes in local regions of the state space.

[0059] Preferably, the single forward inference process, which is completed sequentially by the state embedding layer, the policy convolutional layer, and the dual evaluation competition layer, is the entire computational process that converts the reinforcement learning state space features into escape action commands. Its time consumption depends only on the forward propagation computation of each neural network layer, without requiring online gradient backpropagation or network parameter updates. In the deployment practice of the onboard computing unit, the matrix multiplication of the state embedding layer can be completed within a few microseconds, the one-dimensional convolution and fully connected operations of the policy convolutional layer can be further parallelized, and the action generation and evaluation computation of the dual evaluation competition layer can also return results within milliseconds. Therefore, from the completion of the reinforcement learning state space feature construction to the output of the escape action command, the entire inference latency is controlled at a low level, meeting the latency constraints of real-time escape decision-making for unmanned vehicles in unstructured terrain. In emergency situations where the chassis bottoms out, causing some drive wheels to be suspended in the air, and the power output parameters are continuously higher than the preset power threshold while the actual movement speed is still close to zero, the escape decision model can utilize the highly activated response of the one-dimensional temporal convolution channel of the strategy convolution layer to the suspension and slippage feature, as well as the low value evaluation of the high-throttle forward action in this state by the dual evaluation competition layer, to quickly generate escape action instructions with the core of first steering to avoid and then low-speed creeping. This avoids the risk of secondary bottoming out or vehicle sideslip caused by erroneous high-thrust attempts, demonstrating the technical effectiveness of the carefully designed four-layer network structure and the efficient closed-loop mapping of actual escape actions.

[0060] Optionally, controlling the unmanned vehicle to perform an escape operation according to the escape action command includes: Analyze the escape command and extract the target steering angle and longitudinal acceleration parameters; Based on the target steering angle parameters, the steering actuator of the unmanned vehicle is controlled to adjust the steering. Based on the target longitudinal acceleration parameters, the power actuator of the unmanned vehicle is controlled to output power.

[0061] Preferably, the process of controlling the unmanned vehicle to execute traction-avoidance operations according to the traction-avoidance action commands is a key execution step that transforms the numerical decision results generated by the traction-avoidance decision model into the vehicle's physical motion. This process begins with the parsing of the traction-avoidance action commands, and then applies the obtained motion control parameters to the steering and power actuators of the unmanned vehicle, forming coordinated motion control in both lateral and longitudinal dimensions. In unstructured traction scenarios, steering adjustment and power output are not isolated open-loop operations, but rather require adaptive distribution and execution readiness verification of the command parameters during execution, taking into account the characteristics of the traction condition and the response constraints of the vehicle's underlying actuators. This ensures that the traction-avoidance action commands can be transformed into physical motions that effectively change the vehicle's posture and position.

[0062] Preferably, when parsing the escape action command to extract the target steering angle parameter and the target longitudinal acceleration parameter, the data frame structure of the escape action command is decomposed domain by domain. The escape action command is directly output by the action generation branch network of the dual evaluation competition layer in the escape decision model. Its data frame contains a numerical pair, where the first numerical bit corresponds to the target steering angle parameter and the second numerical bit corresponds to the target longitudinal acceleration parameter. The parsing process first reads the frame header identifier of the data frame to confirm that the command is a valid escape action command type. Then, according to the preset numerical pair encoding format, the first and second numerical bits are separated and extracted from the effective payload segment of the data frame. The target steering angle parameter is represented by a signed radian value or degree value, with the positive and negative signs corresponding to the left and right yaw directions of the unmanned vehicle's steering actuator, respectively. The target longitudinal acceleration parameter is represented by a signed meter-per-second square value, with the positive and negative signs corresponding to acceleration and deceleration braking intentions, respectively. Through this structured parsing method, the escape action command is decomposed into two motion control components with clear physical meanings that can be directly received by the actuator controller.

[0063] Preferably, after obtaining the target steering angle parameters, they are numerically converted according to the control interface protocol of the unmanned vehicle's steering actuator, thereby controlling the steering actuator to perform steering adjustment. The steering actuator typically consists of a steering motor, steering linkage, and position feedback sensor, and its control interface receives the desired steering angle position command. The difference between the target steering angle parameters and the actual steering angle transmitted back by the position feedback sensor at the current moment is calculated to obtain the steering angle deviation. A closed-loop position adjustment calculation is performed on this steering angle deviation. The adjustment calculation can employ a proportional-integral-derivative (PID) control law, with the proportional, integral, and derivative coefficients tuned based on the inertia and response bandwidth of the steering actuator. The steering motor drive current command obtained through the position adjustment calculation drives the steering motor to move the steering linkage, causing the wheels to deflect to the angle position specified by the target steering angle parameters. In situations where a vehicle is stuck, such as when one wheel of an autonomous vehicle is stuck in a mud pit while the other wheel still has traction, the extrication command may require a large steering angle to adjust the vehicle's orientation. The steering actuator executes this angle adjustment with a high response speed, which helps the vehicle's center of gravity shift towards the side with traction, creating effective traction conditions for subsequent power output.

[0064] Preferably, while performing steering adjustments, the power actuator of the unmanned vehicle is controlled to output power based on the target longitudinal acceleration parameters. The power actuator includes a drive motor controller, a drive motor, and a transmission system, and its control interface receives the desired longitudinal acceleration command or equivalent torque command. The target longitudinal acceleration parameters are input into a designed acceleration-torque mapping model. This model converts the longitudinal acceleration command into the desired output torque of the drive motor based on the unmanned vehicle's current mass estimate, wheel rolling radius, and estimated driving resistance of the current terrain. Considering the drastic changes in the coefficient of friction between the tires and the ground in unstructured entrapment terrain, an adhesion utilization constraint factor is introduced during the torque conversion process. This factor is dynamically adjusted by real-time monitoring of the slip rate of the drive wheels. When the slip rate exceeds a preset upper limit threshold, the adhesion utilization constraint factor is set to a value less than one to reduce the amplitude of the desired output torque and prevent the drive wheels from spinning at high speed on low-adhesion surfaces, which would lead to energy dissipation and increased ground erosion. The final torque command, after constraint adjustment, is sent to the drive motor controller, which controls the drive motor to output the corresponding driving torque, which is then transmitted to the wheels through the transmission system.

[0065] Preferably, during the continuous execution of power output, the target longitudinal acceleration parameter is further segmented and refined on the time axis to adapt to the continuous action sequence during the entrapment and escape process. When the entrapment decision model outputs an entrapment action command corresponding to a composite entrapment strategy consisting of first turning and then crawling backward, the target longitudinal acceleration parameter is divided into two stages in time: the first stage is accompanied by a turning process with a large turning angle, where the target longitudinal acceleration parameter is zero or a small positive value to maintain a slight forward movement of the vehicle; the second stage, after the turning angle is adjusted to the correct position, switches the target longitudinal acceleration parameter to a negative value, driving the unmanned vehicle to move away from the entrapment area in a low-speed and stable reverse manner. When executing the above segmented commands, the power actuator continuously adjusts the direction and magnitude of the output torque of the drive motor according to the changes in the sign and magnitude of the target longitudinal acceleration parameter in each stage, achieving a smooth transition between acceleration and reverse drive, and avoiding mechanical impact loads on the transmission system due to sudden reversal of torque direction.

[0066] Preferably, throughout the entire process of steering adjustment and power output, the actual steering angle is continuously obtained from the position feedback sensor of the steering actuator, and the actual output torque is obtained from the drive motor controller of the power actuator. These actual execution state data are compared with the target steering angle parameters and target longitudinal acceleration parameters in the escape command, forming a continuous control loop of execution-feedback-adjustment. When the execution deviation is within the preset deviation tolerance range, the current execution strategy remains unchanged; when the execution deviation exceeds the preset deviation tolerance range, the output of the power actuator is adaptively fine-tuned first to suppress instantaneous load fluctuations caused by sudden terrain changes, while maintaining the steering actuator stable at the angle specified by the target steering angle parameters. This collaborative control design at the execution level allows the unmanned vehicle to continue moving towards the escape direction with minimal attitude disturbance even when encountering sudden terrain changes such as loosening obstacles or local ground subsidence during the escape process, until the straight-line distance between the current position coordinates and the starting position coordinates of the trapped position reaches the preset escape distance threshold, thus completing the entire execution chain from escape decision to successful escape.

[0067] Optionally, after controlling the unmanned vehicle to execute the escaping operation according to the escaping action command, the autonomous decision-making escaping method further includes: Obtain the current position coordinates and the starting position coordinates of the stranded vehicle; Calculate the straight-line distance between the current position coordinates and the starting position coordinates of the trapped location; If the straight-line distance does not reach the preset escape distance threshold, return to the step of obtaining the environmental state information of the environment where the unmanned vehicle is located and the unmanned vehicle's own operation information.

[0068] Preferably, after the unmanned vehicle executes the escape operation according to the escape action command, this application does not directly determine that the escape task is over, but instead initiates a post-verification of the escape effect and a closed-loop re-entry control phase. The core of this phase is to use spatial position measurement to quantify and verify the cumulative execution effect of the escape action, and use the verification result as the basis for determining whether to continue activating the reinforcement learning perception-decision-execution link. In unstructured trapped scenarios, the execution of a single escape action command may result in the unmanned vehicle not actually leaving the initial trapped area even though it has executed the escape action command, due to reasons such as local terrain collapse, obstacle displacement under force, or vehicle suspension interference. Therefore, introducing a re-entry mechanism based on position comparison can automatically trigger a new round of environmental state information and its own operation information acquisition when the unmanned vehicle has not completely escaped the trapped state, thereby driving the escape decision model to regenerate escape action commands adapted to the new situation, forming a strategy iteration driven by the escape progress.

[0069] Preferably, the processing of acquiring the current position coordinates of the unmanned vehicle is a synchronous position acquisition operation performed immediately after the completion of a single complete execution cycle of the escape action command. The unmanned vehicle, through its onboard positioning calculation unit, records the two-dimensional Cartesian coordinates obtained at this moment through fusion calculation by the global navigation satellite system receiver and the inertial measurement unit as its current position coordinates. These current position coordinates are a numerical pair consisting of an abscissa and a ordinate component, and their reference coordinate system is consistent with the coordinate system of the initial position coordinates when trapped. This coordinate system consistency ensures the comparability of subsequent straight-line distance calculations and avoids measurement distortion introduced by different coordinate references. When the availability of the positioning signal is reduced due to the terrain of the trapped area, such as canyons, dense forests, or building obstructions, the processing of acquiring the current position coordinates can supplement it with dead reckoning data from the wheeled odometer as a positioning interpolation source. The last reliable global positioning point and the dead reckoning increment are weighted and synthesized to maintain the continuous availability of the current position coordinates.

[0070] Preferably, while collecting and recording the current position coordinates, the starting position coordinates of the entrapment are retrieved from the trajectory record saved at the moment the entrapment status marker was first set. These starting position coordinates are a pair of two-dimensional Cartesian coordinate values ​​that are synchronously frozen and stored by the positioning and calculation unit at the moment the unmanned vehicle is detected to be in a entrapment state and a entrapment status confirmation signal is generated. The starting position coordinates remain unchanged throughout the subsequent escape process, serving as a fixed reference origin for measuring the escape progress. Before the entrapment marker is cleared, these starting position coordinates are continuously provided for post-evaluation verification of the escape effect and closed-loop re-entry control, and are not updated with the unmanned vehicle's tentative movements or local position changes, thus constituting a benchmark parameter anchored both temporally and spatially in the escape progress assessment.

[0071] Preferably, after obtaining the coordinates of the current position and the initial position of being trapped, a straight-line distance is calculated between them to obtain the escape assessment distance used to determine the degree of escape completion. The straight-line distance calculation is performed in two-dimensional Euclidean space. The difference between the x-coordinate component of the current position and the x-coordinate component of the initial position is squared, and the difference between the y-coordinate component of the current position and the y-coordinate component of the initial position is squared. The sum of these two squared values ​​is then taken as the square root, which is the escape assessment distance. This escape assessment distance numerically and intuitively reflects the net spatial deviation of the unmanned vehicle from its initial trapped point at the current moment. Unlike methods that use the total mileage of the vehicle's travel trajectory as the escape metric, using Euclidean straight-line distance as the assessment benchmark more directly reflects the core objective of the escape mission—physically freeing the unmanned vehicle from the trapped area—without imposing additional constraints on the detours the vehicle might take during the escape process.

[0072] Preferably, after calculating the escape assessment distance, it is compared numerically with a preset escape distance threshold to determine whether the escape task is considered complete. The preset escape distance threshold is a distance parameter selected based on the vehicle's dimensions, the typical spatial scale of the trapped scenario, and the task's requirements for thorough escape. For example, for a conventional off-road autonomous vehicle, it can be set to five meters, meaning that only when the vehicle's center of gravity is more than five meters away from the initial trapped point is it considered to have substantially escaped the core area of ​​the dense obstacle field that could lead to re-entry into a stuck state. When the escape assessment distance is greater than or equal to the preset escape distance threshold, the escape operation is deemed to have achieved the expected effect, and the autonomous vehicle has successfully escaped from the trapped area. At this point, the trapped state identifier is cleared, the reinforcement learning state space feature construction process for the trapped scenario is terminated, and the control mode is restored to the normal driving decision mode.

[0073] Preferably, when the escape assessment distance is less than the preset escape distance threshold, i.e., the straight-line distance has not yet reached the preset escape distance threshold, it is determined that the current escape operation has not achieved the expected effect. At this time, the trapped state marker is not cleared, but the step of returning to the execution state to obtain the environmental state information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operation information is triggered. Specifically, returning to the execution state means taking the current moment as the starting point of a new perception-decision-execution cycle, re-collecting obstacle distribution information of the current trapped site through the environmental perception module to update the environmental state information, re-collecting the current wheel speed and chassis attitude through on-board perception devices such as the chassis controller local area network bus and wheel speed sensors to update the vehicle's own operation information, and then reconstructing the reinforcement learning state space features of the current moment based on the updated environmental state information and the vehicle's own operation information. The newly constructed reinforcement learning state space features are input again into the escape decision model. Since the environmental state information and the vehicle's own operational information have incorporated the latest feedback—such as terrain changes, obstacle position shifts, and vehicle attitude adjustments—resulting from the previous escape action command execution, the escape decision model can re-infer based on these updated state features, generating the next set of escape action commands that matches the current situation. This return mechanism transforms the escape process from a single, open-loop action execution into a multi-stage, progressive escape decision chain driven by the escape assessment distance and possessing feedback correction capabilities.

[0074] Preferably, during multiple return executions, the numerical trend of the escape assessment distance can also be used as an additional metric to monitor the effectiveness of the escape strategy. When the escape assessment distance does not show a substantial increase compared to the previous return execution, or even decreases, it indicates that the action strategy favored by the current escape decision model may be stuck in strategic repetition or has reached the exploration boundary in the trapped terrain. At this time, a stochastic perturbation strategy on the input features of the escape decision model can be triggered. For example, a small Gaussian noise can be introduced into some components of the environmental feature vector in the reinforcement learning state space features to prompt the dual evaluation competition layer to explore candidate escape action command directions different from the current strategy in subsequent action generation. By combining the escape assessment distance, the preset escape distance threshold, and the feature perturbation strategy into an integrated escape continuous verification and strategy escape architecture, this application can automatically break out of the local repetition of the action strategy when the unmanned vehicle faces complex working conditions such as being unable to escape after multiple forward and backward movements following chassis bottoming out, and continuously advance towards the goal of achieving an escape assessment distance greater than or equal to the preset escape distance threshold.

[0075] like Figure 2 As shown in the figure, this is an embodiment of an autonomous decision-making and escape device for unmanned vehicles in distress scenarios based on reinforcement learning. The autonomous decision-making and escape device includes: The information acquisition module is used to acquire environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information when it is detected that the unmanned vehicle is in a distressed state. The feature construction module is used to construct the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information. An action decision module is used to input the reinforcement learning state space features into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The execution control module is used to control the unmanned vehicle to perform the traction removal operation according to the traction removal action command.

[0076] like Figure 3 As shown, a computer-readable storage medium is provided according to an embodiment of this application. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the autonomous decision-making escape method as described in any embodiment of this application.

[0077] like Figure 4 As shown in the figure, an unmanned vehicle according to an embodiment of this application includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to implement the steps of the autonomous decision-making and escape method as described in the embodiment of this application.

[0078] The above Figures 2-4 For a more detailed and exemplary explanation, please refer to the above. Figure 1 This will not be elaborated upon here.

Claims

1. A method for autonomous decision-making of an unmanned vehicle trapped in a scene based on reinforcement learning, characterized in that, The autonomous decision-making escape method includes: When the unmanned vehicle is detected to be in a distressed state, the environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information are obtained. Based on the environmental state information and its own operational information, construct the reinforcement learning state space features at the current moment; The reinforcement learning state space features are input into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The unmanned vehicle is controlled to perform an escape operation according to the escape action command.

2. The autonomous decision making and escape method according to claim 1, characterized in that, The following steps are used to detect if the unmanned vehicle is in a distressed state: Obtain the actual moving speed and power output parameters of the unmanned vehicle; Determine whether the power output parameter is greater than a preset power threshold, and determine whether the actual moving speed is continuously less than a preset speed threshold within a preset time window; When it is determined that the power output parameter is greater than the preset power threshold, and the actual moving speed is continuously less than the preset speed threshold within the preset time window, it is determined that the unmanned vehicle is in a trapped state.

3. The autonomous decision making and escape method of claim 1, wherein, The step of constructing the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information includes: Extract the relative distance and relative angle features of the target obstacle relative to the unmanned vehicle from the environmental state information; The wheel speed characteristics and chassis attitude characteristics of the unmanned vehicle are extracted from its own operating information; The relative distance feature, the relative angle feature, the wheel speed feature, and the chassis attitude feature are fused to obtain the reinforcement learning state space feature.

4. The autonomous decision making and escape method of claim 1, wherein, The escape decision model is trained through the following steps: An initial network model is constructed based on the deep reinforcement learning algorithm, and the network parameters of the initial network model are initialized. Control the virtual unmanned vehicle to make tentative movements in a simulated trapped scenario and collect the tentative state sequence; The trial state sequence is input into the initial network model for action value evaluation processing to obtain the corresponding evaluation result sequence; The exploratory movement is fed back and calculated according to a preset reward function to obtain a reward score sequence; The network parameters are updated using the reward score sequence and the evaluation result sequence until the network parameters meet the convergence condition. The converged initial network model is then determined as the escape decision model.

5. The autonomous decision making and escape method according to claim 4, characterized in that, The step of performing feedback calculations on the tentative movements according to a preset reward function to obtain a reward score sequence includes: Obtain the position offset and obstacle distance of the virtual unmanned vehicle after it performs an exploratory movement; Determine whether the distance to the obstacle is less than a preset safety threshold; When it is determined that the distance to the obstacle is less than the preset safety threshold, the negative incentive value corresponding to the collision penalty item is calculated based on the distance to the obstacle; When the position offset is determined to represent that the virtual unmanned vehicle is far away from the initial trapped point, the positive incentive value corresponding to the escape reward item is calculated according to the magnitude of the position offset. The negative incentive values ​​and the positive incentive values ​​are weighted and summed to obtain the reward scores in the reward score sequence.

6. The autonomous decision making and escape method of claim 1, wherein, The control of the unmanned vehicle to perform the escaping operation according to the escaping action command includes: Analyze the escape command and extract the target steering angle and longitudinal acceleration parameters; Based on the target steering angle parameters, the steering actuator of the unmanned vehicle is controlled to adjust the steering. Based on the target longitudinal acceleration parameters, the power actuator of the unmanned vehicle is controlled to output power.

7. The autonomous decision making and escape method of claim 1, wherein, After the unmanned vehicle is controlled to execute the escape operation according to the escape action command, the autonomous decision-making escape method further includes: Obtain the current position coordinates and the starting position coordinates of the stranded vehicle; Calculate the straight-line distance between the current position coordinates and the starting position coordinates of the trapped location; If the straight-line distance does not reach the preset escape distance threshold, return to the step of obtaining the environmental state information of the environment where the unmanned vehicle is located and the unmanned vehicle's own operation information.

8. A device for autonomous decision-making of an unmanned vehicle trapped in a scene based on reinforcement learning, characterized in that, The autonomous decision-making escape device includes: The information acquisition module is used to acquire environmental status information of the environment in which the unmanned vehicle is located and the unmanned vehicle's own operating information when it is detected that the unmanned vehicle is in a distressed state. The feature construction module is used to construct the reinforcement learning state space features at the current moment based on the environmental state information and the self-operation information. An action decision module is used to input the reinforcement learning state space features into a pre-trained obstacle avoidance decision model to determine the obstacle avoidance action command of the unmanned vehicle; wherein, the obstacle avoidance decision model is trained based on a deep reinforcement learning algorithm; The execution control module is used to control the unmanned vehicle to perform the traction removal operation according to the traction removal action command.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the autonomous decision-making escape method as described in any one of claims 1 to 7.

10. An unmanned vehicle, characterized in that It includes a memory and a processor, the memory storing a computer program, the processor running the computer program to implement the steps of the autonomous decision-making escape method according to any one of claims 1 to 7.

Images