A reinforcement learning method for navigation safety in mobile robots

Abstract

This invention provides a mobile robot navigation safety reinforcement learning method, belonging to the field of robot navigation technology based on reinforcement learning. It combines a safety reinforcement learning cost generation method using a large language model, aiming to solve the problem of high difficulty and time consumption in designing constraint mathematical cost functions for safety reinforcement learning in high-risk real-world domains. This invention addresses this through two core modules: a dangerous state action extraction network filters key unsafe data for input into a large language model, reducing interference and time consumption; a cost prediction module transforms the single and multi-constraint evaluation results of the large language model into quantified costs, and fuses the costs and reward functions using the Lagrange multiplier method for policy learning. This method enables mobile robots to learn safe policies, achieving task performance comparable to traditional safety reinforcement learning, while significantly reducing the threshold for cost function design, improving learning efficiency and safety, and can be widely applied to high-risk, resource-constrained safety reinforcement learning scenarios.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot navigation based on reinforcement learning, and particularly relates to a mobile robot navigation safety reinforcement learning method. BACKGROUND

[0002] In recent years, deep reinforcement learning (DRL) has achieved remarkable application results in multiple technical fields, and can be effectively applied to robot control, automated manufacturing, recommendation systems, and whole-body control of humanoid robots, etc., providing technical support for intelligent decision-making and execution of complex tasks. In the field of robot control, deep reinforcement learning learns the optimal strategy by allowing the agent to interact with the environment, showing the potential to solve dynamic and complex problems; in automated manufacturing and recommendation systems, it can continuously optimize operation processes and recommendation logic based on data feedback, improving the efficiency of scene applications; in high-precision scenarios such as humanoid robot control, it also provides a feasible path for realizing the control of robot agile movement.

[0003] The first challenge faced by deep reinforcement learning in penetrating into real-world scenarios is the safety risk in the exploration phase. The agent needs to continuously interact with the environment to explore in order to learn the optimal strategy, but in a real physical environment, this unbounded exploration can easily lead to dangerous behavior, such as a robot colliding with surrounding equipment or personnel during work, or an autonomous driving agent deviating from a safe route and entering a dangerous area, which not only can cause equipment damage or personnel injury, but also can force the exploration process to be interrupted, further prolonging the training cycle.

[0004] To solve this problem, safe reinforcement learning (Safe RL) is introduced, which adds safety constraints to the reinforcement learning process to regulate the exploration behavior of the agent and reduce the probability of risk events. However, existing safe reinforcement learning still has key technical bottlenecks, with the core problem being the difficulty of designing a mathematical cost function for task constraints: not only does it require domain experts to invest a lot of time in manual modeling, but it is also difficult to cover complex scenarios with multiple constraints coexisting; and the manually designed cost function is easily influenced by subjective experience, making it difficult to accurately match the needs of dynamically changing scenarios, resulting in limited adaptability and practicality of safe reinforcement learning, which cannot efficiently support the intelligent control needs of real-world high-risk scenarios such as robot navigation and autonomous driving.

[0005] Compared to traditional manually designed constraint equations, describing safety constraints in natural language has significant advantages: on the one hand, it conforms to human expression habits, and ordinary technical personnel can define constraints through intuitive expressions such as "avoid colliding with obstacles", without the need for experts to translate complex mathematical formulas, greatly reducing the technical threshold and time cost; on the other hand, it is more flexible and can easily cover single-scenario simple constraints or multi-scenario superimposed constraints (such as "maintain a safe distance from obstacles and control speed"), and can be quickly adjusted as the scene changes without the need to reconstruct mathematical models.

[0006] The problems and defects of the prior art are:

[0007] (1) High safety risk in traditional deep reinforcement learning exploration process: it relies on the unbound interaction between the agent and the environment to collect samples, which may easily lead to dangerous behaviors such as collision and entering dangerous areas in real-world scenarios such as robot navigation and autonomous driving; such behaviors not only threaten safety and damage equipment, but also interrupt sample collection and prolong the training cycle, making it difficult for traditional deep reinforcement learning to be directly applied to high-risk scenarios.

[0008] (2) High cost and high threshold for mathematical safety constraints: existing safe reinforcement learning needs to convert the safety requirements of the scene into a mathematical cost function, which relies on domain experts to invest a lot of time in manual modeling, not only consuming high labor costs, but also requiring high mathematical modeling skills from technical personnel, making it difficult for ordinary scenarios to be quickly applied.

[0009] (3) Poor adaptability to multi-constraint scenarios: artificially designed mathematical cost functions are difficult to cover complex scenarios with multiple constraints coexisting, such as simultaneously meeting the requirements of "avoiding collision with obstacles", "controlling driving speed", and "staying away from dangerous areas"; function parameters need to be repeatedly adjusted, and constraint conflicts are likely to occur, making it difficult to efficiently adapt to dynamically changing scene safety requirements.

[0010] (4) Natural language constraints cannot be directly utilized: in real-world scenarios, humans are more accustomed to describing safety constraints in natural language such as "avoid approaching obstacles" and "do not enter dangerous areas", but existing technologies lack the ability to directly convert natural language constraints into cost information usable for safe reinforcement learning, making it difficult for the flexibility and intuitiveness of natural language to play a role, further limiting the scene adaptation range of safe reinforcement learning. SUMMARY

[0011] To solve the above problems, the present application proposes a mobile robot navigation safe reinforcement learning method (SAC-LLMC) that supports language-to-cost automatic conversion. This method is applied to the field of mobile robot navigation safety learning, combining the natural language understanding capabilities of large language models (LLMs) with the policy learning needs of safe reinforcement learning (Safe RL), and specifically includes the following steps:

[0012] Construct a cost generation system, which includes a dangerous state-action extraction network and a cost prediction module, and initialize the parameters of the dangerous state-action extraction network, the cost prediction module, and the agent policy network of safe reinforcement learning;

[0013] The dangerous state-action extraction network filters the state and action data in the mobile robot navigation scene and extracts state-action pairs with potential dangers, which are input into the large language model;

[0014] The large language model outputs a corresponding evaluation result based on an input dangerous state-action pair;

[0015] The cost prediction module receives the evaluation result and converts it into a quantified constraint cost value;

[0016] The quantified constraint cost value is fused with a reward function of safe reinforcement learning through the Lagrange multiplier method to generate a policy update loss function;

[0017] The agent policy network optimizes parameters based on the policy update loss function to guide the mobile robot to learn a navigation safety policy.

[0018] Preferably, the dangerous state-action extraction network is a binary classifier, and its training state-action pair data It is automatically collected during the exploration process of the agent in the unconstrained simulation environment, wherein represents the state of the agent at time , and represents the action taken by the agent at time ; the dangerous state-action extraction network takes the state-action pair of the agent as input and labels the input state-action pair as safe or unsafe, wherein if a dangerous situation occurs within the first ten actions of the agent starting from a specific state, the state-action pair is labeled as unsafe, otherwise it is labeled as safe; after training, the dangerous state-action extraction network is used to determine whether the current state-action pair of the agent is safe, and only the state-action pair classified as unsafe is input to the large language model (LLMs) for evaluation.

[0019] Preferably, the new state-action pair data generated during the policy training process of the agent will be used for further optimization training of the dangerous state-action extraction network to improve the accuracy of the network in judging the safety of the state-action pair.

[0020] Preferably, the cost prediction module realizes the cost prediction of unsafe state-action pairs by fine-tuning a pre-trained BERT model; the training process of the cost prediction module includes: based on the preset key phrases (including safety, distance from obstacles, distance from dangerous areas, etc.), the evaluation text generated by the large language model (LLMs) for each agent state-action pair is labeled, where 1 is labeled when it completely matches the key phrase, and 0 is labeled when it does not match, forming a semantic matching dataset; the pre-trained BERT model is fine-tuned using the semantic matching dataset, so that the fine-tuned BERT model can more accurately predict the matching probability of the evaluation text generated by the large language model (LLMs) and the preset key phrase; when predicting the cost, if the matching probability of each key phrase predicted by the fine-tuned BERT model is higher than the preset threshold (such as 0.8), the agent state-action pair is assigned a cost value corresponding to the corresponding key phrase; in a multi-constraint scene, the final cost of the agent state-action pair is the weighted sum of the cost values generated by the matching of the evaluation text generated by the large language model (LLMs) under each constraint and the corresponding preset key phrase.

[0021] Preferably, when the cost value output by the cost prediction module is fused with the reward function of the safety reinforcement learning by the Lagrange multiplier method, the fusion process adjusts the weight coefficient of the cost value based on the safety constraint priority; wherein the safety constraint priority is set according to the specific application scene (such as medical operation, automatic driving, robot operation), in a multi-constraint obstacle avoidance task, for high-priority safety constraints (such as avoiding collision with objects), the weight coefficient of the corresponding cost value in the fusion process is 0.6-0.8, and for low-priority safety constraints (such as avoiding areas with no collision but potential danger such as potholes, damaged roads, etc.), the weight coefficient of the corresponding cost value in the fusion process is 0.2-0.4; the result after fusion is used for policy learning of the agent, so that the agent prioritizes high-priority safety constraints while considering low-priority safety constraints during learning, and finally generates a safety policy that meets the multi-constraint conditions.

[0022] Preferably, the set of preset key phrases can be dynamically expanded according to specific navigation tasks, and new phrases (such as "safe distance from pedestrians" and "avoid dynamic obstacles") need to be supplemented with semantic matching data by manual labeling to adapt to the constraint requirements of different scenes.

[0023] Preferably, to maximize the expected reward while meeting the safety constraints, the agent policy network optimizes the parameters by maximizing the following defined Lagrange objective function , specifically:

[0024]

[0025]

[0026]

[0027] wherein, is a reward function, is a policy entropy term, is a non-negative weight, is an action policy of the mobile robot in state at time step t, is an action taken by the mobile robot in state at time step t, is a cost, is a discount factor, is a total time step, is an upper bound on the cumulative cost over total time steps , is a Lagrangian function, is a reward-entropy objective term for the policy, is a cumulative cost constraint term for the policy.

[0028] Preferably, the agent policy network parameters are stored with experience replay mechanism to store history state-action-reward-cost data during the optimization process, and a batch of data is randomly sampled for each parameter update.

[0029] Preferably, to avoid damage to the mobile robot during navigation, the method imposes safety constraints on the robot action at each time step. The optimization objective is to maximize the expected cumulative reward defined as follows, specifically:

[0030]

[0031] wherein, is a reward function, is a policy entropy term, is a non-negative weight, and the optimized policy must satisfy the following safety constraints at each time step

[0032] ,

[0033] wherein is an upper bound on the constraint violation cost at each time step , is an action policy of the mobile robot in state at time step t, is an action taken by the mobile robot in state at time step t, is a predicted cost of a state-action pair output by the cost prediction module,​ is a discount factor, is a total time step.

[0034] Preferably, the control strategy is trained in the simulation environment in the interactive training step, and the SAC-Lag algorithm is used to train the safety strategy; the core of the SAC-Lag algorithm is: first, a large language model (LLM) is used to generate an evaluation text for a robot dangerous state-action pair, and then a cost prediction module is used to convert the evaluation text into a quantitative constraint cost, and then the quantitative constraint cost is fused and optimized with the task reward by adaptively adjusting the Lagrange coefficient, so as to maximize the navigation benefit while strictly controlling the constraint violation and ensuring the safety of the strategy.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1. Breakthrough of the traditional safety reinforcement learning "constraint mathematical" bottleneck, realize the automatic conversion of language constraints to quantitative cost: avoid the dependence on the field experts to manually convert the safety requirements into complex mathematical cost functions.

[0037] 2. Accurate screening of unsafe state-action pairs, reducing the evaluation burden and error of large language models: directly let LLMs evaluate all state-action pairs, which is easy to cause low evaluation efficiency and large fluctuation of results due to too large data volume.

[0038] 3. Optimization of the generation process of the cost value, so that the cost value is more accurate: directly using large language to realize the mapping of state-action pairs to cost value has great instability, and using the cost prediction module to realize the mapping of large language model evaluation to cost value can increase the accuracy and stability of the generated cost value.

[0039] 4. Efficiently adapt to multi-constraint complex scenarios: for multi-constraint scenarios, only need to design multiple natural language constraints and set different prompt words for the large language model, avoiding the complex operation of designing multiple constraint functions. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall method flowchart of the present application.

[0041] Figure 2 is a schematic diagram of the first task of the three tasks in embodiment 2 of the present application, and the task one requires the mobile robot to keep a safe distance from the obstacle.

[0042] Figure 3 is a schematic diagram of the second task of the three tasks in embodiment 2 of the present application, and the task two requires the mobile robot to avoid the dangerous area.

[0043] Figure 4is a schematic diagram of the third task in three tasks in embodiment 2 of the present application, task three requires the mobile robot to keep a safe distance from the obstacle while avoiding the dangerous area.

[0044] Figure 5 is the reward comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task one in embodiment 2 of the present application.

[0045] Figure 6 is the cost comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task one in embodiment 2 of the present application.

[0046] is the reward comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task two in embodiment 2 of the present application.

[0047] is the cost comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task two in embodiment 2 of the present application.

[0048] Figure 9 is the reward comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task three in embodiment 2 of the present application.

[0049] Figure 10 is the cost comparison curve of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in task three in embodiment 2 of the present application.

[0050] Figure 11 is the navigation execution path schematic diagram of the mobile robot corresponding to the final convergence strategy of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in the task one in embodiment 2 of the present application.

[0051] Figure 12 is the navigation execution path schematic diagram of the mobile robot corresponding to the final convergence strategy of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in the task two in embodiment 2 of the present application.

[0052] Figure 13 is the navigation execution path schematic diagram of the mobile robot corresponding to the final convergence strategy of the SAC-LLMC algorithm of the present application and the SAC, SAC-Lag and PPO-CP algorithms in the task three in embodiment 2 of the present application.

[0053] Figure 14For the second scene of the third task of embodiment 2 of the present application, the SAC-LLMC algorithm of the present application and the final convergence strategy of the SAC, SAC-Lag and PPO-CP algorithms correspond to the mobile robot navigation execution path diagram. DETAILED DESCRIPTION

[0054] Embodiment 1

[0055] As shown in the flow chart of the mobile robot navigation safety reinforcement learning method provided by the present application for supporting automatic conversion of language to cost. Figure 1

[0056] Before starting the mobile robot navigation safety reinforcement learning training, the natural language constraint needs to be designed according to the safety requirements and scene characteristics of the current navigation task: a single constraint task corresponds to a single accurate constraint, and multiple constraint tasks are designed simultaneously multiple sets of independent and full-scene requirement covering constraints to ensure complete mapping of the task safety boundary. At the same time, the prompt word engineering is carried out for the constraint, the task scene background, the evaluation requirement definition and the output format specification are supplemented in the prompt word, the association cognition of the natural language constraint and the task scene is established by the large language model, and the subsequent evaluation results of the robot state-action pair are ensured to meet the safety requirements, have rationality and accuracy.

[0057] At the same time, the present application pre-trains a dangerous state-action extraction network, which is a binary classification network, taking the state-action pair of the agent as input and taking whether the state-action pair is safe as output, wherein is the state of the agent in the environment at time , and is the action taken in this state. The present application will train this network in an unconstrained simulation environment, if none of the constraints are violated after reaching in the 10 time steps, then is marked as safe, otherwise it is marked as unsafe. The present application will use the data automatically collected in the unconstrained simulation environment as the training set of the network to train the dangerous state-action extraction network.

[0058] When pre-training the dangerous state-action extraction network, the cross-entropy loss function is used as the optimization objective of network parameter update, and the specific expression is as follows:

[0059]

[0060] wherein, is the total number of training samples, is the true label (1 for safe and 0 for unsafe) of the th state-action pair .​ the prediction output of the network for the first sample (i.e. the probability of being safe). By minimizing the deviation between the network prediction value and the true label through the loss function, the parameter optimization of the dangerous state-action extraction network is realized, and the classification accuracy of the safety of the state-action pair is improved.

[0061] After the training of the dangerous state-action extraction network is completed, the safety of the state-action pair generated by the mobile robot during the navigation process is first determined by the network: if the determination result is unsafe, the unsafe state-action pair is input into the large language model (LLMs) optimized by the prompt engineering, and the large language model (LLMs) generates the corresponding evaluation text of the dangerous degree of the unsafe state-action pair in combination with the natural language constraint designed before training (including multiple sets of natural language constraints under multiple constraint tasks).

[0062] Meanwhile, the fine-tuning cost prediction module of the application is used to quantify the cost of the state-action pair evaluated by the large language model; the cost prediction module is based on the pre-trained BERT model, which is fine-tuned through the self-collected semantic matching data set; the construction method of the semantic matching data set is: for the evaluation text generated by the large language model (LLMs) for each mobile robot state-action pair, the preset key phrase (including safety, proximity or distance to obstacles, proximity or distance to dangerous areas, etc.) is labeled, if the evaluation text and the preset key phrase are completely matched (such as matching with "proximity to obstacles"), it is labeled as 1, if not matched, it is labeled as 0; the fine-tuned BERT model is used to predict the matching probability of the evaluation text generated by the large language model (LLMs) and each preset key phrase, when the matching probability is higher than the preset threshold (such as 0.8 in the experiment), the corresponding cost value of the preset key phrase is assigned to the corresponding mobile robot state-action pair, that is, the cost is assigned according to the following cost calculation rule:

[0063]

[0064] If there are multiple sets of natural language constraints for the mobile robot navigation task, the final cost of the mobile robot state-action pair calculated by the cost prediction module is the weighted sum of the cost values generated by the evaluation text generated by the large language model (LLMs) under each set of constraints and the corresponding preset key phrase.

[0065] ​After the cost quantification of unsafe state-action pairs is completed, the SAC-LLMC algorithm is used to fuse the reward function and the cost value output by the cost prediction module, so as to realize the optimization training of the mobile robot navigation safety strategy; the SAC-LLMC algorithm is different from the traditional safety reinforcement learning method based on Lagrange, does not depend on the constraint cost function, only takes the predicted cost output by the cost prediction module as the core optimization basis, and is improved on the basis of the Soft Actor-Critic (SAC) algorithm combined with the Lagrange multiplier method.

[0066] Specifically, in order to avoid the mobile robot from being damaged in navigation, a safety constraint is imposed on the robot action at each time step, and the corresponding constraint optimization objective function is set to maximize the following expression:

[0067]

[0068] wherein, is a reward function, is a policy entropy term, is a non-negative weight (used to balance the reward and the entropy term), and the above formula satisfies wherein is an upper limit of the constraint violation cost at each time step , is an action policy of the mobile robot in a state , is an action taken by the mobile robot in a state , is a predicted cost of a state-action pair output by the cost prediction module, is a discount factor, is a total time step. At the same time, two independent critic networks are set: one is a cumulative discounted reward network used to fit the cumulative discounted reward value; the other is a constraint violation excess cost network used to fit the excess cost value of constraint violation; in the constraint Markov decision process (CMDP) optimization, an adaptive Lagrange multiplier

[0069] is introduced, and the above constraint objective function is converted into an unconstrained maximum-minimum Lagrange function:

[0070]

[0071]

[0072]

[0073] wherein, ​​​is the upper bound of cumulative cost in total time steps, is the Lagrangian function, is the reward-entropy objective term of the policy, is the cumulative cost constraint term of the policy.

[0074] Since the above unconstrained Lagrangian function needs to be approximated by a neural network , there is no analytical solution, so the gradient descent method is used to update the Lagrange multiplier and the policy network , i.e. the output of , the update of the Lagrange multiplier is based on the following loss function:

[0075]

[0076] where is the experience replay buffer (stores state-action pair data generated during robot navigation); when the total time step output by the cost prediction module exceeds , the is increased to enhance the constraint strength; when the current policy meets the cost constraint, is reduced to balance the reward acquisition and constraint satisfaction;

[0077] For each fixed , the policy network is updated by minimizing the following loss function:

[0078]

[0079] where is a non-negative weight parameter (used to control the randomness of the policy); through the above optimization, the mobile robot navigation safety policy that meets the natural language safety constraint is finally obtained.

[0080] Embodiment 2:

[0081] The technical solutions and technical effects of the present application will be further described below in combination with specific experiments.

[0082] ​​​To verify the potential effectiveness of the mobile robot navigation safety reinforcement learning method supporting automatic conversion of language to cost proposed in the application, the embodiment carries out experiments in three task scenarios of the mobile robot, which are respectively "obstacle avoidance" single constraint navigation scenario, "dangerous area avoidance" single constraint navigation scenario and "obstacle avoidance + dangerous area avoidance" multi-constraint navigation scenario. Through the design of scenarios with different constraints, the applicability and effectiveness of the method in various navigation scenarios are comprehensively verified.

[0083] (1) Experimental task: the embodiment involves three mobile robot related tasks, and the schematic diagrams of the tasks are respectively Figures 2 to 4 .

[0084] Single constraint task one obstacle avoidance: in this task, a car needs to travel from a fixed starting point to a fixed end point, and needs to avoid obstacles in the process. The constraint condition "maintain a safe distance from the obstacle to avoid collision" is conveyed through natural language.

[0085] Single constraint task two dangerous area avoidance: in this task, the car needs to travel from a fixed starting point to a fixed end point, and needs to complete the obstacle avoidance operation in the process; at the same time, the constraint condition "prohibit access to dangerous areas" is conveyed through natural language to limit the navigation safety range of the car.

[0086] Multi-constraint task complex obstacle avoidance with dangerous area: in this task, the car needs to travel from a random starting point to a random end point, and needs to complete the obstacle avoidance operation in the process; at the same time, two constraint conditions are conveyed through natural language: "maintain a safe distance from the obstacle to avoid collision" and "prohibit access to dangerous areas", to limit the navigation safety boundary of the car in complex scenarios.

[0087] (2) Experimental setup: Four different agents were tested for each task in this example, divided into two categories: one is the basic (safety) reinforcement learning agent, including the standard SAC agent and SAC-Lag agent; the other is the improved agent proposed by the present application, namely the SAC-LLMC agent (as the core test object) and the comparative variant PPO-CP based on its architecture. All agents use a fully connected neural network (FCN) as a function approximator: the structure of the policy network (actor) is [128, 64, 64, 32, 2], and the structure of the critic network is [128, 64, 64, 1]; the hidden layer uses a Relu activation function, and the output layer of the critic network uses a linear activation. Among them, the SAC agent contains 1 policy network and 2 reward critic networks; the SAC-Lag and SAC-LLMC agents retain 1 policy network and add 1 constraint critic network to the 2 reward critic networks for processing safety constraint signals. Both the policy network and the critic network use a dynamic learning rate λ, with an initial value of 0.001, which is attenuated to 1 / 4 of the current value every 50 training rounds. For SAC-Lag and SAC-LLMC agents, the initial value of the Lagrange multiplier β is 0.001, and the Adam optimizer is used with an initial learning rate δ of 0.025, which is attenuated to 0.25 times the current value every 50 training rounds. In the three tasks, the vehicle can perform forward, backward, left turn, and right turn actions, with a forward / backward speed range of [0, 2] and a left / right angular velocity range of [0, 2]. The vehicle state in the three tasks includes speed, angular velocity, position, and distance to the nearest obstacle; in tasks two and three, the proximity to the dangerous area is indicated by 1 (close) and 0 (not close).

[0088] (3) Experimental results: To verify the potential effectiveness of the SAC-LLMC method proposed by the present application, the learning performance of the four agents (SAC, SAC-Lag, PPO-CP, and SAC-LLMC) in different tasks is compared and analyzed.

[0089] In terms of learning process performance comparison, Figures 5 to 10 The learning reward curves and cost curves of the four agents in the three tasks are presented respectively: the vertical coordinate represents the cumulative reward value (reflecting task completion efficiency) or cumulative cost value (reflecting safety constraint satisfaction degree) of the agent in the target navigation environment for a single round, and the horizontal coordinate represents the number of training rounds (reflecting the learning iteration process); to ensure the reliability and statistical significance of the experimental results, the curve data of task one and task two are obtained by averaging the values of 10 independent repeated experiments, and the curve data of task three are obtained by averaging the values of 5 independent repeated experiments to reduce the interference of random errors on the results.

[0090] In terms of final policy effect verification,Figures 11 to 14 Further, the route test results of the final convergence strategies of the four agents in the three tasks are compared: by intuitively presenting the execution paths of each agent in the actual navigation scene, the comprehensive performance of different algorithms in meeting safety constraints (such as obstacle avoidance and keeping away from dangerous areas) and achieving navigation goals (such as from the starting point to the end point) can be clearly observed.

[0091] In combination with FIG. 5 (reward curve of task one), FIG. 6 (reward curve of task one), and FIG. 11 (route test diagram of the final strategy of task one), the performance comparison and analysis of the SAC-LLMC method of the present application and the SAC, SAC-Lag, and PPO-CP algorithms in the prior art are as follows: compared with the SAC algorithm, although the reward learning of SAC-LLMC is slightly slower, the constraint learning ability is significantly better, the cumulative generation value of SAC is high and fluctuates greatly, Figure 11 the path of the middle is close to the obstacle and does not master the safety constraint, while the cumulative generation value of SAC-LLMC is low and stable, and the path always keeps a safe distance from the obstacle; compared with the SAC-Lag algorithm, the reward acquisition and constraint satisfaction performance of the two are comparable, but SAC-Lag artificially designs the constraint function, and SAC-LLMC replaces this design by using natural language constraints and a large language model evaluation, reducing the application threshold and adaptation cost; compared with the PPO-CP algorithm, SAC-LLMC is better in reward and constraint learning, the cumulative reward converges faster, the cumulative generation value decreases faster and fluctuates less, and it can more efficiently balance the task and safety constraints. In summary, SAC-LLMC has three major core advantages: first, constraint learning is effective and can stably master safety constraints, overcoming the lack of constraints in SAC; second, the design cost is low, and there is no need for artificial design of the constraint function, which can be adapted to the scene through natural language constraints; third, the performance balance is better, and it is better than PPO-CP in reward efficiency and constraint stability, and more efficiently achieves the navigation and safety goals.

[0092] In combination with Figure 7 (reward curve of task two), Figure 8 (reward curve of task two), and Figure 12 (route test diagram of the final strategy of task two), the methods of task two and task one are basically the same, and the core difference is that the safety constraint is replaced by "do not approach dangerous areas". In combination with the corresponding reward curve and route test diagram analysis: the SAC algorithm still does not master the constraint, the path crosses the dangerous area, and the cumulative generation value is high; SAC-Lag can meet the constraint, but it needs to artificially design the constraint function; PPO-CP lacks stability in reward convergence and constraint satisfaction; and SAC-LLMC does not need to design additional constraints, but can stably avoid dangerous areas through natural language constraints, with a low cumulative generation value and a reward convergence efficiency close to SAC-Lag, and overall performance better than SAC and PPO-CP, continuing the core advantages in task one.

[0093] In combination Figure 9 (task three reward curve), Figure 10 (task three reward curve) and Figure 13 、 Figure 14 (task three final strategy route test chart), task three is a multi-constraint scene, on the basis of tasks one and two, the double constraints of "keeping a safe distance from obstacles" and "not approaching dangerous areas" are superimposed, and the starting point and the end point of the car are set to be random, and the complexity of the scene is improved. According to the analysis of the corresponding experimental charts: SAC algorithm has no constraint learning ability, the path is close to the obstacle and crosses the dangerous area, and the cumulative generation value is significantly high; SAC-Lag needs to be artificially designed for double constraints, and the adaptation cost is high; PPO-CP has further insufficient stability under double constraints, the cumulative reward fluctuates greatly, and the generation value decreases slowly; and SAC-LLMC does not need to be designed for constraints, and can adapt to double constraints through natural language only, and can still avoid risks under random starting and ending points, with low cumulative generation value and efficient reward convergence. In complex multi-constraint scenes, it still continues to have core advantages, highlighting stronger scene adaptability and robustness.

[0094] Looking at the three tasks, SAC-LLMC shows significant universal advantages: in single-constraint (obstacle avoidance / dangerous area avoidance) tasks, without manually designing constraint functions, it can meet the constraints through natural language only, and its performance is better than that of SAC which has no constraints and PPO-CP which has insufficient stability; in complex tasks with multiple constraints and random starting and ending points, it can still efficiently balance the double constraints, avoid the high adaptation cost problem of SAC-Lag, and always maintain low cumulative generation value and efficient reward convergence, highlighting strong scene adaptability, robustness and practical value.

[0095] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0096] Although the specific embodiments of the present application have been described above, they are not intended to limit the scope of protection of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A method for mobile robot navigation safety reinforcement learning, characterized in that, The method comprises the following steps: Construct a cost generation system including a dangerous state-action extraction network and a cost prediction module; initialize the parameters of the dangerous state-action extraction network, the cost prediction module and the agent policy network of the safety reinforcement learning; The dangerous state-action extraction network filters state and action data in a mobile robot navigation scene, extracts state-action pairs with potential danger, and inputs them into a large language model. The dangerous state-action extraction network is a binary classifier, and the state-action pair data is trained automatically collected by the agent during exploration in the unconstrained simulation environment, wherein represents the state of the agent at time in the environment, represents the action taken by the agent at time in the environment; the dangerous state-action extraction network takes the state-action pair of the agent as input, and labels the input state-action pair as safe or unsafe, wherein if the agent violates the constraint within the first ten actions from a specific state, the state-action pair is labeled as unsafe, otherwise it is labeled as safe; after training, the dangerous state-action extraction network is used to determine whether the current state-action pair of the agent is safe, and only the state-action pairs classified as unsafe are input into the large language model for evaluation; The large language model outputs the corresponding evaluation results based on the input dangerous state-action pair; The cost prediction module receives the evaluation results and converts them into quantified constraint cost values; specifically, during the training of the cost prediction module, based on the pre-set key phrases, the evaluation text generated by the large language model for each agent state-action pair is labeled, wherein a complete match with the key phrase is labeled as 1, and a non-match is labeled as 0, forming a semantic matching data set; the pre-trained BERT model is fine-tuned using the semantic matching data set, so that the fine-tuned BERT model can more accurately predict the matching probability of the evaluation text generated by the large language model and the pre-set key phrases; during the cost prediction, if the matching probability predicted by the fine-tuned BERT model with each key phrase is higher than a pre-set threshold, the agent state-action pair is assigned a cost value corresponding to the corresponding key phrase; in a multi-constraint scenario, the final cost of the agent state-action pair is the weighted sum of the cost values generated by the evaluation text generated by the large language model under each constraint and the corresponding pre-set key phrases; The quantified constraint cost value and the reward function are fused by the Lagrange multiplier method to generate a target function for policy updating; The agent policy network updates the parameters based on the target function to guide the mobile robot to learn the navigation safety policy.

2. The method of claim 1, wherein: The new state-action pair data generated during the policy training process of the agent will be used to further optimize the training of the dangerous state-action extraction network to improve the accuracy of the network in judging the safety of the state-action pair.

3. The method of claim 1, wherein: The quantified constraint cost value and the reward function of the safety reinforcement learning are fused by the Lagrange multiplier method, and the weight coefficient of the constraint cost value is adjusted based on the safety constraint priority; wherein the safety constraint priority is set according to the specific application scenario, and in a multi-constraint obstacle avoidance task, the weight coefficient of the corresponding constraint cost value in the fusion is 0.6-0.8 for high-priority safety constraints, and 0.2-0.4 for low-priority safety constraints; the fused result is used for agent policy learning, so that the agent prioritizes high-priority safety constraints while considering low-priority safety constraints during the learning process, and finally generates a safety policy that meets the multi-constraint conditions.

4. The method of claim 1, wherein: The set of pre-set key phrases is dynamically expanded according to the specific navigation task, and new phrases need to be supplemented with semantic matching data through manual labeling to adapt to the constraint requirements of different scenarios.

5. The method of claim 1, wherein: To maximize the expected reward while satisfying the safety constraints, the agent policy network updates parameters by maximizing a Lagrangian objective function defined as follows Specifically, the agent policy network updates parameters by maximizing a Lagrangian objective function defined as follows wherein, is a reward function, is a policy entropy term, is a non-negative weight, is an action policy of the mobile robot in state at time step t, is an action taken by the mobile robot in state at time step t, is a cost, is a discount factor, is a total time step, is an upper bound on the cumulative cost over total time steps t, is a Lagrangian objective function, is a reward-entropy objective term for the policy, is a cumulative cost constraint term for the policy.

6. The method of claim 1, wherein: During the optimization process, the agent policy network parameters use the experience replay mechanism to store historical state-action-reward-cost data, and randomly sample batch data for each parameter update.

7. The method of claim 1, wherein: To avoid damage to the mobile robot during navigation, safety constraints are imposed on the robot's actions at each time step, and the optimization goal is to maximize the expected cumulative reward defined as follows: where, is the reward function, is the policy entropy term, is a non-negative weight, while the optimized policy must satisfy the following safety constraints at each time step ​ ， wherein is an upper bound on the constraint violation cost for each time step , is the action policy of the mobile robot in state at time step is the action taken by the mobile robot in state at time step is the predicted cost of state-action pair output by the cost prediction module, is a discount factor, is the total time step.

Images