SRL-based intelligent networked automobile function degradation decision-making method and system

Abstract

The invention discloses an SRL-based intelligent networked automobile function degradation decision-making method and system. The method comprises the following steps: step 1, acquiring environment information and monitoring a trigger event; 2, grading the system risk assessment indexes of the trigger event monitored in the step 1, determining a corresponding automobile safety integrity grade according to an actual driving mode, and obtaining the system risk of the automobile; 3, according to the environment information collected in the step 1, integrating the risk fields of the static obstacles, the dynamic obstacles and the road boundaries, and utilizing a driving risk field model to obtain the driving risk of the vehicle; 4, describing the function degradation decision-making process of the intelligent networked automobile through a CMDP model to obtain an optimal function degradation strategy; wherein the state space S and the cost function of the CMDP model both comprise the system risk in the step 2 and the driving risk in the step 3. According to the method, efficient, safe and stable function degradation decision making of the intelligent network connection automobile in a complex and changeable scene can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent networked vehicles, and in particular to a method and system for making decisions on function degradation of intelligent networked vehicles based on SRL (Safety Reinforcement Learning). Background Art

[0002] Functional degradation of intelligent connected vehicles means that in certain circumstances, when an intelligent connected vehicle cannot normally perform all its autonomous driving functions, it automatically enters a low-level functional mode to ensure the safety of the vehicle and passengers. The core of the functional degradation design is that in abnormal or emergency situations, the system can automatically adjust the driving mode according to the current operating risks to give priority to meeting basic safety needs.

[0003] The mainstream function degradation decision-making method is usually based on rule logic. Its principle is to monitor the vehicle's operating status, environmental information and system health status in real time. When potential risks or system abnormalities are detected, it automatically switches to a low-level driving mode according to preset rules, such as manual takeover, emergency braking, etc. Typical degradation scenarios include sensor failure, communication interruption, actuator abnormality and extreme environment. However, this rule-based logic method has limitations such as insufficient adaptability, conservative decision-making, limited risk balance ability and difficulty in processing high-dimensional data. Rule logic relies too much on preset conditions and is difficult to deal with abnormalities in unknown or complex scenarios. At the same time, it lacks dynamic quantification of risks, which easily leads to overly conservative degradation decisions and affects driving efficiency. To overcome these limitations of traditional methods, SRL is introduced as a new type of function degradation decision-making method. Safety reinforcement learning can dynamically adjust the degradation strategy according to the current operating status and environmental information of the vehicle through online learning and policy optimization, and achieve a balance between risk and performance while meeting safety requirements by introducing safety constraints. Compared with traditional methods, it can also take into account safety, comfort and efficiency at the same time, effectively process high-dimensional perception data through nonlinear function approximation technology, extract features from it and generate optimal decisions for complex scenarios.

[0004] In summary, the function degradation decision method based on safety reinforcement learning can significantly improve the intelligence and robustness of the system, and provide a safer and more stable function degradation solution for intelligent connected vehicles in complex environments. However, existing research still has certain deficiencies in the state space, action space, reward function, and cost function design of safety reinforcement learning function degradation decision-making, and cannot fully meet the needs of practical applications. Therefore, it is urgent to carry out in-depth research in this field to further improve the reliability and applicability of function degradation decisions.

[0005] In the prior art, there is a method that proposes a lane keeping system functional degradation method based on ODD (Operational Design Domain) to improve the safety of autonomous vehicles in dynamic and complex environments. The ODD space is constructed by causal inference and vehicle dynamics stability theory, and a lane detection monitoring method based on counterfactual inference and a lateral control monitoring method based on yaw stability boundary fitting are proposed. When the ODD monitoring finds that the performance is degraded, the corresponding functional degradation strategy is triggered. In addition, in order to quantitatively evaluate the safety benefits of the proposed method, it is recorded in "Enhancing Autonomous Vehicle Safety Based on Operational Design Domain Definition, Monitoring, and Functional Degradation" (Jiang Z, Pan W, Liu J, et al. A Case Study on Lane Keeping System [J]. IEEE Transactions on Intelligent Vehicles, 2024): This paper designs a subset simulation algorithm based on Kriging, which significantly reduces the computing resource requirements. Experiments show that this method performs well in reducing lane departure rate and provides an innovative, explainable and scalable safety enhancement framework for autonomous vehicles. The ODD-based downgrade decision is characterized by its high accuracy and precision in the design of basic element downgrades. However, compared with the functional downgrade decision method based on safety reinforcement learning, its ability to mine and predict implicit risks in the scene is limited, and it is difficult to fully utilize the potential information in dynamic scenes. This method relies on predefined ODD space and monitoring strategies, lacks adaptability to unknown or complex scenes, and has deficiencies in multi-objective balance (such as safety, comfort, and efficiency) and real-time optimization capabilities. In contrast, the method based on safety reinforcement learning achieves dynamic adjustment through risk prediction and strategy learning, and can more flexibly respond to intelligent downgrade needs in complex scenes.

[0006] In the prior art, there is also a degradation method based on the traditional rules of the positioning system, which establishes a degradation mode of "normal operation-positioning degradation-control degradation-manual takeover", designs the degradation measures of dead reckoning positioning and lane keeping control; divides the urban scene into straight road scene, intersection scene and typical on-ramp scene; based on the effective duration of dead reckoning and the quality of lane lines, designs degradation strategies for specific urban scenes according to the degradation mode and degradation measures; finally, when positioning failure occurs, the autonomous driving vehicle selects the corresponding strategy for functional degradation according to the scene in which it is located. It is recorded in "A Vehicle Function Degradation Method for Integrated Positioning Failure in Urban Scenarios" (Tsinghua University CN202411007843.4[P].2024-10-15): This paper proposes a vehicle function degradation strategy under positioning failure for autonomous driving vehicles in urban scenes based on trajectory tracking based on the positioning system, which improves the reliability of the autonomous driving system in dealing with positioning system failure and ensures the safety of vehicle operation.

[0007] Although the traditional rule-based downgrade method of the positioning system has certain feasibility and pertinence, it has limitations such as insufficient adaptability, lack of dynamic optimization ability, limited multi-objective trade-off ability, and difficulty in making full use of high-dimensional data. It relies on preset downgrade modes and strategies, which makes it difficult to deal with abnormal situations in unknown or complex scenarios. At the same time, it cannot dynamically adjust strategies according to real-time risks and environmental changes, which easily leads to overly conservative or inefficient downgrade decisions. In addition, the ability to utilize complex perception data is weak, which limits the decision-making accuracy and system robustness. Summary of the invention

[0008] The object of the present invention is to provide a method and system for intelligent connected vehicle function degradation decision-making based on SRL to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0009] To achieve the above object, the present invention provides a method for decision-making on function degradation of an intelligent connected vehicle based on SRL, which comprises:

[0010] Step 1, collect environmental information and monitor trigger events;

[0011] Step 2: Classify the system risk assessment indicators of the triggering events monitored in step 1. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle. system ;

[0012] Step 3: Based on the environmental information collected in step 1, the risk fields of static obstacles, dynamic obstacles and road boundaries are integrated, and the driving risk R of the vehicle is obtained using the driving risk field model. driving ;

[0013] Step 4: Use the CMDP model to describe the intelligent connected vehicle function degradation decision process to obtain the optimal function degradation strategy; where the state space S and cost function c of the CMDP model are risk Both include the system risk R in step 2 system Take the driving risk R in step 3 driving .

[0014] Furthermore, the state space S of step 4 also includes the state quantities X of N surrounding vehicles, and the state quantity X of the i-th surrounding vehicle in X is i , p i , Δx i , Δy i , They are binary variables indicating whether the surrounding vehicle i can be observed, the relative distance in the longitudinal direction, the relative distance in the lateral direction, the relative speed in the longitudinal direction, and the relative speed in the lateral direction from the vehicle itself.

[0015] Furthermore, the risk cost c of step 4 risk Expressed as: c risk =κ 1 R driving +κ 2 R system , κ 1 and κ 2 is the weight coefficient.

[0016] Furthermore, the objective function of the CMDP model in step 4 is expressed as formula (1), and the constraint is expressed as formula (2):

[0017]

[0018]

[0019] The optimal function degradation strategy π in step 4 * It is expressed as:

[0020] In the formula, η is the safety threshold, Indicates the target value corresponding to *; The target cumulative reward under the current strategy π is used to evaluate the performance of the current strategy π; is the target cumulative cost under the current strategy π, which is used to evaluate the security of the current strategy π; γ∈[0,1) is the discount factor; r(s t ,a t ) and c(s t ,a t ) are respectively the state s at time t t And action decision a tCorresponding reward function and cost function;

[0021] Let Π be a set of strategies, a t Sampling from the random strategy π∈Π, the constrained optimization problem is transformed into a maximum-minimum problem through the Lagrange multiplier λ: If the constraint is violated, increase λ.

[0022] Furthermore, in step 4, the action space of the CMDP model a h is the hth discrete action in the action space A;

[0023] The constrained optimization problem is solved by the Discrete-SAC security reinforcement learning network based on the Lagrangian method; the Discrete-SAC security reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where:

[0024] The policy subnetwork includes the action distribution vector π θ (s t ), the value subnetwork includes the current flexible action value function q ω (s t ) and the flexible state value function V(s t ), the cost subnetwork includes the action cost function

[0025] π θ (s t )=[π θ (a 1 |s t ),π θ (a 2 |s t ),…,π θ (a h |s t ),…,π θ (a |A| |s t )] T (3)

[0026] q ω (s t )=[Q ω (s t ,a 1 ),Q ω (s t ,a 2 ),…,Q ω (s t ,a h ),…,Q ω (s t ,a|A| )] T (4)

[0027] V(s t )=π θ (s t )T[q ω (s t )-ξlogπ θ (s t )] (5)

[0028]

[0029] Where: π θ (a h |s t ) is in state s t Next select action a h The probability of θ is the policy sub-network parameter, Q ω (s t ,a h ) is state s t Next select action a h The action value of ω is the value sub-network parameter For state s t Next select action a h The flexible cost value, ω c is the cost sub-network parameter;

[0030] The loss function J of the policy sub-network is transformed into π (θ) is modified to And use it in state s t Next select action a t Flexible cost value approximate The loss function J(λ) of the Lagrange multiplier λ is expressed as: Represents the experience pool.

[0031] The present invention also provides a smart connected vehicle function degradation decision system based on SRL, which includes:

[0032] An environmental perception module, which is used to collect environmental information;

[0033] A trigger event acquisition module, which is used to monitor trigger events;

[0034] The risk assessment module is used to classify the system risk assessment indicators of the trigger events monitored by the environmental perception module. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle.system ;

[0035] The function degradation decision module is used to obtain the driving risk R of the vehicle based on the environmental information collected by the environmental perception module, the risk fields of static obstacles, dynamic obstacles and road boundaries, and the driving risk field model. driving ;

[0036] The execution control module is used to describe the intelligent connected vehicle function degradation decision process through the CMDP model to obtain the optimal function degradation strategy; wherein the state space S and cost function c of the CMDP model risk The system risk R includes the risk assessment module system Driving risk R based on function degradation decision module driving .

[0037] Furthermore, the state space S of the execution control module also includes the state quantities X of N surrounding vehicles, and the state quantity X of the i-th surrounding vehicle in X is i , p i , Δx i , Δy i , They are binary variables indicating whether the surrounding vehicle i can be observed, the relative distance in the longitudinal direction, the relative distance in the lateral direction, the relative speed in the longitudinal direction, and the relative speed in the lateral direction from the vehicle itself.

[0038] Furthermore, the risk cost c of executing the control module risk Expressed as: c risk =κ 1 R driving +κ 2 R system , κ 1 and κ 2 is the weight coefficient.

[0039] Furthermore, the objective function of the CMDP model of the execution control module is expressed as formula (1), and the constraint is expressed as formula (2):

[0040]

[0041] The optimal function degradation strategy π* of the executive control module is expressed as:

[0042] In the formula, η is the safety threshold, Indicates the target value corresponding to *; The target cumulative reward under the current strategy π is used to evaluate the performance of the current strategy π; is the target cumulative cost under the current strategy π, which is used to evaluate the security of the current strategy π; γ∈[0,1) is the discount factor; r(st , a t ) and c(s t , a t ) are respectively the state s at time t t And action decision a t Corresponding reward function and cost function;

[0043] Let Π be a set of strategies, a t Sampling from the random strategy π∈Π, the constrained optimization problem is transformed into a maximum-minimum problem through the Lagrange multiplier λ: If the constraint is violated, increase λ.

[0044] Furthermore, in the execution control module, the action space of the CMDP model a h is the hth discrete action in the action space A;

[0045] The constrained optimization problem is solved by the Discrete-SAC security reinforcement learning network based on the Lagrangian method; the Discrete-SAC security reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where:

[0046] The policy subnetwork includes the action distribution vector π θ (s t ), the value subnetwork includes the current flexible action value function q ω (s t ) and the flexible state value function V(s t ), the cost subnetwork includes the action cost function

[0047] π θ (s t )==[π θ (a 1 |s t ), π θ (a 2 |s t ), …, π θ (a h |s t ), …, π θ (a |A| |s t )] T (3)

[0048] q ω (s t )=[Q ω (s t , a 1 ), Q ω (s t, a 2 ), …, Q ω (s t , a h ), …, Q ω (s t , a |A| )] T (4)

[0049] V(s t )=π θ (s t ) T [q ω (s t )-ξlogπ θ (s t )] (5)

[0050]

[0051] Among them: its π θ (a h |s t ) is in state s t Next select action a h The probability of θ is the policy sub-network parameter, Q ω (s t , a h ) is state s t Next select action a h The action value of ω is the value sub-network parameter For state s t Next select action a h The flexible cost value, ω c is the cost sub-network parameter;

[0052] The loss function J of the policy sub-network is transformed into π (θ) is modified to And use it in state s t Next select action a t Flexible cost value approximate The loss function J(λ) of the Lagrange multiplier λ is expressed as: Represents the experience pool.

[0053] The present invention has the following advantages due to the adoption of the above technical solution:

[0054] Compared with the ODD-based function downgrading method, the present invention can dynamically mine the implicit risk information in the scene, and achieve real-time optimization through strategy learning, significantly improving the adaptability and multi-objective balance ability in complex scenes. Compared with the traditional rule-based function downgrading method, the present invention abandons the reliance on preset modes, has stronger risk prediction ability and processing ability for high-dimensional perception data, therefore, the present invention can realize efficient, safe and stable function downgrading decisions for intelligent networked vehicles in complex and changing scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a decision flow chart for functional degradation of intelligent connected vehicles based on safety reinforcement learning.

[0056] Figure 2 This is a schematic diagram of the functional degradation system structure based on secure reinforcement learning. DETAILED DESCRIPTION

[0057] In the drawings, the same or similar reference numerals are used to represent the same or similar elements or elements with the same or similar functions. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0058] In the description of the present invention, the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the scope of protection of the present invention.

[0059] The present invention intends to design a method and system for intelligent networked vehicle function degradation decision-making based on safety reinforcement learning. By constructing a comprehensive function degradation framework combining environmental information collection, system risk assessment and driving risk analysis, the safe operation of the vehicle in a complex environment is ensured. The method includes: step one, collecting the status information of the target vehicle, obstacles and road boundaries, and monitoring the environmental weather conditions, communication quality status and software and hardware failure status in real time; step two, performing a system risk assessment on the triggering event based on the exposure probability, controllability and severity; step three, constructing a driving risk field model by integrating the risk fields of static obstacles, dynamic obstacles and road boundaries; step four, using CMDP (Constrained Markov Decision Process) to describe the function degradation decision process, and designing the state space, action space, reward function and cost function, and finally solving it through the Discrete-SAC (English full name Soft Actor-Critic, Chinese full name soft actor-critic) safety reinforcement learning algorithm based on the Lagrangian method, so as to obtain the optimal function degradation strategy.

[0060] Compared with the method in technology one, the present invention constructs a dynamic and adaptive functional downgrade decision framework through security reinforcement learning, which can not only mine the risk information hidden in the scene, but also optimize the strategy in a real-time dynamic environment, making up for the lack of adaptability to complex scenes and multi-objective balancing ability of technology one. Compared with the method in technology two, the present invention abandons the reliance on preset rules, realizes risk prediction and flexible decision-making through reinforcement learning, and significantly enhances the processing ability of unknown scenes and high-dimensional perception data while improving the accuracy and efficiency of downgrade decisions.

[0061] The present invention aims to provide a method and system for intelligent connected vehicle function degradation decision-making based on safety reinforcement learning for urban road scenarios, so as to achieve the minimum risk function degradation decision of intelligent connected vehicles.

[0062] The specific implementation of the method is described in detail below with reference to the accompanying drawings. Figure 1 As shown, the following steps are included:

[0063] Step 1: Collect environmental information and monitor trigger events.

[0064] Environmental information mainly includes the current state information of the target vehicle, the state information of obstacles and the road boundary information.

[0065] The target vehicle is an intelligent connected car. The status information of the target vehicle is the real-time status at the current moment, such as dynamic parameters such as position, speed, acceleration, heading angle, and current driving mode. These status information are mainly collected by the environmental perception sensors configured on the target vehicle. Environmental perception sensors mainly include visual cameras (including monocular, binocular stereo vision, panoramic vision and infrared cameras) and radar ranging sensors (lidar, millimeter wave radar, ultrasonic radar, etc.).

[0066] Obstacles include static obstacles and dynamic obstacles. Static obstacles and dynamic obstacles are distinguished according to the speed state of the obstacle at the current moment. Obstacles whose speed is greater than 0 at the current moment are dynamic obstacles, and obstacles whose speed is equal to 0 at the current moment are static obstacles. For example, the surrounding vehicles in motion are regarded as dynamic obstacles, and the broken-down vehicles are regarded as static obstacles.

[0067] The road boundary information is the position of the boundary lines on the left and right sides of the lane, including the relative position, width, curvature of the lane lines and the lane topology structure of multiple lanes.

[0068] Trigger events are specific situations or events that may cause system failure or performance degradation. These events can be internal, such as hardware failure or software defects, or external, such as environmental disturbances, changes in traffic conditions, etc.

[0069] For example, common triggering events include: 1) Environmental factors, such as rainy and snowy weather, haze, strong winds, etc., which may affect the performance of sensors, causing the perception system to be unable to accurately identify obstacles or road markings, increasing driving risks; 2) Communication quality, such as data packet loss, delay or signal interference, will cause the real-time communication with other vehicles or infrastructure to fail, thereby affecting decision-making and path planning. 3) Hardware failures, such as sensor failure, actuator failure (such as brake system or steering system problems) and computing platform crashes, will directly affect the control and response of the vehicle, and may cause system failure or safety accidents. By monitoring and responding to these triggering events, the system can ensure the stability and safety of autonomous vehicles in complex environments.

[0070] "Monitoring trigger events" means that the system obtains key data that may affect vehicle safety by real-time monitoring of environmental information, communication status, and hardware operating status. These key data include: the impact of weather conditions such as rain, snow, haze, and strong winds on sensor performance in environmental information, and the detection of obstacles and road boundaries in traffic scenes; data integrity in communication status, such as packet loss rate, latency, and signal interference. Monitoring communication status can ensure real-time interaction with other vehicles and infrastructure; the working conditions of sensors, actuators (such as brakes and steering systems), and computing platforms in hardware operating status to avoid control response failures due to faults. Through comprehensive detection and analysis of this information, the safety and stability of the system are guaranteed.

[0071] Step 2: Classify the system risk assessment indicators of the triggering events monitored in step 1. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle. system .

[0072] System risk assessment is the process of analyzing potential failures and risks that may be encountered by intelligent connected vehicles during operation. It is a combination of the probability of a triggering event occurring at the system level, the controllability of the damage, and the severity after it occurs. Therefore, the system risk assessment indicators used in this embodiment include: exposure probability, controllability, and severity, as shown in Table 1 below. The system risk is then divided into four levels: none, low, medium, and high based on the system risk assessment indicators.

[0073] Among them, the exposure probability indicates the possibility of the triggering event being exposed to danger under given environmental and operating conditions, and can be used as an indicator to measure the frequency of system hazardous events. Controllability measures the ability of the people involved in the triggering event (including the driver, passengers or nearby people outside the vehicle) to avoid specific injuries or losses through timely response. Severity is used to assess the degree of damage to one or more related targets in potential system hazardous events. Among them, "hazardous event" can be understood as a triggering event that needs to be combined with a specific scenario to form a hazardous event. Damage to the braking system is a triggering event. When the front vehicle slows down, it becomes a hazardous event, that is, a collision with the front vehicle.

[0074] Table 1 System risk assessment index classification

[0075]

[0076]

[0077] According to the actual driving mode, the corresponding ASIL (English full name "Automotive Safety Integration Level", Chinese full name "Automotive Safety Integrity Level") table is obtained. ASIL is divided into four levels from low to high, A, B, C, and D, among which A is the lowest safety level and D is the highest safety level. As shown in Table 2, in addition to the above four levels, there is also a QM level in the table, which means no impact on safety.

[0078] In this embodiment, the determined ASIL is used as the system risk value. That is to say, according to the level of the system risk assessment index obtained by evaluation, the corresponding system risk value is indexed using the ASIL table, the system risk value is converted into a safety target, similar safety targets are merged, and the merged ASIL is the highest level among the merged items. Accordingly, the safety target can also be gradually decomposed into specific requirements in the operating environment of the intelligent connected vehicle.

[0079] Table 2 ASIL examples

[0080]

[0081] Finally, the system risk R of the ego vehicle can be system It can be defined as:

[0082] R system =[L QM ,L A ,L B ,L C ,L D ]

[0083] Where: L QM The system has no risk, indicating that the ASIL level is QM; L A The system risk is extremely low, indicating that the ASIL level is A; L B The system risk is low, indicating that the ASIL level is B; L C Where, is the system risk, indicating that the ASIL level is C; L D The system risk is high, indicating an ASIL level of D.

[0084] Step 3: Based on the environmental information collected in step 1, the risk fields of static obstacles, dynamic obstacles and road boundaries are integrated, and the driving risk R of the vehicle is obtained using the driving risk field model. driving .

[0085] The overall driving risk of intelligent connected vehicles is expressed as the static obstacle risk field U s , Dynamic obstacle risk field U d , Road Boundary Risk Field U r sum.

[0086] In this embodiment, the driving risk R of the vehicle is driving Defined as R driving =U s +U d +U r .

[0087] Among them, the static obstacle risk field U s Characterizing the risk brought by static obstacles in the operation scene of intelligent connected vehicles mainly considers the relative distance and approach direction between the vehicle and the static obstacles. This embodiment uses a high-order Gaussian function to describe the static obstacle risk field U s , which enables U s The peak of the U is flatter and the field strength at the edge of the static obstacle is similar to its shape, which improves the U s Adaptability and accuracy to obstacles of different shapes. Specifically, U s Defined as:

[0088]

[0089] Where: A s is the static obstacle field strength coefficient; β is a constant that controls the shape of the risk field and is used to adjust the rate and shape of risk field attenuation. s The values ​​of x and β are obtained by testing according to specific scenarios and specific requirements, and they are constants if they are met. obs With y obs are the horizontal and vertical coordinates of the static obstacle respectively; σ x With σ y are the lateral and longitudinal shape functions of the static obstacle, respectively, which are used to define the influence range of the static obstacle risk field. Specifically, the function expression is σ x =k x L obs and σ y =k y W obs ; L obs With W obs are the longitudinal length and lateral width of the obstacle respectively; k x With k x They are the horizontal and vertical shape constants of static obstacles, respectively, and are usually greater than 0.

[0090] Dynamic obstacle risk field U dCharacterizing the risk of dynamic obstacles in the operation scenario of intelligent connected vehicles mainly considers the relative distance, relative speed and approach direction between the vehicle and the dynamic obstacle. When the approach direction is consistent with the relative speed direction, the smaller the relative distance means the greater the possibility of collision. When the speed of the vehicle ahead is higher than that of the vehicle ahead, the risk of collision will gradually decrease as the vehicle ahead gradually moves away from the vehicle ahead. Specifically, this embodiment will d Defined as:

[0091]

[0092] In the formula, A d is the dynamic obstacle field strength coefficient, and its value is obtained through testing according to specific scenarios and specific requirements, and is a constant; α∈[0,1] is a constant for adjusting the effect of obstacle length on the field; σ v is the relative velocity shape function, which is used to define the influence range of the dynamic obstacle risk field. Specifically, the function expression is σ v =k v |v obs -v|;v obs and v are the speeds of the obstacle and the vehicle respectively; k v is the relative velocity shape constant, which is usually greater than 0; in addition, ξ(·) is the sign function describing the relative motion direction between the dynamic obstacle and the ego vehicle, which is defined as ξ=sign(v obs -v), if v obs >v, then ξ=1; otherwise, then ξ=-1.

[0093] Road Boundary Risk Field U r Characterize the risk brought by the intelligent networked vehicle approaching the road boundary (such as the shoulder, guardrail, etc.), mainly considering the relative position and geometric characteristics between the vehicle and the left and right boundaries, and usually using a second-order Gaussian function to generate a smooth risk field. Specifically, this embodiment uses U r Defined as:

[0094]

[0095] In the formula, A r is the field strength coefficient of the road boundary, and its value is obtained through testing according to specific scenarios and specific requirements, and is a constant; left With y right are the ordinates of the left and right boundaries of the road respectively; σ r is the road boundary shape function, which is used to define the influence range of the road boundary risk field. Specifically, the function expression is σ r =k r (W road -W); W road and W are the lateral widths of the road and the vehicle respectively; kr It is a road boundary shape constant, usually greater than 0.

[0096] therefore,

[0097] Step 4: Use the CMDP model to describe the intelligent connected vehicle function degradation decision process to obtain the optimal function degradation strategy. The state space S and cost function c of the CMDP model are risk Both include the system risk R in step 2 system Take the driving risk R in step 3 driving .

[0098] CMDP is an extension of the standard MDP (Markov Decision Process) that incorporates constraints into the decision process. In this embodiment, the intelligent connected vehicle function degradation decision process is described as a CMDP model, which specifies safety requirements as constraints. Specifically, the CMDP model is described as a tuple<S,A,P,r,c,η,γ> , where S is the state space, A is the action space, and P is the action a at time t t ∈A from the state s at time t t To the state s at time t+1 t+1 The transition probability p(s t+1 |s t , a t ), is the reward function, is the cost function.

[0099] 1. State space S

[0100] The state space S refers to the set of all possible states that the agent can observe during its interaction with the environment. Through the environmental perception system, the intelligent connected car can obtain dynamic road traffic status information, and this information will be used as the input of the decision model. Therefore, the selection of the state space S is crucial to help the model make reasonable and safe decisions.

[0101] In this embodiment, the state space S mainly includes the state quantities of surrounding vehicles, the driving risk of the vehicle itself and its system risk, which can be expressed as: S = [X, R driving , R system ], where R driving is the driving risk of the vehicle obtained in step 3; R system is the system risk of the vehicle obtained in step 2, X is the state of all surrounding vehicles, X = [X 1 , X 2 , …, X i , …, X N ], where N is the maximum number of observable vehicles, X iis the state quantity of the surrounding vehicle i, p i is a binary variable indicating whether the surrounding vehicle i can be observed, Δx i with Δy i are the relative distances between the surrounding vehicle i and the vehicle in the longitudinal and lateral directions, and are the relative speeds of the surrounding vehicle i and the vehicle in the longitudinal and lateral directions, respectively.

[0102] 2. Action Space A

[0103] The action space A defines the set of all possible actions that the agent can perform. The present invention uses the normal operation mode, degraded operation mode, driver takeover mode and end operation mode of the intelligent networked vehicle as the discrete action space. A in this embodiment can be expressed as: A = [A 0 , A 1 , A 2 , A 3 ], where A 0 A is the normal operating mode; 1 A is a degraded operation mode; 2 Take over mode for the driver; A 3 To end the running mode.

[0104] When a smart connected car selects an action, it will execute the corresponding driving model to achieve a specific driving task or deal with a specific situation. The following is a detailed description of each mode and its corresponding driving model:

[0105] (1) Normal operation mode A 0 :In normal operation mode, the vehicle drives autonomously according to the set driving trajectory and control strategy. This mode is suitable for normal driving under standard conditions. In this case, all control functions of the vehicle are normal, and no additional safety intervention measures are required to ensure smooth and efficient driving. In this mode, the system focuses on path tracking and speed control to achieve a safe and comfortable driving experience. Classic autonomous driving decision-making and planning algorithms such as State Lattice and EM Planner can be used, and trajectory tracking control can be performed.

[0106] (2) Degraded operation mode A 1: In the degraded operation mode, the system detects that some system functions are abnormal or some hardware fails, so that the vehicle cannot operate in normal automatic driving mode. It will reduce the speed of the vehicle and limit the driving range of the vehicle. It can only drive in a pre-set area or only work in a specific road environment. However, it still has basic safe driving capabilities, namely AEB (automatic emergency braking), ACC (adaptive cruise control), LKA (lane keeping assist) and other L2 level assisted driving key functions to reduce risks and ensure driving safety.

[0107] (3) Driver takeover mode A 2 : In the driver takeover mode, the intelligent connected vehicle can use the longitudinal control model (Intelligent Driver Model, IDM) and lane change decision model (Minimizing Overall Braking Induced by Lane changes, MOBIL) to adjust its own driving behavior according to the real-time traffic conditions and surrounding environment.

[0108] IDM is a longitudinal control model widely used in autonomous driving and traffic flow simulation to describe the acceleration and deceleration behavior of vehicles in following scenarios. The model sets reasonable acceleration, deceleration, and safety distance parameters to ensure that the vehicle can follow the vehicle in front without collision. The specific formula is as follows:

[0109]

[0110] Where: v(t) and a(t) are the velocity and acceleration of the vehicle at the current moment respectively; a max is the maximum acceleration of the vehicle; v 0 The cruising speed of the ego vehicle target; s is the actual distance between the ego vehicle and the preceding vehicle; s * (v,Δv) is the safe distance between the vehicle and the preceding vehicle. Specifically, where s 0 is the static safety distance, T is the target following time, b max is the maximum deceleration of the vehicle.

[0111] MOBIL is a lane change decision model for traffic flow control and autonomous driving. Its core goal is to determine whether to perform a lane change by evaluating the impact of the current lane change on the overall traffic flow, so as to improve traffic efficiency and ensure lane change safety. The model can determine whether to perform a lane change based on the acceleration changes of the driver or autonomous driving vehicle in the current lane and the target lane, as well as the impact of the lane change on the vehicles in front and behind. It uses a lane change incentive function to measure whether the lane should be changed at the moment. The specific formula is as follows:

[0112] Δa ego+p·(Δa new +Δa old )>a th

[0113] Where: Δa ego is the acceleration change of the vehicle after changing lanes; Δa new The acceleration change of the front vehicle in the target lane; Δa old The acceleration change of the following vehicle in the original lane; p is the weight factor, which is used to adjust the impact of lane change on the overall traffic flow; a th is the preset minimum excitation value threshold.

[0114] (4) End operation mode A 3 : In the end operation mode, the vehicle will start the emergency braking model and use the maximum braking deceleration until it comes to a complete stop. This mode is suitable for emergencies, such as when the driver cannot control the vehicle or the system detects a serious fault, and it will quickly decelerate to maximize the safety of all passengers.

[0115] 3. Reward function r

[0116] The total reward function r(t) in this embodiment is composed of the safety reward r safety , Comfort Reward comfort , traffic efficiency reward r efficiency and mode switching reward r mode It consists of four parts, namely: r(t) = r safety +r comfort +r efficiency +r mode The total reward function combines safety, comfort, traffic efficiency and mode switching to comprehensively measure the driving performance of intelligent connected vehicles, so that intelligent connected vehicles can balance driving comfort, efficiency and stability while ensuring safety and make the best downgrade decision.

[0117] (1) Safety reward r safety :For the decision-making of intelligent connected vehicles, safety is the most important factor. Safety rewards can be designed based on indicators such as the relative distance between the vehicle and other vehicles, speed difference, driving risk, etc. Specifically, the safety reward r safety It can be defined as:

[0118]

[0119] Where: TTC is the time to collision, which is used to assess the risk of collision between the vehicle and the obstacle ahead or other vehicles, and is defined as TTC = d / v rel , and d and v rel Indicates the relative distance and speed between the vehicle and the vehicle in front; TTC maxis the maximum allowed collision time; K is the weight coefficient, and its value is determined through experimental tests and based on the specific effects.

[0120] (2) Comfort reward r comfort : Comfort is mainly related to the vehicle's acceleration and lane-changing behavior. Excessive acceleration changes or frequent lane changes may affect the riding experience. Specifically, the comfort reward r comfort It can be defined as:

[0121]

[0122] Where: is the longitudinal jerk of the vehicle; is the lateral jerk of the vehicle; is the steering wheel angular velocity of the vehicle; α, β and γ are weight coefficients, and their values ​​are determined through experimental tests based on specific effects.

[0123] (3) Traffic efficiency reward r efficiency :Traffic efficiency can be designed based on vehicle speed, traffic flow, road conditions, etc. The goal is to reduce traffic congestion as much as possible and increase the average speed of vehicles. Specifically, the traffic efficiency reward r efficiency It can be defined as:

[0124]

[0125] Where: v(t) is the speed of the vehicle; v ave is the average vehicle speed of traffic flow; θ∈(0,1) is the average vehicle speed coefficient, which can be taken as about 0.2 according to the specific effect.

[0126] (4) Mode switching reward r mode : Since mode switching may affect the stability of the system, a penalty reward can be designed to avoid frequent mode switching. Specifically, the mode switching reward r mode It can be defined as:

[0127]

[0128] Where: a t and a t+1 They are the action decisions at time t and time t+1 respectively.

[0129] 4. Cost function c

[0130] The total cost function c(t) in this embodiment is composed of the control cost c control and risk cost C risk It consists of two parts, namely: c(t) = c control +c riskThe total cost function aims to optimize the driving performance of intelligent connected vehicles by limiting excessive control behavior and reducing system risks. By balancing the control cost and risk cost, intelligent connected vehicles can improve driving efficiency and comfort while ensuring safety and make optimal decisions.

[0131] (1) Control cost c control : The control cost is mainly used to limit the frequent changes of vehicle control, avoid excessive acceleration, steering angle, etc., and make the vehicle operation smoother. Specifically, the control cost c control It can be defined as:

[0132] c control =λ 1 |a(t)|+λ 2 |δ(t)|

[0133] Where: a(t) is the acceleration of the vehicle; δ(t) is the steering angle of the vehicle; λ 1 With λ 2 is the weight coefficient, and the specific values ​​are obtained through testing.

[0134] (2) Risk cost c risk :System risk and driving risk are important parts of the cost function, which are mainly calculated by the relative distance and speed difference between the vehicle and other traffic participants. Specifically, the risk cost c risk It can be defined as:

[0135] c risk =κ 1 R driving +κ 2 R system

[0136] Where: κ 1 and κ 2 is the weight coefficient, and the specific values ​​are obtained through testing.

[0137] 5. Constrained Optimization Problem

[0138] On this basis, the problem of secure reinforcement learning can be described by the following optimization proposition, and the optimal strategy can be obtained by solving this proposition

[0139]

[0140] Where: η is the safety threshold, Indicates the target value corresponding to *; The target cumulative reward under strategy π is used to evaluate the performance of the current strategy π; is the target cumulative cost under strategy π, which is used to evaluate the safety of the current strategy π; γ∈[0,1) is the discount factor; r(s t , a t ) and c(s t , a t ) are the reward function and the cost function respectively.

[0141] Let Π be a set of strategies, a t Sampling from a random strategy π∈Π, i.e., a t ~π(·|s t ), the constrained optimization problem can be transformed into a maximum-minimum problem through Lagrange multipliers:

[0142]

[0143] Where: λ is the Lagrange multiplier, which increases once the constraint is violated.

[0144] 6. Discrete-SAC algorithm based on Lagrangian method

[0145] The safety reinforcement learning network based on Lagrangian method Discrete-SAC is used for the function degradation decision of intelligent connected vehicles, that is, the above-mentioned solution of CMDP constraint optimization problem can be used to determine the best driving strategy in the discrete action space A. The Discrete-SAC safety reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where:

[0146] (1) Value sub-network and strategy sub-network

[0147] Assume a 1 , a 2 , …, a |A| is a discrete action in the action space, that is, the action space is h = 1, 2, ... |A|, |A| is the size of the action space, a t The value range is The flexible action value function of SAC-Discrete outputs a vector of size |A|, which consists of the action value of each action, that is, Defined as:

[0148] q ω (s t )=[Q ω (s t , a 1 ), Q ω (s t , a 2 ), …, Q ω (s t , a|A| )] T

[0149] Where: q ω (s t ) is the flexible action value function; Q ω (s t , a h ) is in state s t Next select a h The action value, ω, is obtained by minimizing the flexible Bellman residual J q (ω) The learned value sub-network parameters. The strategy can directly output the action distribution, which is a finite-dimensional vector. The vector is composed of t The probability of selecting each action is composed of π θ :S→[0,1] |A| , specifically expressed as:

[0150] π θ (s t )=[π θ (a 1 |s t ), π θ (a 2 |s t ), …, π θ (a |A| |s t )] T

[0151] Where: π θ (a h |s t ) is in state s t Next select action a h The probability of; θ is the parameter of the strategy sub-network, θ is the loss function J π (θ) The learned policy sub-network parameters.

[0152] In the discrete action space, the flexible state value function V(s t ) is calculated using the following formula:

[0153] V(s t )=π θ (s t ) T [q ω (s t )-ξlogπ θ (s t )]

[0154] Where: ξ is the deterministic entropy term ξlogπ learned through the loss function G(ξ) θ (st ) relative to the reward term q ω (s t ) temperature parameter.

[0155] The policy subnetwork parameters θ can be learned by minimizing the following loss function:

[0156]

[0157] The loss function of the temperature parameter can be defined as:

[0158]

[0159] Where: is a hyperparameter representing the target entropy, The value subnetwork parameters ω of the soft action-value function can be learned by minimizing the soft Bellman residual:

[0160]

[0161] In the formula, represents the experience pool, Q ω (s t , a t ), In state s t Next select action a t The estimated value and target value of the action value. In particular, is estimated using the target Q network, Represents the state s at time t+1 t+1 The flexible state value under is:

[0162]

[0163] Where: are the parameters of the target value sub-network.

[0164] (2) Cost subnetwork

[0165] Established a c The cost subnetwork with parameters is used to approximate the cost value function It is defined as:

[0166]

[0167] Where: For state s t Select action a h The flexible cost value, ω c Through the loss function The learned cost sub-network parameters. Similarly, the loss function of the cost sub-network is defined as:

[0168]

[0169] Where: In state s t Select action a t The current and target state cost values.

[0170]

[0171] Where: is in state s t+1 The target cost function of are the parameters of the target cost subnetwork.

[0172] (3) Lagrange multiplier

[0173] First, since there is already a cost subnetwork, the loss function of the policy subnetwork can be modified by adding the cost value through the Lagrange multiplier. Specifically:

[0174]

[0175] Where: λ is the Lagrange multiplier. Further, we can get:

[0176]

[0177] Then, use the state s t Next select action a t Flexible cost value approximate Right now:

[0178]

[0179] Finally, the loss function of the Lagrange multiplier is expressed as:

[0180]

[0181] Where: J(λ) is the loss of the Lagrange multiplier λ.

[0182] In safety reinforcement learning, the use of CMDP can effectively integrate safety constraints into the learning process, ensuring that when optimizing strategies, not only the maximization of rewards is focused on, but also that the system does not violate safety restrictions during execution, so that safety and performance goals are balanced, avoiding the risk of ignoring safety issues in traditional MDP. The advantage of the present invention is that it integrates system risk and operational risk into reinforcement learning, so that intelligent connected vehicles can select driving modes according to the current motion state and risk state, that is, to achieve minimum risk function degradation.

[0183] SAC is a Lagrangian-based Discrete-SAC secure reinforcement learning algorithm. It is a secure reinforcement learning algorithm that introduces the Lagrangian method on the basis of the ordinary reinforcement learning algorithm (Discrete-SAC algorithm). The present invention has no innovation in this regard. The state space, action space, reward function and cost function are all specially designed by the present invention. First, the Lagrangian method can effectively integrate the constraints in the CMDP into the optimization process of reinforcement learning, ensuring that the constraints of the system are met while optimizing the rewards. Secondly, the Lagrangian multiplier method balances and regulates the constraints by adjusting the multiplier value, avoiding violation of the constraints during the optimization process, thereby improving the safety and stability of the system. In addition, the discrete SAC based on the Lagrangian method can handle complex multi-objective optimization problems, maximize long-term rewards while ensuring constraints, has high flexibility and scalability, and can adapt to different types of constraints and goals.

[0184] In addition, the present invention also provides a smart connected vehicle function degradation decision system based on safety reinforcement learning for urban road scenarios, which includes an environment perception module, a trigger event acquisition module, a risk assessment module, a function degradation decision module and an execution control module. Figure 2 As shown, the detailed description of the structure of each module is as follows:

[0185] The environment perception module has the functions of acquiring and processing lidar information, camera information, communication information, positioning information and road boundary information. It is mainly responsible for collecting vehicle surrounding environment information, vehicle status information and road geometry information. Through multi-sensor fusion technology, it obtains obstacle information (such as obstacle type, position, speed, etc.), vehicle status information (such as position, yaw angle, speed, acceleration, etc.) and road boundary information (such as the relative position, width, curvature of lane lines and the topological structure of multiple lanes).

[0186] The trigger event acquisition module is responsible for real-time monitoring of events that may affect vehicle safety. These events can come from environmental weather conditions, communication quality conditions, or software and hardware failures. For example, environmental weather conditions such as bad weather, strong winds, rain and snow may affect the performance of sensors; communication quality conditions such as data packet loss, delay or interference may cause communication problems between intelligent connected vehicles and other traffic participants; software and hardware failures such as sensors, computing units, and actuators cannot operate normally. By monitoring these potential risk events, the trigger event acquisition module can provide key information for subsequent risk assessment and function degradation decisions.

[0187] The risk assessment module is used to classify the system risk assessment indicators of the trigger events monitored by the environmental perception module. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle. system The system risk level is determined by quantitatively analyzing the exposure probability, controllability and severity of the triggering event; a driving risk field model is constructed by combining obstacle distance, relative speed and road boundary information to generate an overall risk assessment result, which is used as the basis for functional degradation decisions.

[0188] The function degradation decision module is used to obtain the driving risk R of the vehicle using the driving risk field model based on the environmental information collected by the environmental perception module and the risk fields of static obstacles, dynamic obstacles and road boundaries. driving . The function degradation decision module adopts a method based on safety reinforcement learning and combines the risk assessment results to generate a function degradation strategy. The decision modes include normal operation, degraded operation, driver takeover and end of operation. The module introduces a modeling method based on the constrained Markov decision process and uses the Discrete-SAC safety reinforcement learning algorithm based on the Lagrangian method to optimize decisions. Among them, the reward function comprehensively considers four aspects: safety, comfort, traffic efficiency and mode switching to ensure the comprehensive performance of the vehicle in different risk scenarios; the cost function is mainly used to limit vehicle control changes and reduce risk levels, thereby balancing system safety and performance.

[0189] The execution control module is used to describe the intelligent connected vehicle function degradation decision process through the CMDP model to obtain the optimal function degradation strategy; among them, the state space S and cost function c of the CMDP model risk The system risk R includes the risk assessment module system Driving risk R based on function degradation decision module drivingThe execution control module has the functions of normal operation mode control, degraded operation mode control, driver takeover mode control and end operation mode control. The module selects the corresponding control strategy based on the output of the function degradation decision module, combined with the current state and operating environment of the vehicle, and ensures that the vehicle completes the task safely and stably in different modes through precise control of actuators such as steering wheel angle, throttle and brake.

[0190] In one embodiment, the state space S of the execution control module also includes the state quantities X of N surrounding vehicles, the state quantity X of the i-th surrounding vehicle in X i , p i , Δx i , Δy i , They are binary variables indicating whether the surrounding vehicle i can be observed, the relative distance in the longitudinal direction, the relative distance in the lateral direction, the relative speed in the longitudinal direction, and the relative speed in the lateral direction from the vehicle itself.

[0191] In one embodiment, the risk cost c of executing the control module risk Expressed as: c risk =κ 1 R driving +κ 2 R system , κ 1 and κ 2 is the weight coefficient.

[0192] In one embodiment, the objective function of the CMDP model of the execution control module is expressed as formula (1), and the constraint is expressed as formula (2).

[0193] In one embodiment, in the execution control module, the action space of the CMDP model a h is the hth discrete action in the action space A;

[0194] The constraint optimization problem is solved by the Discrete-SAC security reinforcement learning network based on the Lagrangian method; the Discrete-SAC security reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where: the strategy sub-network includes the action distribution vector π θ (s t ), the value subnetwork includes the current flexible action value function q ω (s t ) and the flexible state value function V(s t ), the cost subnetwork includes the action cost function The loss function J of the policy sub-network is transformed into π (θ) is modified to And use it in state s t Next select action a t Flexible cost value approximate The loss function J(λ) of the Lagrange multiplier λ is expressed as: Represents the experience pool.

[0195] In the above embodiments, simply modifying the state space, action space, reward function, and cost function in the constrained Markov decision process of the present invention, or using other secure reinforcement learning algorithms to solve the constrained Markov decision process, can achieve roughly the same effect, but all belong to the main idea of ​​the present invention.

[0196] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent connected vehicle function degradation decision-making based on SRL, characterized in that: include: Step 1, collect environmental information and monitor trigger events; Step 2: Classify the system risk assessment indicators of the triggering events monitored in step 1. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle. system ; Step 3: Based on the environmental information collected in step 1, the risk fields of static obstacles, dynamic obstacles and road boundaries are integrated, and the driving risk R of the vehicle is obtained using the driving risk field model. driving ; Step 4: Use the CMDP model to describe the intelligent connected vehicle function degradation decision process to obtain the optimal function degradation strategy; where the state space S and cost function c of the CMDP model are risk Both include the system risk R in step 2 system Take the driving risk R in step 3 driving .

2. The SRL-based intelligent connected vehicle function degradation decision method according to claim 1, characterized in that: The state space S of step 4 also includes the state quantities X of N surrounding vehicles, and the state quantity X of the i-th surrounding vehicle in X i , p i , Δx i , Δy i , They are binary variables indicating whether the surrounding vehicle i can be observed, the relative distance in the longitudinal direction, the relative distance in the lateral direction, the relative speed in the longitudinal direction, and the relative speed in the lateral direction from the vehicle itself.

3. The SRL-based intelligent connected vehicle function degradation decision method according to claim 1, characterized in that: The risk cost of step 4 c risk Expressed as: c risk =κ1R driving +κ2R system , κ1 and κ2 are weight coefficients.

4. The SRL-based intelligent connected vehicle function degradation decision method according to any one of claims 1 to 3, characterized in that: The objective function of the CMDP model in step 4 is expressed as formula (1), and the constraints are expressed as formula (2): The optimal function degradation strategy π in step 4 * It is expressed as: Where η is the safety threshold, Indicates the target value corresponding to *; The target cumulative reward under the current strategy π is used to evaluate the performance of the current strategy π; is the target cumulative cost under the current strategy π, which is used to evaluate the security of the current strategy π; γ∈[0,1) is the discount factor; r(s t , a t ) and c(s t , a t ) are respectively the state s at time t t And action decision a t Corresponding reward function and cost function; Let Π be a set of strategies, a t Sampling from the random strategy π∈Π, the constrained optimization problem is transformed into a maximum-minimum problem through the Lagrange multiplier λ: If the constraint is violated, increase λ.

5. The SRL-based intelligent connected vehicle function degradation decision method according to claim 4, characterized in that: In step 4, the action space of the CMDP model a h is the hth discrete action in the action space A; The constrained optimization problem is solved by the Discrete-SAC security reinforcement learning network based on the Lagrangian method; the Discrete-SAC security reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where: The policy subnetwork includes the action distribution vector π θ (s t ), the value subnetwork includes the current flexible action value function q ω (s t ) and the flexible state value function V(s t ), the cost subnetwork includes the action cost function p θ (s t )==[π θ (a 1 |s t ),p θ (a 2 |s t ),…,p θ (a h |s t ),…,p θ (a |A| |s t )] T (3) q ω (s t )=[Q ω (s t ,a 1 ),Q ω (s t ,a 2 ),…,Q ω (s t ,a h ),…,Q ω (s t ,a |A| )] T (4) V(s t )=π θ (s t ) T [q ω (s t )-ξlogπ θ (s t )] (5) Where: π θ (a h |s t ) is in state s t Next select action a h The probability of θ is the policy sub-network parameter, Q ω (s t , a h ) is state s t Next select action a h The action value of ω is the value sub-network parameter For state s t Next select action a h The flexible cost value, ω c is the cost sub-network parameter; The loss function J of the policy sub-network is transformed into π (θ) is modified to And use it in state s t Next select action a t Flexible cost value approximate The loss function J(λ) of the Lagrange multiplier λ is expressed as: J Represents the experience pool.

6. A SRL-based intelligent connected vehicle function degradation decision system, characterized in that: include: An environmental perception module, which is used to collect environmental information; A trigger event acquisition module, which is used to monitor trigger events; The risk assessment module is used to classify the system risk assessment indicators of the trigger events monitored by the environmental perception module. According to the actual driving mode, the corresponding vehicle safety integrity level can be determined to obtain the system risk R of the vehicle. system ; The function degradation decision module is used to obtain the driving risk R of the vehicle based on the environmental information collected by the environmental perception module, the risk fields of static obstacles, dynamic obstacles and road boundaries, and the driving risk field model. driving ; The execution control module is used to describe the intelligent connected vehicle function degradation decision process through the CMDP model to obtain the optimal function degradation strategy; wherein the state space S and cost function c of the CMDP model risk The system risk R includes the risk assessment module system Driving risk R based on function degradation decision module driving .

7. The SRL-based intelligent connected vehicle function degradation decision system according to claim 1, characterized in that: The state space S of the execution control module also includes the state quantities X of N surrounding vehicles, the state quantity X of the i-th surrounding vehicle in X i , p i , Δx i , Δy i , They are binary variables indicating whether the surrounding vehicle i can be observed, the relative distance in the longitudinal direction, the relative distance in the lateral direction, the relative speed in the longitudinal direction, and the relative speed in the lateral direction from the vehicle itself.

8. The SRL-based intelligent connected vehicle function degradation decision system according to claim 8, characterized in that: The risk cost of executing the control module c risk Expressed as: c risk =k1R driving +k2R system , k1 and κ2 are weight coefficients.

9. The SRL-based intelligent connected vehicle function degradation decision system according to any one of claims 6 to 8, characterized in that: The objective function of the CMDP model of the execution control module is expressed as formula (1), and the constraints are expressed as formula (2): Optimal functional degradation strategy π for execution control module * It is expressed as: Where η is the safety threshold, Indicates the target value corresponding to *; The target cumulative reward under the current strategy π is used to evaluate the performance of the current strategy π; is the target cumulative cost under the current strategy π, which is used to evaluate the security of the current strategy π; γ∈[0,1) is the discount factor; r(s t ,a t ) and c(s t ,a t ) are respectively the state s at time t t And action decision a t Corresponding reward function and cost function; Let Π be a set of strategies, a t Sampling from the random strategy π∈Π, the constrained optimization problem is transformed into a maximum-minimum problem through the Lagrange multiplier λ: If the constraint is violated, increase λ.

10. The SRL-based intelligent connected vehicle function degradation decision system according to claim 9, characterized in that: Action space of CMDP model in the execution control module a h is the hth discrete action in the action space A; The constrained optimization problem is solved by the Discrete-SAC security reinforcement learning network based on the Lagrangian method; the Discrete-SAC security reinforcement learning network specifically includes a strategy sub-network, a value sub-network and a cost sub-network, where: The policy subnetwork includes the action distribution vector π θ (s t ), the value subnetwork includes the current flexible action value function q ω (s t ) and the flexible state value function V(s t ), the cost subnetwork includes the action cost function p θ (s t )=[π θ (a 1 |s t ),p θ (a 2 |s t ),…,p θ (a h |s t ),…,p θ (a |A| |s t )] T (3) q ω (st)=[Q ω (s t ,a 1 ),Q ω (s t ,a 2 ),…,Q ω (s t ,a h ),…,Q ω (s t ,a |A| )] T (4) V(s t )=π θ (s t ) T [q ω (s t )-ξlogπ θ (s t )] (5) Where: π θ (a h |s t ) is in state s t Next select action a h The probability of θ is the policy sub-network parameter, Q ω (s t , a h ) is state s t Next select action a h The action value of ω is the value sub-network parameter For state s t Next select action a h The flexible cost value, ω c is the cost sub-network parameter; The loss function J of the policy sub-network is transformed into π (θ) is modified to And use it in state s t Next select action a t Flexible cost value approximate The loss function J(λ) of the Lagrange multiplier λ is expressed as: Represents the experience pool.

Images