Safety control strategy optimization method and system based on data timeliness

Abstract

The invention relates to the technical field of safety control, in particular to a safety control strategy optimization method and system based on data timeliness, and the method comprises the following steps: S1, calculating the data timeliness of each experience in an experience pool, Ai = tcurrent-ti, calculating the sampling probability of the experience according to the data timeliness, # imgabs0 # imgabs1 # extracts a batch of experiences from the experience pool based on the sampling probability, and the extracted experiences are utilized to train an Actor network and a Critic network; s2, collecting a vehicle state at the current time, inputting the vehicle state into the trained Actor network, and outputting to obtain a control action; s3, collecting a vehicle state after the vehicle control system executes the control action, inputting the collected vehicle state and the control action into the trained Critic network, and outputting to obtain a safety state evaluation result; and S4, adjusting the trained Actor network based on the safety state evaluation result. According to the method, the safety and the stability in an uncertain environment can be improved by utilizing a sampling strategy based on time characteristics.

Description

Technical Field

[0001] The present invention relates to the field of security control technology, and in particular to a security control strategy optimization method and system based on data timeliness. Background Art

[0002] In modern intelligent control systems, achieving efficient real-time control involves many challenges. Traditional control methods often perform poorly in the face of nonlinearity, uncertainty, and complex dynamics. To address these challenges, methods based on deep learning and reinforcement learning have received widespread attention in recent years. The Deep Deterministic Policy Gradient (DDPG) algorithm combines the representation power of deep neural networks with the advantages of policy gradient methods, and can effectively handle control problems in continuous action spaces and continuous state spaces. However, the traditional DDPG method does not fully consider the timeliness of data when sampling experience, which can easily lead to a decrease in learning efficiency and may cause unsafe behaviors during the exploration process.

[0003] In addition, due to the high requirements of real-time information in the complex application scenarios of transportation systems, traditional mathematical modeling methods often use latency or throughput as a metric. These methods often ignore the complex timing relationships between different components and are not sufficient to express the timeliness of information. Traditional methods have insufficient precision in describing the temporal semantics of model interactions, which makes it impossible to flexibly and accurately analyze and evaluate the behavior of the system in different application scenarios. For example, in intelligent transportation systems, sensors need to collect, transmit and process environmental information in real time, and the timeliness of information is crucial to the performance and safety of the system. However, as time goes by, information will gradually age, and the accumulation of outdated information may cause the system to make wrong decisions. Summary of the invention

[0004] The purpose of the present invention is to provide a security control strategy optimization method and system based on data timeliness to improve security and stability in uncertain environments.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a security control strategy optimization method based on data timeliness, comprising the following steps:

[0007] S1, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Based on the sampling probability, a batch of experience is extracted from the experience pool, and the extracted experience is used to train the Actor network and the Critic network; iis the timeliness of the data of the ith experience, t current is the current time, t i is the experience timestamp, P i is the sampling probability of the ith experience, ∈ is a constant, N is the total number of experiences in the experience pool, A max is the maximum value of data timeliness;

[0008] S2, collects the vehicle status at the current time, inputs the vehicle status into the trained Actor network, and outputs the control action;

[0009] S3, collecting the vehicle state after the vehicle control system executes the control action, and inputting the collected vehicle state and the control action into the trained Critic network, and outputting a safety state assessment result;

[0010] S4: Adjust the trained Actor network based on the safety status assessment result.

[0011] In the above scheme, in S1, the i-th experience in the experience pool is represented by e=(s t ,a t ,r t ,s t+1 ,t i ), s t is the current state, a t To control the action, r t For immediate rewards, t+1 The next state after the current state.

[0012] In a further optimized solution, S1 comprises the following steps:

[0013] S11, initialize the Actor network, Critic network, and Gaussian process model;

[0014] S12, obtain the current time t collected by the sensor current The vehicle status s t , and record the timestamp U(t);

[0015] S13, Gaussian process model is used to model the uncertainty of the vehicle state space model, and we have: t+1 =f(s t )+g(s t ) t +d(s t ), f(s t )+g(s t ) t Construct a priori model, d(s t ) represents the deviation between the prior model and the real environment;

[0016] S14, the vehicle state s at the current time t Input Actor network, Actor network generates control action u rl Output and calculate the immediate reward r t , r t = -α·d(s t )-β·v t +γ·AoI(t), where is the vehicle speed, α, β, γ are weight coefficients;

[0017] S15, define the control barrier function h(s t ):h(s t )=s safe -s t , where s safe For the safe state boundary, construct the constraint conditions of the control barrier function and generate the compensation control action u cbf :

[0018]

[0019] Among them, η is a positive number, which represents the decay rate of the function;

[0020] S16, the final control action a t Applied to vehicle control systems, a t =u rl +u cbf ;

[0021] S17, collecting vehicle control system execution control action a t The vehicle status after t+1 , the current state s t , control action a t , Instant Rewards t , next state s t+1 , experience timestamp t i Construct new experience and store it in the experience pool;

[0022] S18, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Extract a batch of experiences from the experience pool based on sampling probability;

[0023] S19, the vehicle state s at the current time t and the control action u output by the Actor network rlInput the Critic network to calculate the target Q value, and use the extracted batch of experience and the target Q value to update the parameters of the Critic network, and fit the actual Q value by minimizing the loss function of the Critic network;

[0024] S20, calculating the action gradient corresponding to the Actor network according to the updated Critic network, and updating the parameters of the Actor network;

[0025] Steps S12-S20 are executed repeatedly until the desired target or the maximum experimental setting time is reached.

[0026] In the above scheme, the Gaussian process (GP) is used to simulate the system dynamics and uncertainty, and the policy exploration process is limited to the set of safe policies defined by CBF to ensure the safety of the nonlinear system in the entire learning process. A time description mechanism based on data timeliness is adopted, and data timeliness is introduced as the weight of sampling probability in the experience sampling process to ensure that the system can use the latest information to make the best decision when dealing with uncertainty and dynamic changes, thereby improving the robustness and adaptability of the system.

[0027] In a second aspect, the present invention further provides a security control strategy optimization system based on data timeliness, comprising:

[0028] The network training module is used to calculate the data timeliness of each experience in the experience pool. i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Based on the sampling probability, a batch of experience is extracted from the experience pool, and the extracted experience is used to train the Actor network and the Critic network; i is the timeliness of the data of the ith experience, t current is the current time, t i is the experience timestamp, P i is the sampling probability of the ith experience, ∈ is a constant, N is the total number of experiences in the experience pool, A max is the maximum value of data timeliness;

[0029] The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action;

[0030] A safety assessment module, used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output a safety state assessment result;

[0031] The optimization and adjustment module is used to adjust the trained Actor network based on the safety status assessment result.

[0032] In a third aspect, the present invention provides a computer program product, comprising computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps of the security control strategy optimization method based on data timeliness of the present invention.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium comprising computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps in the security control strategy optimization method based on data timeliness of the present invention.

[0034] In a fifth aspect, the present invention provides an electronic device, comprising: a memory storing program instructions; a processor connected to the memory, executing the program instructions in the memory, and implementing the steps in the security control strategy optimization method based on data timeliness of the present invention.

[0035] Compared with the prior art, the present invention has the following technical advantages:

[0036] 1. The present invention is an improved DDPG method based on data timeliness (Age of Information, AoI). In the experience sampling process, data timeliness is introduced as the weight of sampling probability to ensure that the latest and relevant experience that contributes most to strategy generation is given priority, thereby improving the security and stability of the control strategy.

[0037] 2. Introduce the control barrier function (CBF) to control and compensate for unsafe behaviors during the exploration process, thereby generating a safer control strategy.

[0038] For other advantages of the present invention, please refer to the relevant description in the embodiment section. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 The present invention provides a flowchart of the steps of a security control strategy optimization method based on data timeliness in an embodiment of the present invention.

[0041] Figure 2 4 is a flowchart of the steps of the network training process in an embodiment of the present invention.

[0042] Figure 3 Schematic diagram of the implementation of the network training process in the embodiment of the present invention.

[0043] Figure 4 This is a block diagram of the components of the security control strategy optimization system based on data timeliness provided in an embodiment of the present invention.

[0044] Figure 5 A block diagram of the components of an electronic device. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] See also Figure 1 , a security control strategy optimization method based on data timeliness provided in this embodiment includes the following steps:

[0047] S1, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Based on the sampling probability, a batch of experience is extracted from the experience pool, and the extracted experience is used to train the Actor network and the Critic network; i is the data timeliness of the i-th experience in the experience pool, t current is the current time, t i is the experience timestamp, i.e., the time when the experience was generated, P i is the sampling probability, ∈ is a constant, N is the total number of experiences in the experience pool, A max It is the maximum value of data timeliness, that is, the maximum value of data timeliness values ​​of all calculated experiences.

[0048] S2 collects the vehicle status at the current time, inputs the vehicle status into the trained Actor network, and outputs the control action.

[0049] The method of the present invention is applied to vehicle control, and the vehicle state s includes the lateral position x of the vehicle, the longitudinal position y of the vehicle, the heading angle θ of the vehicle, and the speed v of the vehicle. The control actions include, for example, throttle, brake, steering angle, etc.

[0050] S3, collecting the vehicle state after the vehicle control system executes the control action, and inputting the collected vehicle state and the control action into the trained Critic network, and outputting a safety state assessment result.

[0051] The control action output by the Actor network will be input into the vehicle control system for execution. After execution, the vehicle status will be collected again. The safety of the control action can be reflected based on the vehicle status after the control action is executed.

[0052] S4: Adjust the trained Actor network based on the safety status assessment result.

[0053] It should be noted that the adjustment in step S4 can be either zero adjustment or non-zero adjustment. The purpose of step S4 is to further optimize the Actor network through feedback of the evaluation results, so that the control action it outputs is safer and more stable. Therefore, when the trained Actor network is sufficiently reliable, no further adjustment may be required, that is, zero adjustment. When the trained Actor network still has room for further improvement, adjustment may be required, that is, non-zero adjustment. By continuously optimizing the Actor network during actual use, it can output safer and more reliable control actions.

[0054] After the control action is output, the vehicle's control system will adjust the vehicle according to these actions. For example, if it is the throttle, it will increase or decrease the acceleration, and the steering angle will adjust the direction of the vehicle. After the control action is executed, the vehicle state changes, and the sensor continues to collect new state data. Through the feedback of new data, the Critic network can evaluate the quality of the decision (that is, whether the operation is safe, smooth, and effective). This feedback can continue to be used to adjust the model within a certain time window, or further optimize the control strategy through retraining. For example, if the sensor detects that the vehicle is too close to the obstacle in front, the saved controller model will avoid the collision by generating a stronger braking action. Or if the vehicle deviates from the predetermined trajectory, the saved controller model will adjust the steering wheel angle according to the feedback data of the path tracking to ensure that the vehicle returns to the correct driving track.

[0055] See also Figure 2 and Figure 3 The process of training the Actor network and the Critic network in S1 above may include the following steps:

[0056] S11, initialize the Critic network Q(s, a|θ Q ), Actor network μ(s|θ μ ) and Gaussian process model, where s represents the state of the vehicle, a represents the control action, and θ Q and θ μ They are the parameters of the Critic network and the Actor network respectively.

[0057] Initialization of the Critic network and the Actor network: The input layer of the Critic network receives the vehicle state s and action a, and the input layer of the Actor network receives the vehicle state s. The number of hidden layers is set to 2, the number of neurons in each layer is 256, and the hidden layer uses the ReLU activation function to improve the nonlinear fitting ability of the network. The output layer of the Critic network outputs a scalar value, which represents the value estimate under a given state and action, and the output layer of the Actor network outputs action a. Use uniform distribution uniform_(-init_w,init_w) to initialize the weights and biases of the output layer, and the default value of init_w is 3e-3.

[0058] Initialize the Gaussian process (GP) model to estimate the system dynamics and its uncertainty. The kernel function of the Gaussian process model is chosen as the radial basis function (RBF) kernel, which is defined as follows:

[0059]

[0060] Among them, k(s i ,s j ) represents the kernel function of the Gaussian process model, l is the kernel width parameter, ||s i -s j || is the Euclidean distance between states, S i , S j They represent state i and state j respectively.

[0061] Initialization of Gaussian process (GP) model: The GP model is used to estimate the error of state variables. The radial basis function (RBF) kernel is selected as the kernel function to capture the nonlinear relationship between states. The errors are calculated using the state and action data collected in the environment, and the GP model is fitted based on these errors.

[0062] The GP model is used to estimate the dynamics of the system and its uncertainty. As supplementary information, it can help improve the strategy generation of the Actor network, especially when the system model is uncertain or the environment changes. The uncertainty estimation provided by the GP model can improve the robustness of the strategy.

[0063] S12, obtain the current time t collected by the sensor current The vehicle status s t , record the timestamp U(t), which is the moment when the vehicle status is sent from the sensor and received by the system. This timestamp can be used to measure the latency of the data and the age of information (AoI) of the data.

[0064] In this embodiment, the autonomous driving application is taken as an example. The sensors include cameras, laser radars, etc. The safety control strategy is to control the vehicle to avoid collision. Considering the kinematic model of the vehicle, it can be represented by the following state space model:

[0065]

[0066] Among them, x and y are the lateral and longitudinal positions of the vehicle on the plane, respectively, "·" represents the first-order derivative, θ is the heading angle of the vehicle, v is the speed of the vehicle, and a ω is the angular velocity, a v is the acceleration, w x 、w y 、w θ 、w v Therefore, the vehicle state s includes the lateral position x of the vehicle, the longitudinal position y of the vehicle, the heading angle θ of the vehicle, and the speed v of the vehicle. The control input a in the control strategy includes the acceleration control input a v and angular velocity control input a ω .

[0067] S13, using the Gaussian process model to model the uncertainty of the vehicle's state space model, we have:

[0068] s t+1 =f(s t )+g(s t ) t +d(s t )

[0069] f(s t )+g(s t ) t The prior model obtained by constructing the system dynamics and kinematics modeling, a t represents the control action at time t, s t represents the state S at time t, d(s t ) represents the deviation between the prior model and the real environment, that is, the path tracking error. The Gaussian process model learns the actual observation data (such as the difference between the actual motion trajectory of the vehicle and the prediction of the prior model) to calculate this deviation d(s t ) for estimation.

[0070] For any state s * ∈s, d(s given by the GP model * ) is:

[0071]

[0072] is the observation noise variance, y nis the observed value of the corresponding state, k(s * ,s * ) is the kernel function, K is the kernel matrix between the state pairs in the training data set, which represents the correlation between the states, and I is the unit matrix, which is used to ensure numerical stability, especially in the inverse calculation of the kernel matrix. * ) and σ 2 (s * ) represent any state s * The mean and variance of k * Indicates state s * The kernel vector between the training set state and the training set state is used to measure their similarity. The predicted mean given is the dynamics estimate under a given state s. As more and more data are collected, the model can learn the relationship between states more accurately, and μ(s) can better estimate d(s t ), and the uncertainty σ 2 (s) will decrease. This process actually gradually reduces the unmodeled dynamics d(s t ), thus making u(s) closer to the actual system dynamics.

[0073] In control systems, the posterior distribution obtained using the Gaussian process model can provide accurate state prediction for the controller and adjust the control strategy based on uncertainty information to ensure the safety and robustness of the system.

[0074] S14, the vehicle state s at the current time t Input Actor network, Actor network generates control action u rl Output. That is, according to the current state s t , the Actor network generates control actions a t . Calculate the instant reward r t , and update the information age AoI(t).

[0075] The immediate reward takes into account factors such as the vehicle's driving safety, stability, and path tracking error, so there is r t = -α·d(s t )-β·v t +γ·AoI(t),AoI(t)=t current -U(t), where, d(s t ) is the path tracking error, v t is the vehicle speed, α, β, and γ are all weight coefficients. Since the larger the error, the worse the vehicle control effect, in order to encourage the reduction of error, a penalty term -α·d(s t ). Too fast or too slow speed will affect safety and driving stability, so -β·v tTo punish excessive speed. In this way, the control system will tend to maintain a reasonable speed range to ensure safety and stability. AoI(t) indicates the update level of sensor data used by the system. The newer the data, the more accurate the system's perception of the environment and vehicle status. Therefore, in the reward function, the newer the information, the higher the reward.

[0076] The Actor network inputs the vehicle's state information, such as position information, speed, etc., and passes this state information to the neural network. The network generates control actions through forward propagation. This is the action of the reinforcement learning strategy, i.e. u rl .

[0077] In each time step, the information age accumulates over time. Assuming that the length of each time step is Δt, then in the absence of new data, the information age AoI(t) increases linearly with time: AoI(t+1)=AoI(t)+Δt. If the sensor collects new data at time t and is received by the system, the information age AoI(t) is reset to 0, indicating that the latest data is used: AoI(t+1)=0.

[0078] S15, define the control barrier function h(s t ):h(s t )=s safe -s t , where s safe is the safe state boundary. Construct the constraint conditions of the control barrier function and generate the compensation control action u cbf :

[0079]

[0080] Among them, η is a positive number, which represents the decay rate of the function, ensuring that h(s t ) will not change too fast over time and approach the danger boundary. f(s) and g(s) are the previous kinetic equations t+1 =f(s t )+g(s t ) t +d(s t ), a t =u rl +u cbf ,f(s t ) is the drift term, describing the natural evolution of the system when there is no control input. g(s t ) t Indicates control input a t The impact on the system state, where g(s t ) represents the influence coefficient or mode of the control input in the current state.

[0081] In this step, the control barrier function (CBF) is used to constrain the safety state to ensure that the vehicle is always within the safe range during the exploration and learning process.

[0082] S16, the final control action a t Applied to vehicle control systems, a t =u rl +u cbf .

[0083] It should be noted here that during the network training process, the control action a of the vehicle control system is input t =u rl +u cbf , that is, the control action u output by the Actor network rl and compensation control action u cbf The comprehensive result of the input vehicle control system control action a in actual application t That is, the control action output by the Actor network. Therefore, the control action output by the Actor network is represented by a t Instead of u rl .

[0084] Compensation control actions can more reliably ensure safety. For example, when the system receives new status information s t , showing the vehicle's deviation from the lane center d(s t ) exceeds the allowed range, at which point the control action u output by the Actor network rl The vehicle may try to keep on the predetermined path, but the CBF compensation control detects the danger of lane deviation and immediately generates a safety compensation action u cbf , pull the vehicle back to a safe area.

[0085] S17, each time the vehicle performs a control action a t After that, collect the vehicle state s at this time t+1 , the current state s t , control action a t , Instant Rewards t , next state s t+1 and the experience timestamp t i As a new experience added to the experience pool, the experience timestamp is t i Represents the moment when experience i is generated. Each experience stored in the experience pool is represented as:

[0086] e=(s t ,a t ,r t ,s t+1 ,t i );

[0087] S18, for each experience in the experience pool, calculate its data timeliness according to the current time and the experience timestamp, calculate the sampling probability P of the experience according to the calculated data timeliness, and extract a batch of experiences from the experience pool based on the sampling probability, so that fresh information has a higher sampling probability to train the Actor and Critic network. The sampling probability of experience i is:

[0088]

[0089] ∈ is a small constant used to prevent the denominator from being zero, N is the total number of experiences in the experience pool, and A i is the data timeliness of experience i, defined as the current time t current With the experience timestamp t i The difference between:

[0090] A i =t current -t i ;

[0091] S19, the vehicle state s at the current time t and the control action u output by the Actor network rl Input the Critic network to calculate the target Q value, and use the extracted batch of experience and target Q value to update the parameters of the Critic network, and fit the true Q value by minimizing the loss function of the Critic network.

[0092] The Critic network receives the state and action as input and calculates the target Q value according to the Bellman equation.

[0093] y t =r t +γQ'(s t+1 ,μ'(s t+1 ∣θ μ '));

[0094] where Q' and μ' are the target networks and γ is the discount factor. t Represents the target Q value of the Critic network.

[0095] Use the sampled batch data and the target Q value to update the parameters θ of the Critic network Q , fit the true Q value by minimizing the loss function of the Critic network.

[0096]

[0097] Among them, y t =r t +γQ′(s',μ'(s';θ μ');θ Q’ ), r t is the immediate reward, η Q is the learning rate, γ is the discount factor. θ Q’ represents the parameters of the target Critic network, μ'(s'; θ μ ') represents the next action, Q′(s',μ'(s';θ μ ');θ Q‘ ) represents the output Q value of the target Critic network.

[0098] S20, calculate the action gradient corresponding to the Actor network based on the updated Critic network, and update the parameters θ of the Actor network μ , so that the Actor network chooses the optimal next action a t+1 .

[0099]

[0100] Among them, η μ is the learning rate.

[0101] Steps S12-S20 are executed repeatedly until the desired target or the maximum experimental setting time is reached.

[0102] For vehicle control systems, the delay between two key factors is mainly considered: the delay between sensor data acquisition and processing, and the delay between information transmission and control decision. In this regard, the present invention introduces the time description mechanism based on data timeliness into the experience sampling process, calculates the data timeliness of each experience, and calculates the sampling probability according to the timeliness of the information, ensuring that the system can make decisions using the latest and most relevant experience, thereby improving learning efficiency and the safety of control strategies.

[0103] The present invention optimizes the vehicle control strategy using an improved DDPG algorithm to achieve adaptive generation of safety control strategies in complex and uncertain environments. During the strategy optimization process, the control barrier function (CBF) is used to dynamically constrain the control action to ensure that the vehicle is always within a safe range during the exploration and learning process to avoid dangerous operations. At the same time, it ensures that the control strategy meets safety requirements at each state transition. The Gaussian process model is used to model the dynamic behavior and uncertainty of the vehicle system, providing a more accurate system state prediction for the DDPG algorithm, thereby improving the effect of strategy optimization.

[0104] See also Figure 4 Based on the same inventive concept, this embodiment also provides a security control strategy optimization system based on data timeliness, including:

[0105] The network training module is used to calculate the data timeliness of each experience in the experience pool. i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Based on the sampling probability, a batch of experience is extracted from the experience pool, and the extracted experience is used to train the Actor network and the Critic network; i is the timeliness of the data of the ith experience, t current is the current time, t i is the experience timestamp, P i is the sampling probability of the ith experience, ∈ is a constant, N is the total number of experiences in the experience pool, A max is the maximum value of data timeliness;

[0106] The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action;

[0107] A safety assessment module, used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output a safety state assessment result;

[0108] The optimization and adjustment module is used to adjust the trained Actor network based on the safety status assessment result.

[0109] The specific operations performed by each component module can refer to the relevant descriptions in the above method steps, for example, the network training module performs Figure 2 The steps shown are used to train the Critic network and the Actor network, which will not be repeated here to save space.

[0110] like Figure 5 As shown, this embodiment also provides an electronic device, which may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that this figure is exemplary, and other types of structures may be used to supplement or replace this structure to achieve data extraction, report generation, communication or other functions.

[0111] like Figure 5 As shown, the electronic device may further include: an input unit 43, a display unit 44 and a power supply 45. It is worth noting that the electronic device does not necessarily have to include Figure 5 In addition, electronic devices may also include Figure 5 For components not shown, reference may be made to the prior art.

[0112] The processor 41 is sometimes also called a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 41 receives inputs and controls the operations of various components of the electronic device.

[0113] The memory 42 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store information such as configuration information of the processor 41 and instructions executed by the processor 41. The processor 41 may execute the program stored in the memory 42 to implement information storage or processing. In one embodiment, the memory 42 also includes a buffer memory, i.e., a buffer, to store intermediate information.

[0114] An embodiment of the present invention further provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed in an electronic device, the program product enables the electronic device to execute the operation steps included in the method of the present invention.

[0115] An embodiment of the present invention further provides a storage medium storing computer-readable instructions, wherein the computer-readable instructions enable an electronic device to execute the operation steps included in the method of the present invention.

[0116] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0118] The above-described embodiments are only specific implementations of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications, replacements and improvements within the technical scope disclosed by the present invention, and these modifications, replacements and improvements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A security control strategy optimization method based on data timeliness, characterized in that: The following steps are involved: S1, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Extract a batch of experience from the experience pool based on sampling probability, and use the extracted experience to train the Actor network and the Critic network; Among them, A i is the timeliness of the data of the ith experience, t current is the current time, t i is the empirical timestamp, P i is the sampling probability of the ith experience, ∈ is a constant, N is the total number of experiences in the experience pool, A max is the maximum value of data timeliness; S2, collects the vehicle status at the current time, inputs the vehicle status into the trained Actor network, and outputs the control action; S3, collecting the vehicle state after the vehicle control system executes the control action, and inputting the collected vehicle state and the control action into the trained Critic network, and outputting a safety state assessment result; S4: Adjust the trained Actor network based on the safety status assessment result.

2. The security control strategy optimization method based on data timeliness according to claim 1 is characterized in that: In S1, the i-th experience in the experience pool is represented by e=(s t ,a t ,r t ,s t+1 ,t i ), s t is the current state, a t To control the action, r t For immediate rewards, t+1 The next state after the current state.

3. The security control strategy optimization method based on data timeliness according to claim 2 is characterized in that: The S1 comprises the following steps: S11, initialize the Actor network, Critic network, and Gaussian process model; S12, obtain the current time t collected by the sensor current The vehicle status s t , and record the timestamp U(t); S13, Gaussian process model is used to model the uncertainty of the vehicle state space model, and we have: t+1 =f(s t )+g(s t ) t +d(s t ), f(s t )+g(s t ) t Construct a priori model, d(s t ) represents the deviation between the prior model and the real environment; S14, the vehicle state s at the current time t Input Actor network, Actor network generates control action u rl Output and calculate the immediate reward r t , r t = -α·d(s t )-β·v t +γ·AoI(t),AoI(t)=t current -U(t), where v t is the vehicle speed, α, β, γ are weight coefficients; S15, define the control barrier function h(s t ):h(s t )=s safe -s t , where s safe As the safe state boundary, construct the constraint conditions of the control barrier function and generate the compensation control action u cbf : Among them, η is a positive number, which represents the decay rate of the function; S16, the final control action a t Applied to vehicle control systems, a t =u rl +u cbf ; S17, collecting vehicle control system execution control action a t The vehicle status after t+1 , the current state s t , control action a t , Instant Rewards t , next state s t+1 , experience timestamp t i Construct new experience and store it in the experience pool; S18, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Extract a batch of experiences from the experience pool based on sampling probability; S19, the vehicle state s at the current time t and the control action u output by the Actor network rl Input the Critic network to calculate the target Q value, and use the extracted batch of experience and the target Q value to update the parameters of the Critic network, and fit the actual Q value by minimizing the loss function of the Critic network; S20, calculating the action gradient corresponding to the Actor network according to the updated Critic network, and updating the parameters of the Actor network; Steps S12-S20 are executed repeatedly until the desired target or the maximum experimental setting time is reached.

4. The method for optimizing security control strategies based on data timeliness according to claim 3 is characterized in that: In S11, the initialization of the Critic network and the Actor network both use the uniform distribution uniform_(-init_w,init_w) to initialize the weights and biases of the output layer, and the default value of init_w is 3e-3.

5. The security control strategy optimization method based on data timeliness according to claim 3 is characterized in that: The vehicle state s includes the lateral position x of the vehicle, the longitudinal position y of the vehicle, the heading angle θ of the vehicle, and the velocity v of the vehicle.

6. A security control strategy optimization system based on data timeliness, characterized in that: include: The network training module is used to calculate the data timeliness of each experience in the experience pool. i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Extract a batch of experience from the experience pool based on sampling probability, and use the extracted experience to train the Actor network and the Critic network; Among them, A i is the timeliness of the data of the ith experience, t current is the current time, t i is the empirical timestamp, P i is the sampling probability of the ith experience, ∈ is a constant, N is the total number of experiences in the experience pool, A max is the maximum value of data timeliness; The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action; A safety assessment module, used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output a safety state assessment result; The optimization and adjustment module is used to adjust the trained Actor network based on the safety status assessment result.

7. The security control strategy optimization system based on data timeliness according to claim 6 is characterized in that: The network training module performs the following operations to train the Actor network and the Critic network: S11, initialize the Actor network, Critic network, and Gaussian process model; S12, obtain the current time t collected by the sensor current The vehicle status s t , and record the timestamp U(t); S13, Gaussian process model is used to model the uncertainty of the vehicle state space model, and we have: t+1 =f(s t )+g(s t ) t +d(s t ), f(s t )+g(s t ) t Construct a priori model, d(s t ) represents the deviation between the prior model and the real environment; S14, the vehicle state s at the current time t Input Actor network, Actor network generates control action u rl Output and calculate the immediate reward r t , r t = -α·d(s t )-β·v t +γ·AoI(t), where is the vehicle speed, α, β, γ are weight coefficients; S15, define the control barrier function h(s t ):h(s t )=s safe -s t , where s safe As the safe state boundary, construct the constraint conditions of the control barrier function and generate the compensation control action u cbf : Among them, η is a positive number, which represents the decay rate of the function; S16, the final control action a t Applied to vehicle control systems, a t =u rl +u cbf ; S17, collecting vehicle control system execution control action a t The vehicle status after t+1 , the current state s t , control action a t , Instant Rewards t , next state s t+1 , experience timestamp t i Construct new experience and store it in the experience pool; S18, calculate the timeliness of each experience in the experience pool, A i =t current -t i , calculate the empirical sampling probability based on the timeliness of the data, Extract a batch of experiences from the experience pool based on sampling probability; S19, the vehicle state s at the current time t and the control action u output by the Actor network rl Input the Critic network to calculate the target Q value, and use the extracted batch of experience and the target Q value to update the parameters of the Critic network, and fit the actual Q value by minimizing the loss function of the Critic network; S20, calculating the action gradient corresponding to the Actor network according to the updated Critic network, and updating the parameters of the Actor network; Steps S12-S20 are executed repeatedly until the desired target or the maximum experimental setting time is reached.

8. A computer program product comprising computer readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps in the security control strategy optimization method based on data timeliness described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium comprising computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the steps in the security control strategy optimization method based on data timeliness described in any one of claims 1 to 5 are implemented.

10. An electronic device, characterized in that: include: Memory, which stores program instructions; A processor is connected to the memory and executes program instructions in the memory to implement the steps in the security control strategy optimization method based on data timeliness described in any one of claims 1 to 5.

Images