Security control policy optimization method and system based on data timeliness

Abstract

This invention relates to the field of security control technology, and in particular to a method and system for optimizing security control strategies based on data timeliness. The method includes the following steps: S1, calculating the data timeliness of each experience in the experience pool, A i =t current -t i The process involves: S1) calculating the sampling probability of experience based on data timeliness, extracting a batch of experience from the experience pool based on the sampling probability, and using the extracted experience to train the Actor network and Critic network; S2) collecting the vehicle state at the current time and inputting the vehicle state into the trained Actor network to output the control action; S3) collecting the vehicle state after the vehicle control system executes the control action, and inputting the collected vehicle state and the control action into the trained Critic network to output the safety status assessment result; and S4) adjusting the trained Actor network based on the safety status assessment result. This invention utilizes a time-based sampling strategy to improve safety and stability in uncertain environments.

Description

Technical Field

[0001] This invention relates to the field of security control technology, and in particular to a method and system for optimizing security control strategies based on data timeliness. Background Technology

[0002] Achieving efficient real-time control in modern intelligent agent control systems presents numerous challenges. Traditional control methods often perform poorly in the face of nonlinearity, uncertainty, and complex dynamics. To address these challenges, methods based on deep learning and reinforcement learning have received widespread attention in recent years. The Deep Deterministic Policy Gradient (DDPG) algorithm combines the representational power of deep neural networks with the advantages of policy gradient methods, effectively handling control problems in continuous action spaces and persistent state spaces. However, traditional DDPG methods do not fully consider the timeliness of data when sampling experience, which can easily lead to decreased learning efficiency and may result in unsafe behaviors during the exploration process.

[0003] Furthermore, due to the highly demanding real-time requirements of complex transportation system applications, traditional mathematical modeling methods often use latency or throughput as metrics. These methods frequently overlook the complex temporal relationships between different components and are insufficient to express the timeliness of information. Traditional methods suffer from insufficient accuracy in describing the temporal semantics of model interactions, making it impossible to flexibly and accurately analyze and evaluate the system's behavior in different application scenarios. For example, in intelligent transportation systems, sensors need to collect, transmit, and process environmental information in real time, and the timeliness of this information is crucial to the system's performance and safety. However, as time goes on, information gradually becomes outdated, and the accumulation of outdated information may lead the system to make incorrect decisions. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for optimizing security control strategies based on data timeliness, so as to improve security and stability in uncertain environments.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a method for optimizing security control strategies based on data timeliness, comprising the following steps:

[0007] S1, calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is calculated. A batch of experiences is drawn from the experience pool based on sampling probability, and the drawn experiences are used to train the Actor network and the Critic network; where A... iFor the timeliness of the data of the i-th experience, t current t represents the current time. i For experience timestamps, P i Let A be the sampling probability of the i-th experience, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability of the i-th experience. max This represents the maximum value for data timeliness.

[0008] S2: Collect the vehicle status at the current time, input the vehicle status into the trained Actor network, and output the control action;

[0009] S3, collect the vehicle state after the vehicle control system executes the control action, and input the collected vehicle state and the control action into the trained Critic network, and output the safety status assessment result.

[0010] S4, adjust the trained Actor network based on the security status assessment results.

[0011] In the above scheme, in S1, the i-th experience in the experience pool is represented as e = (s t ,a t ,r t ,s t+1 ,t i ), s t As the current state, a t To control the action, r t For instant rewards, s t+1 It is the next state after the current state.

[0012] In a further optimized solution, step S1 includes the following steps:

[0013] S11, Initialize the Actor network, Critic network, and Gaussian process model;

[0014] S12, obtain the current time t of sensor data acquisition. current Vehicle status s t And record the timestamp U(t);

[0015] S13, the Gaussian process model models the uncertainties in the vehicle's state-space model, with: s t+1 =f(s) t )+g(s t )a t +d(s t ), f(s t )+g(s t )a t Constructing a priori model, d(s) t () represents the deviation between the prior model and the real environment;

[0016] S14, change the vehicle status s at the current time. t Input the Actor network, and the Actor network generates control actions u rl Output the value and calculate the immediate reward r. t r t =-α·d(s t )-β·v t +γ·AoI(t), where is the vehicle speed, and α, β, and γ are all weighting coefficients;

[0017] S15, Define the control barrier function h(s) t ): h(s t ) = s safe -s t , where s safe To establish a safe state boundary, constraints are constructed for the control barrier function, and compensating control actions u are generated. cbf :

[0018]

[0019] Where η is a positive number representing the decay rate of the function;

[0020] S16, final control action a t Applied to vehicle control systems, a t =u rl +u cbf ;

[0021] S17, Collect data on the control actions executed by the vehicle control system (a) t The vehicle status after s t+1 , set the current state s t Control action a t Instant rewards t Next state s t+1 Experience timestamp t i New experiences are created and stored in the experience pool;

[0022] S18, Calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is calculated. A batch of experience is drawn from the experience pool based on sampling probability;

[0023] S19, change the vehicle status s at the current time. t and the control actions output by the Actor network u rlThe Critic network is input to calculate the target Q value, and the parameters of the Critic network are updated using a batch of extracted experience and the target Q value. The true Q value is fitted by minimizing the loss function of the Critic network.

[0024] S20: Calculate the action gradient of the Actor network based on the updated Critic network, and update the parameters of the Actor network.

[0025] Repeat steps S12-S20 until the desired goal or the maximum experimental duration is reached.

[0026] In the above scheme, Gaussian processes (GP) are used to simulate system dynamics and uncertainties, restricting the policy exploration process to the set of safe policies defined by the CBF, thus ensuring the safety of the nonlinear system throughout the learning process. A time-based description mechanism based on data timeliness is adopted, and data timeliness is introduced as a weight in the sampling probability during the empirical sampling process. This ensures that the system can make optimal decisions using newer information when dealing with uncertainties and dynamic changes, thereby improving the system's robustness and adaptability.

[0027] Secondly, the present invention also provides a security control strategy optimization system based on data timeliness, comprising:

[0028] The network training module is used to calculate the timeliness of data for each experience in the experience pool. i =t current -t i Based on the timeliness of the data, the empirical sampling probability is calculated. A batch of experiences is drawn from the experience pool based on sampling probability, and the drawn experiences are used to train the Actor network and the Critic network; where A... i For the timeliness of the data of the i-th experience, t current t represents the current time. i For experience timestamps, P i Let A be the sampling probability of the i-th experience, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability of the i-th experience. max This represents the maximum value for data timeliness.

[0029] The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action.

[0030] The safety assessment module is used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output the safety state assessment result.

[0031] An optimization and adjustment module is used to adjust the trained Actor network based on the security status assessment results.

[0032] Thirdly, the present invention provides a computer program product including computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps in the security control strategy optimization method based on data timeliness of the present invention.

[0033] Fourthly, the present invention provides a computer-readable storage medium including computer-readable instructions, characterized in that the computer-readable instructions, when executed by a processor, implement the steps in the security control strategy optimization method based on data timeliness of the present invention.

[0034] Fifthly, the present invention provides an electronic device, comprising: a memory storing program instructions; and a processor connected to the memory, executing the program instructions in the memory to implement the steps in the security control strategy optimization method based on data timeliness of the present invention.

[0035] Compared with the prior art, the present invention has the following technical advantages:

[0036] 1. This invention is an improved DDPG method based on Age of Information (AoI). In the experience sampling process, the age of information is introduced as a weight of the sampling probability to ensure that the latest and most relevant experience that contributes most to strategy generation is given priority, thereby improving the security and stability of the control strategy.

[0037] 2. Introduce a Control Barrier Function (CBF) to control and compensate for unsafe behaviors during the exploration process, thereby generating a safer control strategy.

[0038] Other advantages of this invention are described in the embodiments section. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the steps of the security control strategy optimization method based on data timeliness provided in this embodiment of the invention.

[0041] Figure 2 This is a flowchart illustrating the steps of the network training process in an embodiment of the present invention.

[0042] Figure 3 This is a diagram illustrating the implementation architecture of the network training process in an embodiment of the present invention.

[0043] Figure 4 This is a block diagram of the components of the security control strategy optimization system based on data timeliness provided in this embodiment of the invention.

[0044] Figure 5 This is a block diagram of the components of an electronic device. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Please see Figure 1 This embodiment provides a method for optimizing security control strategies based on data timeliness, including the following steps:

[0047] S1, calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is calculated. A batch of experiences is drawn from the experience pool based on sampling probability, and the drawn experiences are used to train the Actor network and the Critic network; where A... i For the timeliness of the i-th experience in the experience pool, t current t represents the current time. i P is the experience timestamp, i.e., the time when the experience was generated. i Let A be the sampling probability, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability. max This represents the maximum value of data timeliness, which is the maximum value among all calculated empirical data timeliness values.

[0048] S2 collects the vehicle status at the current time and inputs the vehicle status into the trained Actor network, outputting the control action.

[0049] The method of this invention is applied to vehicle control, where the vehicle state s includes the vehicle's lateral position x, longitudinal position y, heading angle θ, and speed v. Control actions include, for example, throttle, brake, and steering angle.

[0050] S3: Collect the vehicle state after the vehicle control system executes the control action, and input the collected vehicle state and the control action into the trained Critic network to output the safety status assessment result.

[0051] The control actions output by the Actor network are input into the vehicle control system for execution. After execution, the vehicle status is collected again, and the safety of the control actions can be reflected based on the vehicle status after the execution of the control actions.

[0052] S4, adjust the trained Actor network based on the security status assessment results.

[0053] It should be noted that the adjustment in step S4 can be either zero or non-zero. The purpose of step S4 is to further optimize the Actor network through feedback from the evaluation results, making its output control actions safer and more stable. Therefore, when the trained Actor network is sufficiently reliable, no further adjustment may be needed, i.e., zero adjustment. When the trained Actor network still has room for further improvement, adjustment may be necessary, i.e., non-zero adjustment. By continuously optimizing the Actor network during actual use, its output control actions can be made safer and more reliable.

[0054] After a control action is output, the vehicle's control system adjusts the vehicle accordingly. For example, if the accelerator is pressed, acceleration is increased or decreased; if the steering angle is adjusted, the vehicle's direction is changed. After the control action is executed, the vehicle's state changes, and sensors continue to collect new state data. Through feedback from this new data, the Critic network can evaluate the quality of the decision (i.e., whether the operation was safe, smooth, and effective). This feedback can be used to adjust the model within a certain time window or to further optimize the control strategy through retraining. For example, if the sensors detect that the vehicle is too close to an obstacle ahead, the saved controller model will generate a stronger braking action to avoid a collision. Or, if the vehicle deviates from the predetermined trajectory, the saved controller model will adjust the steering wheel angle based on path tracking feedback data to ensure the vehicle returns to the correct driving path.

[0055] See also Figure 2 and Figure 3 The process of training the Actor network and Critic network in S1 above may include the following steps:

[0056] S11, Initialize the Critic network Q(s, a|θ) Q ), Actor network μ(s|θ) μ ) and Gaussian process model, where s represents the vehicle state, a represents the control action, and θ Q and θ μ These are the parameters for the Critic network and the Actor network, respectively.

[0057] Initialization of the Critic and Actor networks: The input layer of the Critic network receives the vehicle state *s* and action *a*, while the input layer of the Actor network also receives the vehicle state *s*. Both networks have 2 hidden layers with 256 neurons per layer. The hidden layers use the ReLU activation function to improve the network's non-linear fitting capability. The Critic network outputs a scalar value representing the value estimate given the state and action, while the Actor network outputs the action *a*. The weights and biases of the output layers are initialized using a uniform distribution `uniform_(-init_w, init_w)`, where `init_w` defaults to 3e-3.

[0058] A Gaussian process (GP) model is initialized to estimate the system dynamics and its uncertainties. The kernel function for the Gaussian process model is chosen to be a radial basis function (RBF) kernel, defined as follows:

[0059]

[0060] Where, k(s) i ,s j ) represents the kernel function of the Gaussian process model, l is the kernel width parameter, and ||s i -s j || represents the Euclidean distance between states, S i S j Let i and j represent states i and j, respectively.

[0061] Initialization of the Gaussian process (GP) model: The GP model is used to estimate the error of the state variables. A radial basis function (RBF) kernel is selected as the kernel function to capture the nonlinear relationship between states. The error is calculated using state and action data collected in the environment, and the GP model is fitted based on these errors.

[0062] The GP model is used to estimate the dynamics and uncertainties of a system. As supplementary information, it can help improve policy generation in Actor networks, especially when the system model has uncertainties or the environment changes. The uncertainty estimation provided by the GP model can improve the robustness of the policy.

[0063] S12, obtain the current time t of sensor data acquisition. current Vehicle status s t The system records a timestamp U(t), which is the moment when the vehicle status is transmitted from the sensor and received by the system. This timestamp can be used to measure data latency and the data's Age of Information (AoI).

[0064] This embodiment takes an autonomous driving application as an example. Sensors include cameras, LiDAR, etc., and the safety control strategy is to control the vehicle to avoid collisions. Considering the vehicle's kinematic model, it can be represented by the following state-space model:

[0065]

[0066] Where x and y are the lateral and longitudinal positions of the vehicle in the plane, respectively; "·" indicates the first derivative; θ is the vehicle's heading angle; v is the vehicle's velocity; and a ω Let a be the angular velocity. v For acceleration, w x w y w θ w v All are random disturbance terms. Therefore, the vehicle state s includes the vehicle's lateral position x, longitudinal position y, heading angle θ, and velocity v, and the control input a in the control strategy includes acceleration control input a0. v and angular velocity control input a ω .

[0067] S13, using a Gaussian process model to model the uncertainties in the vehicle's state-space model, we have:

[0068] s t+1 =f(s) t )+g(s t )a t +d(s t )

[0069] f(s t )+g(s t )a t The prior model derived from system dynamics and kinematic modeling, a t s represents the control action at time t. t Let S, d(s) represent the state at time t. t d(s) represents the deviation between the prior model and the real environment, i.e., the path tracking error. The Gaussian process model learns from actual observation data (such as the difference between the vehicle's actual trajectory and the prior model's prediction) to adjust this deviation d(s). t Estimate.

[0070] For any state s * ∈s, the GP model gives d(s) * The posterior distribution of ) is:

[0071]

[0072] It is the variance of the observation noise, y nFor the observations of the corresponding state, k(s) * ,s * μ(s) is the kernel function, K is the kernel matrix between state pairs in the training dataset, representing the correlation between states, and I is the identity matrix used to ensure numerical stability, especially when calculating the inverse of the kernel matrix. * ) and σ 2 (s * ) represent any state s * The mean and variance of k. * Represents state s * The kernel vector between the training set states and their corresponding states is used to measure their similarity. The given predicted mean is a dynamic estimate for a given state s. As more data is collected, the model can learn the relationships between states more accurately, and μ(s) can better estimate d(s). t And uncertainty σ 2 (s) will decrease. This process actually gradually reduces the unmodeled dynamic d(s). t The influence of ) makes u(s) closer to the actual system dynamics.

[0073] In control systems, the posterior distribution obtained using the Gaussian process model can provide accurate state predictions for the controller and adjust the control strategy based on uncertainty information, thereby ensuring the safety and robustness of the system.

[0074] S14, change the vehicle status s at the current time. t Input the Actor network, and the Actor network generates control actions u rl Output. That is, based on the current state s. t Actor network generates control action a t Calculate the immediate reward r t And update the information age AoI(t).

[0075] Instant rewards take into account factors such as vehicle driving safety, stability, and path tracking error, therefore r t =-α·d(s t )-β·v t +γ·AoI(t),AoI(t)=t current -U(t), where, d(s t ) represents the path tracking error, v t Let be the vehicle speed, and α, β, and γ be weighting coefficients. Since a larger error means a worse control effect, a penalty term -α·d(s) is set to encourage error reduction. t Both excessively high and low speeds can affect safety and driving stability, therefore -β·v is used. tThis is used to penalize excessive speed. The control system will then tend to maintain a reasonable speed range, ensuring safety and smoothness. AoI(t) represents the update level of the sensor data used by the system; the newer the data, the more accurate the system's perception of the environment and vehicle status. Therefore, in the reward function, the newer the information, the higher the reward.

[0076] The Actor network takes the vehicle's state information, such as position and speed, as input, and transmits this information to the neural network. The network then generates control actions through forward propagation; these actions are the actions of the reinforcement learning policy, i.e., u. rl .

[0077] Within each time step, the information age accumulates over time. Assuming the length of each time step is Δt, then without new data arriving, the information age AoI(t) increases linearly with time: AoI(t+1) = AoI(t) + Δt. If the sensor collects new data at time t and it is received by the system, the information age AoI(t) is reset to 0, indicating that the latest data is being used: AoI(t+1) = 0.

[0078] S15, Define the control barrier function h(s) t ): h(s t ) = s safe -s t , where s safe To establish a safe state boundary, construct constraints for the control barrier function and generate the compensating control action u. cbf :

[0079]

[0080] Where η is a positive number representing the decay rate of the function, ensuring that h(s) t The rate of change with time will not approach the danger boundary too quickly. f(s) and g(s) are the values ​​from the previous dynamic equations. t+1 =f(s) t )+g(s t )a t +d(s t In ), a t =u rl +u cbf ,f(s t g(s) is the drift term, which describes the natural evolution of the system in the absence of control input. t )a t Indicates control input a t The impact on the system state, where g(s) t ) indicates the influence coefficient or method of the control input in the current state.

[0081] In this step, the control barrier function (CBF) is used to constrain the safety state, ensuring that the vehicle remains within a safe range throughout the exploration and learning process.

[0082] S16, final control action a t Applied to vehicle control systems, a t =u rl +u cbf .

[0083] It should be noted here that during network training, the control action 'a' input to the vehicle control system... t =u rl +u cbf That is, the control action u output by the Actor network. rl With compensation control action u cbf The overall result, and in practical applications, the control action a input to the vehicle control system. t This refers to the control action output by the Actor network; therefore, the control action output by the Actor network at this time is represented by 'a'. t Instead of u rl .

[0084] Compensation control actions can more reliably ensure safety. For example, when the system receives new status information s... t This displays the vehicle's deviation from the lane center, d(s). t If the range is exceeded, the Actor network will output the control action u. rl The system may attempt to keep the vehicle on its intended path, but the CBF (Channel Fretting and Flow Control) detects a risk of lane departure and immediately initiates a safety compensation action. cbf They pulled the vehicle back to a safe area.

[0085] S17, in each time the vehicle performs a control action a t Then, collect the vehicle status s at this time. t+1 , set the current state s t Control action a t Instant rewards t Next state s t+1 and experience timestamp t i As a new experience added to the experience pool, the experience timestamp t i This represents the moment when experience i was generated. Each experience stored in the experience pool is represented as:

[0086] e = (s t ,a t ,r t ,s t+1 ,t i );

[0087] S18, for each experience in the experience pool, calculate its data timeliness based on the current time and the experience timestamp. Based on the calculated data timeliness, calculate the sampling probability P of the experience. Based on the sampling probability, draw a batch of experiences from the experience pool, giving fresh information a higher sampling probability for training the Actor and Critic networks. The sampling probability of experience i is:

[0088]

[0089] ∈ is a small constant used to prevent the denominator from being zero, N is the total number of experiences in the experience pool, and A i The timeliness of data for experience i is defined as the current time t. current With experience timestamp t i The difference between them:

[0090] A i =t current -t i ;

[0091] S19, change the vehicle status s at the current time. t and the control actions output by the Actor network u rl The Critic network is input to calculate the target Q-value, and the parameters of the Critic network are updated using a batch of extracted experience and the target Q-value. The true Q-value is fitted by minimizing the loss function of the Critic network.

[0092] The Critic network receives the state and actions as input and calculates the target Q value according to the Bellman equation.

[0093] y t =r t +γQ'(s t+1 ,μ'(s t+1 |θ μ '));

[0094] Where Q' and μ' are the target network, and γ is the discount factor. t This represents the target Q-value of the Critic network.

[0095] The parameters θ of the Critic network are updated using the sampled batch data and the target Q value. Q The true Q-value is fitted by minimizing the loss function of the Critic network.

[0096]

[0097] Among them, y t =r t +γQ′(s',μ'(s';θ μ');θ Q’ ), r t For instant rewards, η Q θ is the learning rate, and γ is the discount factor. Q’ The parameters of the target Critic network are μ'(s'); θ μ ') represents the next action, Q′(s',μ'(s';θ μ ');θ Q‘ ) represents the output Q value of the target Critic network.

[0098] S20, calculate the action gradient of the Actor network based on the updated Critic network, and update the parameters θ of the Actor network. μ This allows the Actor network to select the optimal next action a. t+1 .

[0099]

[0100] Where, η μ This is the learning rate.

[0101] Repeat steps S12-S20 until the desired goal or the maximum experimental duration is reached.

[0102] For vehicle control systems, the delay between two key factors is primarily considered: the delay between sensor data acquisition and processing, and the delay between information transmission and control decision-making. To address this, this invention introduces a time-description mechanism based on data timeliness into the experience sampling process. By calculating the timeliness of each piece of experience and determining the sampling probability based on the timeliness of the information, it ensures that the system can utilize the latest and most relevant experience for decision-making, thereby improving learning efficiency and the safety of the control strategy.

[0103] This invention utilizes an improved DDPG algorithm to optimize vehicle control strategies, enabling adaptive generation of safe control strategies in complex and uncertain environments. During strategy optimization, a control barrier function (CBF) is used to dynamically constrain control actions, ensuring the vehicle remains within a safe range during exploration and learning, preventing dangerous maneuvers, and ensuring the control strategy meets safety requirements at each state transition. A Gaussian process model is used to model the dynamic behavior and uncertainties of the vehicle system, providing more accurate system state predictions for the DDPG algorithm, thereby improving the effectiveness of strategy optimization.

[0104] Please see Figure 4 Based on the same inventive concept, this embodiment also provides a security control strategy optimization system based on data timeliness, including:

[0105] The network training module is used to calculate the timeliness of data for each experience in the experience pool. i =t current -t i Based on the timeliness of the data, the empirical sampling probability is calculated. A batch of experiences is drawn from the experience pool based on sampling probability, and the drawn experiences are used to train the Actor network and the Critic network; where A... i For the timeliness of the data of the i-th experience, t current t represents the current time. i For experience timestamps, P i Let A be the sampling probability of the i-th experience, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability of the i-th experience. max This represents the maximum value for data timeliness.

[0106] The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action.

[0107] The safety assessment module is used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output the safety state assessment result.

[0108] An optimization and adjustment module is used to adjust the trained Actor network based on the security status assessment results.

[0109] The specific operations performed by each component module can be found in the relevant descriptions in the aforementioned method steps, such as the network training module. Figure 2 The steps shown are for training the Critic network and the Actor network; for the sake of brevity, they will not be repeated here.

[0110] like Figure 5 As shown, this embodiment also provides an electronic device that may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that this figure is exemplary, and other types of structures can be used to supplement or replace this structure to achieve data extraction, report generation, communication, or other functions.

[0111] like Figure 5 As shown, the electronic device may also include an input unit 43, a display unit 44, and a power supply 45. It is worth noting that the electronic device is not necessarily required to include these components. Figure 5 All components shown in the image. Furthermore, electronic devices may also include... Figure 5 For components not shown, please refer to existing technologies.

[0112] Processor 41, sometimes also called controller or operation control, may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device.

[0113] The memory 42 may be one or more of the following: a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store configuration information of the processor 41, instructions executed by the processor 41, and other information. The processor 41 can execute programs stored in the memory 42 to perform information storage or processing. In one embodiment, the memory 42 further includes a buffer memory, or buffer, to store intermediate information.

[0114] This invention also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed in an electronic device, the program product causes the electronic device to perform the operation steps included in the method of this invention.

[0115] This invention also provides a storage medium storing computer-readable instructions that cause an electronic device to perform the operation steps included in the method of this invention.

[0116] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The embodiments described above are merely specific implementations of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications, substitutions, and improvements within the technical scope disclosed in the present invention, and these modifications, substitutions, and improvements should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing security control strategies based on data timeliness, characterized in that, Includes the following steps: S1, calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is dynamically calculated. A batch of experiences is extracted from the experience pool based on the sampling probability, and the extracted experiences are used to train the Actor network and the Critic network. Among them, A i For the timeliness of the data of the i-th experience, t current t represents the current time. i For experience timestamps, P i Let A be the sampling probability of the i-th experience, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability of the i-th experience. max This represents the maximum value for data timeliness. S2, collect the vehicle status at the current time, input the vehicle status into the trained Actor network, and output the control action; S3, collect the vehicle state after the vehicle control system executes the control action, and input the collected vehicle state and the control action into the trained Critic network, and output the safety status assessment result. S4, Adjust the trained Actor network based on the security status assessment results; In S1, the i-th experience in the experience pool is represented as e = (s t ,a t ,r t ,s t+1 ,t i ), s t As the current state, a t To control the action, r t For instant rewards, s t+1 The next state after the current state; S1 includes the following steps: S11, Initialize the Actor network, Critic network, and Gaussian process model; S12, obtain the current time t of sensor data acquisition. current Vehicle status s t And record the timestamp U(t); S13, the Gaussian process model models the uncertainties in the vehicle's state-space model, with: s t+1 =f(s) t )+g(s t )a t +d(s t ), f(s t )+g(s t )a t Constructing a priori model, d(s) t ) represents the deviation between the prior model and the real environment; S14, change the vehicle status s at the current time. t Input the Actor network, and the Actor network generates control actions u. rl Output the value and calculate the immediate reward r. t r t =-α·d(s t )-β·v t +γ·AoI(t),AoI(t)=t current -U(t), where v t For vehicle speed, α, β, and γ are weighting coefficients; S15, Define the control barrier function h(s) t ): h(s t ) = s safe -s t , where s safe To establish a safe state boundary, constraints are constructed for the control barrier function, and compensating control actions u are generated. cbf : Where η is a positive number representing the decay rate of the function, f(s) t ) is the drift term, which describes the natural evolution of the system without control input, where g(s) t ) represents the influence coefficient or method of the control input in the current state, u rl For the control actions output by the Actor network, u cbf The compensation control action output by the Actor network; S16, final control action a t Applied to vehicle control systems, a t =u rl +u cbf ; S17, Collect data on the control actions executed by the vehicle control system (a) t The vehicle status after s t+1 , set the current state s t Control action a t Instant rewards t Next state s t+1 Experience timestamp t i New experiences are created and stored in the experience pool; S18, Calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is dynamically calculated. A batch of experience is drawn from the experience pool based on sampling probability; S19, change the vehicle status s at the current time. t and the control actions output by the Actor network u rl The Critic network is input to calculate the target Q value, and the parameters of the Critic network are updated using a batch of extracted experience and the target Q value. The true Q value is fitted by minimizing the loss function of the Critic network. S20: Calculate the action gradient of the Actor network based on the updated Critic network, and update the parameters of the Actor network. Repeat steps S12-S20 until the desired goal or the maximum experimental duration is reached.

2. The security control strategy optimization method based on data timeliness according to claim 1, characterized in that, In S11, both the Critic and Actor networks are initialized using uniform distribution uniform_(-init_w,init_w) to initialize the weights and biases of the output layer, with init_w defaulting to 3e-3.

3. The security control strategy optimization method based on data timeliness according to claim 2, characterized in that, The vehicle state s includes the vehicle's lateral position x, longitudinal position y, heading angle θ, and velocity v.

4. A security control strategy optimization system based on data timeliness, characterized in that, include: The network training module is used to calculate the timeliness of data for each experience in the experience pool. i =t current -t i Based on the timeliness of the data, the empirical sampling probability is dynamically calculated. A batch of experiences is extracted from the experience pool based on the sampling probability, and the extracted experiences are used to train the Actor network and the Critic network. Among them, A i For the timeliness of the data of the i-th experience, t current t represents the current time. i For experience timestamps, P i Let A be the sampling probability of the i-th experience, ∈ be a constant, N be the total number of experiences in the experience pool, and A be the sampling probability of the i-th experience. max This represents the maximum value for data timeliness. The action generation module is used to input the collected vehicle state at the current time into the trained Actor network and output the control action. The safety assessment module is used to input the vehicle state after the vehicle system executes the control action and the control action into the trained Critic network, and output the safety state assessment result. The optimization and adjustment module is used to adjust the trained Actor network based on the security status assessment results; the network training module performs the following operations to train the Actor network and the Critic network: S11, Initialize the Actor network, Critic network, and Gaussian process model; S12, obtain the current time t of sensor data acquisition. current Vehicle status s t And record the timestamp U(t); S13, the Gaussian process model models the uncertainties in the vehicle's state-space model, with: s t+1 =f(s) t )+g(s t )a t +d(s t ), f(s t )+g(s t )a t Constructing a priori model, d(s) t ) represents the deviation between the prior model and the real environment; S14, change the vehicle status s at the current time. t Input the Actor network, and the Actor network generates control actions u. rl Output the value and calculate the immediate reward r. t r t =-α·d(s t )-β·v t +γ·AoI(t), where is the vehicle speed, and α, β, and γ are all weighting coefficients; S15, Define the control barrier function h(s) t ): h(s t ) = s safe -s t , where s safe To establish a safe state boundary, constraints are constructed for the control barrier function, and compensating control actions u are generated. cbf : Where η is a positive number representing the decay rate of the function, f(s) t ) is the drift term, which describes the natural evolution of the system without control input, g(s) t ) represents the influence coefficient or method of the control input in the current state, u rl For the control actions output by the Actor network, u cbf The compensation control action output by the Actor network; S16, final control action a t Applied to vehicle control systems, a t =u rl +u cbf ; S17, Collect data on the control actions executed by the vehicle control system (a) t The vehicle status after s t+1 , set the current state s t Control action a t Instant rewards t Next state s t+1 Experience timestamp t i New experiences are created and stored in the experience pool; S18, Calculate the data timeliness of each experience in the experience pool, A i =t current -t i Based on the timeliness of the data, the empirical sampling probability is dynamically calculated. A batch of experience is drawn from the experience pool based on sampling probability; S19, change the vehicle status s at the current time. t and the control actions output by the Actor network u rl The Critic network is input to calculate the target Q value, and the parameters of the Critic network are updated using a batch of extracted experience and the target Q value. The true Q value is fitted by minimizing the loss function of the Critic network. S20: Calculate the action gradient of the Actor network based on the updated Critic network, and update the parameters of the Actor network. Repeat steps S12-S20 until the desired goal or the maximum experimental duration is reached.

5. A computer program product comprising computer-readable instructions, characterized in that, When executed by a processor, the computer-readable instructions implement the steps in the security control strategy optimization method based on data timeliness as described in any one of claims 1-3.

6. A computer-readable storage medium comprising computer-readable instructions, characterized in that, When executed by a processor, the computer-readable instructions implement the steps in the security control strategy optimization method based on data timeliness as described in any one of claims 1-3.

7. An electronic device, characterized in that, include: Memory, which stores program instructions; The processor, connected to the memory, executes program instructions in the memory to implement the steps in the data timeliness-based security control strategy optimization method according to any one of claims 1-3.

Images