A risk management system based on offline reinforcement learning

Abstract

The application discloses a risk management system based on offline reinforcement learning, comprising: an offline data set generation module: generating an offline data set according to a screening strategy and a four-tuple data format; a sequence data generation module: training a supervised learning model according to the offline data set, and generating sequence data according to the supervised learning model; a risk reconstruction module: calculating a risk adjustment value according to the change value of the sequence data and training data, and reconstructing a risk function according to the risk adjustment value; the training data comprises any one or a combination of multiple items of training times, training time, training completion degree; a minimum risk strategy generation module: calculating a minimum risk value according to the reconstructed risk function, inputting the minimum risk value into the supervised learning model, and obtaining a minimum risk strategy function. The application solves the problems of high policy trial and error cost, low interaction efficiency with the real environment and low policy optimization efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a risk management system based on offline reinforcement learning. Background Art

[0002] Intelligent risk management is an important means of strengthening risk management by applying intelligent technologies. Traditional intelligent risk management relies on machine learning models to optimize the modeling of value at risk and conditional value at risk, thereby supporting risk management at different confidence levels. Intelligent risk management utilizes artificial intelligence technologies to process and analyze massive amounts of data and complex business environments. Through data analysis and model building, it comprehensively assesses and manages different types of risks. For example, machine learning and deep learning algorithms can use historical data and pattern recognition to perform risk prediction and decision support.

[0003] Current intelligent risk management methods primarily include volatility-based risk control, correlation-based risk control, machine learning-based risk assessment models, and reinforcement learning-based risk management. However, these methods only consider risks within shorter timeframes and fail to adequately consider risks over longer timeframes. Reinforcement learning-based risk management requires continuous trial and error with the environment to optimize strategies. However, if strategies are tested directly in a real-world environment, random exploration can lead to losses and inefficient interaction with the real environment. Simulations based on physical theorems often lack precision, making it difficult to achieve optimal strategies.

[0004] In order to reduce the trial-and-error cost of intelligent risk management using reinforcement learning and improve the efficiency of interaction with the real environment and strategy optimization, a risk management system based on offline reinforcement learning is proposed. Summary of the Invention

[0005] The embodiment of the present invention proposes a risk management system based on offline reinforcement learning to at least solve the problems of high cost of trial and error of strategies, low efficiency of interaction with the real environment, and low efficiency of strategy optimization.

[0006] According to one embodiment of the present invention, a risk management system based on offline reinforcement learning is proposed, comprising:

[0007] Offline data set generation module: generates offline data sets based on the filtering strategy and quadruple data format;

[0008] Sequence data generation module: trains a supervised learning model based on the offline dataset and generates sequence data based on the supervised learning model;

[0009] Risk reconstruction module: Calculates risk adjustment value based on the change value of sequence data and training data and reconstructs the risk function based on it; the training data includes any one or a combination of training times, training time, and training completion degree;

[0010] Minimum risk strategy generation module: Calculate the minimum risk value based on the reconstructed risk function, input the minimum risk value into the supervised learning model, and obtain the minimum risk strategy function.

[0011] In an exemplary embodiment, the screening strategy includes any one or more combinations of the strategy optimization principle and the strategy coverage principle; the strategy optimization principle is to select the data corresponding to the optimal strategy under different states, and the strategy coverage principle is to select the data of all strategies under the same or similar states.

[0012] In an exemplary embodiment, the four-tuple data format is: <state s t 、Behavior a t , the next moment state s t+1 , Risk R t >.

[0013] In an exemplary embodiment, the training of a supervised learning model based on an offline dataset includes:

[0014] Randomly sample an initial state s0 from the offline dataset;

[0015] Randomly initialize the parameters of the policy function π(·), where the input of the policy function is the state s t , the output is behavior a t ;

[0016] The input of the supervised learning model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , Risk R t ;

[0017] Supervised learning models are computed using deep neural networks and maximum likelihood estimation algorithms.

[0018] In an exemplary embodiment, generating sequence data according to a supervised learning model includes:

[0019] Step 1: According to the state s t Randomly sample actions a from the policy function t ~ π(s t );

[0020] Step 2: Set the state s t With behavior a tInput into the supervised learning model to calculate the next moment state s t+1 and risk R t ;

[0021] Step 3: Arrange the quadruple data in chronological order < state s t 、Behavior a t , the next moment state s t+1 , Risk R t >, forming a sequence;

[0022] Step 4: Determine whether the sequence reaches the preset length. If so, the process ends and the sequence data of the preset length is obtained; otherwise, the process returns to step 1.

[0023] In an exemplary embodiment, calculating the risk adjustment value based on the change value of the sequence data and the training data includes:

[0024] Calculating the prediction certainty weight value based on the variance of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the standard deviation of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the variance and standard deviation of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the variance and average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the standard deviation and average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value based on the variance, standard deviation and average change value of the sequence data output by the supervised learning model;

[0025] Calculating the convergence weight value according to the reinforcement learning training time, or calculating the convergence weight value according to the number of reinforcement learning training sessions, or calculating the convergence weight value according to the degree of completion of the reinforcement learning training, or calculating the convergence weight value according to the reinforcement learning training time and the number of training sessions, or calculating the convergence weight value according to the reinforcement learning training time and the degree of completion of the training, or calculating the convergence weight value according to the number of reinforcement learning training sessions and the degree of completion of the training, or calculating the convergence weight value according to the reinforcement learning training time, the number of training sessions and the degree of completion of the training;

[0026] The risk adjustment value is calculated based on the negative correlation between the predicted certainty weight value and the risk, or the risk adjustment value is calculated based on the positive correlation between the convergence weight value and the risk, or the risk adjustment value is calculated based on the negative correlation between the predicted certainty weight value and the risk and the positive correlation between the convergence weight value and the risk.

[0027] In an exemplary embodiment, the risk function is calculated based on the positive correlation between the risk value and the risk adjustment value in the sequence data.

[0028] In an exemplary embodiment, calculating the minimized risk value according to the reconstructed risk function includes:

[0029] According to the state s t Calculate discount factors;

[0030] Calculate the minimized risk value based on the discount factor and SAC algorithm.

[0031] In an exemplary embodiment, the minimum risk policy function is a combination of different states and different behaviors corresponding to the minimized risk value obtained after inputting the minimized risk value into the supervised learning model.

[0032] The risk management system based on offline reinforcement learning of the present invention has the following advantages:

[0033] (1) According to the principle of strategy optimization or strategy coverage, data is collected and screened in a real environment, an offline data set is generated, and a supervised learning model is trained based on the offline data set. Compared with traditional risk management technical solutions, it can directly optimize efficient and feasible intelligent risk management strategies without interacting with the real environment.

[0034] (2) The prediction certainty weight value is calculated based on the variance and / or standard deviation and / or mean change value output by the supervised learning model, the convergence weight value is calculated based on the reinforcement learning training time and / or number of trainings and / or training completion, the risk adjustment value is calculated based on the negative correlation between the prediction certainty weight value and the risk and the positive correlation between the convergence weight value and the risk, and the risk function is reconstructed based on the positive correlation between the risk value and the risk adjustment value. Compared with the traditional fixed risk function technical solution, this can improve the convergence of the function under dynamic uncertainty and effectively avoid the errors caused by the environmental model.

[0035] (3) The minimized risk value is calculated based on the reconstructed risk function, and the strategy function under different behavior and state combinations is trained based on the minimized risk value. Compared with traditional risk management technical solutions, by interacting with the environmental model and considering the uncertainty of time series changes, more accurate decision-making can be effectively achieved based on changes in the environment and itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a risk management system based on offline reinforcement learning according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of training a supervised learning model based on an offline data set in a sequence data generation module of a risk management system according to an embodiment of the present invention;

[0038] Figure 3is a flowchart of generating sequence data according to a supervised learning model in a sequence data generation module of a risk management system of an embodiment of the present application;

[0039] Figure 4 is a flowchart of calculating a risk adjustment value according to a change value of sequence data and training data in a risk reconstruction module of a risk management system of an embodiment of the present application;

[0040] Figure 5 is a flowchart of calculating a minimized risk value according to a reconstructed risk function in a minimum risk strategy generation module of a risk management system of an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present application. These are within the scope of protection of the present application.

[0042] The offline reinforcement learning-based risk management system of the present application has application scenarios including but not limited to financial decision-making, automatic driving obstacle identification and early warning, behavior risk identification, climate monitoring and early warning, weather forecasting and other fields requiring observation and decision-making.

[0043] Taking financial decision-making as an example, an offline reinforcement learning-based risk management system of an embodiment of the present application has a structural schematic diagram as shown in Figure 1 which includes:

[0044] An offline data set generation module: generating an offline data set from offline data according to a screening strategy and a four-tuple data format;

[0045] A sequence data generation module: training a supervised learning model according to the offline data set and generating sequence data according to the supervised learning model;

[0046] A risk reconstruction module: calculating a risk adjustment value according to a change value of sequence data and training data and reconstructing a risk function therefrom; the training data includes any one or a combination of multiple items of training times, training time, and training completion degree;

[0047] A minimum risk strategy generation module: calculating a minimized risk value according to the reconstructed risk function, inputting the minimized risk value into the supervised learning model, and obtaining a minimum risk strategy function.

[0048] In this embodiment, the offline reinforcement learning-based risk management system according to the embodiment of the present application obtains a minimum risk strategy function, and market decisions are made according to the minimum risk strategy function, which can achieve more accurate market strategies in response to market environment and changes in itself.

[0049] The screening strategy includes any one or a combination of the strategy optimization principle and the strategy coverage principle. The strategy optimization principle selects data corresponding to the optimal strategy under different states, while the strategy coverage principle selects data for all strategies under the same or similar states. In this embodiment, offline data is collected in a real market environment, and data corresponding to the optimal strategy under different states is selected, or data for all strategies under the same or similar states is selected.

[0050] The four-tuple data format is: <status s t 、Behavior a t , the next moment state s t+1 , Risk R t >.

[0051] In this embodiment, the offline data set includes a total of four-tuple data at multiple time points (T time points in this embodiment). The definition and specific content of each data element are as follows:

[0052] Time t: According to the market environment requirements of the data, set every day, every half day, every hour, every minute, or every second as a time t;

[0053] Status t : The collected environmental state at each time t. The state can include both on-market and off-market data for the day. On-market data includes but is not limited to opening price, closing price, highest price, lowest price, fundamentals, and financial reports. Off-market data includes but is not limited to weather, policy information, bank interest rates, gold prices, oil prices, etc.

[0054] Behavior a t : The relevant behavior or action taken at each time instant t in the environment. The behavior can include a combination of the stock traded, the transaction price, and the transaction volume.

[0055] Risk R t : Risk is defined as Value at Risk (VaR), which means the maximum expected loss within a given confidence level and a certain time limit, as shown in formula (1):

[0056] (1)

[0057] Where α is the quantile point, X=(s t ,a t ) is the current state-behavior combination. The risk value means the maximum expected loss within a given confidence level α and a certain market period.

[0058] The supervised learning model is trained based on the offline data set, and the flow chart is as follows Figure 2 As shown, the steps include:

[0059] Randomly sample an initial state s0 from the offline dataset;

[0060] Randomly initialize the parameters of the policy function π(·), where the input of the policy function is the state s t , the output is behavior a t ;

[0061] The input of the supervised learning model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , Risk R t ;

[0062] Supervised learning models are computed using deep neural networks and maximum likelihood estimation algorithms.

[0063] In this embodiment, an initial state s0 is randomly sampled from the offline data set; at the same time, the parameters of the policy function π(·) are randomly initialized. Here, the policy function is modeled by a neural network, and the parameters represent all the parameters of the neural network. The input of the policy function is the state s t , the output is a continuous vector a t .

[0064] Based on the collected offline dataset, the supervised learning model T is trained. The input of the supervised learning model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , Risk R t , according to the maximum likelihood estimation, the model is optimized using the formula shown in formula (2):

[0065] (2)

[0066] In practical applications, deep neural networks can be used as supervised learning models, and the parameters in the neural network can be optimized through gradient backpropagation. The likelihood function T(·) uses a Gaussian function, and its logarithm is the mean square error.

[0067] The sequence data is generated according to the supervised learning model, and the flow chart is as follows Figure 3 Shown, including:

[0068] Step 1: According to the state s t Randomly sample actions a from the policy function t ~ π(s t );

[0069] Step 2: Set the state st With behavior a t Input into the supervised learning model to calculate the next moment state s t+1 and risk R t ;

[0070] Step 3: Arrange the quadruple data in chronological order < state s t 、Behavior a t , the next moment state s t+1 , Risk R t >, forming a sequence;

[0071] Step 4: Determine whether the sequence reaches the preset length. If so, the process ends and the sequence data of the preset length is obtained; otherwise, the process returns to step 1.

[0072] In this embodiment, based on the randomly sampled initial state s0, behaviors a0~π(s0) are randomly sampled from the policy function. The initial state s0 and behavior a0 are input into the supervised learning model (2) to obtain the state s1 and risk VaR1 at the next moment, resulting in the first set of four-tuple data <state s0, behavior a0, state s1 at the next moment, risk R1>, which constitutes the first set of data in the sequence. The state s1 at the next moment and the behaviors a1~π(s1) randomly sampled from the policy function are then input into the supervised learning model (2) to obtain the state s2 at the next moment and risk VaR2, resulting in the second set of four-tuple data <state s1, behavior a1, state s2 at the next moment, risk R2>, which constitutes the second set of data in the sequence. The preset length of the sequence is set according to the cumulative error requirement. Here, the preset sequence length k=5, that is, the above process is repeated to train 5 sets of data to obtain a sequence of length 5 starting from the initial state.

[0073] The risk adjustment value is calculated based on the change value of the sequence data and the training data. The flow chart is as follows Figure 4 Shown, including:

[0074] Calculate the prediction certainty weight value based on the variance and / or standard deviation and / or mean change value of the sequence data output by the supervised learning model;

[0075] Calculating a convergence weight value based on reinforcement learning training time and / or number of trainings and / or training completion;

[0076] The risk adjustment value is calculated based on the negative correlation between the prediction certainty weight value and risk and / or the positive correlation between the convergence weight value and risk.

[0077] In this embodiment, the prediction certainty of the supervised learning model is the prediction certainty weight value, and the prediction certainty weight value is calculated according to the variance and / or standard deviation and / or average change value of the sequence data output by the supervised learning model, which is: calculating the prediction certainty weight value according to the variance of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the standard deviation of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the variance and standard deviation of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the variance and average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the standard deviation and average change value of the sequence data output by the supervised learning model, or calculating the prediction certainty weight value according to the variance, standard deviation and average change value of the sequence data output by the supervised learning model, and the prediction certainty weight value is represented by the variable p.

[0078] Training convergence is the convergence weight value. The convergence weight value is calculated based on the reinforcement learning training time and / or training times and / or training completion, which is: calculating the convergence weight value based on the positive correlation between the reinforcement learning training time and the convergence weight value, calculating the convergence weight value based on the positive correlation between the reinforcement learning training times and the convergence weight value, calculating the convergence weight value based on the positive correlation between the reinforcement learning training completion and the convergence weight value, calculating the convergence weight value based on the positive correlation between the reinforcement learning training time and training times and the convergence weight value, calculating the convergence weight value based on the positive correlation between the reinforcement learning training time and training times and the convergence weight value, calculating the convergence weight value based on the positive correlation between the reinforcement learning training time and training completion and the convergence weight value, and calculating the convergence weight value based on the positive correlation between the reinforcement learning training times and training times and training completion and the convergence weight value. The convergence weight value is represented by the variable q.

[0079] The negative correlation between prediction certainty weight and risk means that smaller prediction certainty weights are associated with higher risk, and the corresponding risk adjustment value should be larger. The significance of prediction certainty weights is that when the model's prediction certainty is low, a higher risk is assigned (increasing the risk adjustment value), so that the reinforcement learning algorithm minimizes exploration of high-risk states.

[0080] The positive correlation between convergence weight and risk is that a larger convergence weight indicates a higher risk, and the corresponding risk adjustment value should be larger. The significance of the convergence weight is that, at the initial stage (weak convergence), it reduces the penalty for environmental uncertainty (lowers the risk adjustment value) to encourage reinforcement learning exploration. As the strategy gradually converges (strong convergence), it increases the penalty for uncertainty (increases the risk adjustment value) to promote faster model convergence.

[0081] The risk adjustment value calculated according to the negative correlation between the predicted certainty weight value and the risk and / or the positive correlation between the convergence weight value and the risk is any one of the following: calculating the risk adjustment value according to the negative correlation between the predicted certainty weight value and the risk adjustment value, calculating the risk adjustment value according to the positive correlation between the convergence weight value and the risk adjustment value, and calculating the risk adjustment value according to the negative correlation between the predicted certainty weight value and the risk adjustment value and the positive correlation between the convergence weight value and the risk, and the risk adjustment value is represented by the variable e.

[0082] Different ways to calculate the risk adjustment value are:

[0083] Example A1: Calculating the risk adjustment value based on the negative correlation between the prediction certainty weight value and the risk adjustment value.

[0084] The risk adjustment value is calculated based on the negative correlation between the predicted certainty weight value and the risk adjustment value. Specifically, the predicted certainty weight value p is calculated based on the negative correlation between the variance and / or standard deviation and / or average change value output by the supervised learning model and the predicted certainty weight value; the risk adjustment value e is calculated based on the negative correlation between the predicted certainty weight value p and the risk adjustment value. In one embodiment, the calculated risk adjustment value e=o1·p o2 +o3, where o1, o2 (o1·o2<0), and o3 are calculation coefficients obtained by pre-training. In this embodiment, the variance, standard deviation, or average change value of the output of the statistical supervised learning model is calculated. In this embodiment, the variance is taken as an example, and the variance s=1.5 of the output of a supervised learning model (Gaussian model) is calculated. Based on the negative correlation between the variance and the prediction certainty weight value, the prediction certainty weight value p=0.8 (p=g1·s g2 +g3, where g1, g2, and g3 are the calculation coefficients obtained by prior training and g1·g2<0, where g1=1.2, g2=-1, and g3=0), the calculation coefficients obtained by prior training are o1=1, o2=1, and o3=0, and the calculated risk adjustment value is e=o1·p o2 +o3=1×0.8+0=0.8.

[0085] Example A2: Calculating the risk adjustment value based on the positive correlation between the convergence weight value and the risk adjustment value.

[0086] The risk adjustment value is calculated based on the positive correlation between the convergence weight value and the risk adjustment value. Specifically, the convergence weight value q is calculated based on the positive correlation between the reinforcement learning training time and / or the number of trainings and / or the training completion and the convergence weight value; the risk adjustment value e is calculated based on the positive correlation between the convergence weight value q and the risk adjustment value. In one embodiment, the calculated risk adjustment value e=o4·q o5+o6, wherein o4, o5 (o4-o5>0), o6 are calculation coefficients obtained by prior training. In this embodiment, the reinforcement learning training time, training times and training completion degree are recorded. In this embodiment, the training time is used as a parameter for measuring the convergence of training, and the convergence weight value q is set to be between 0 and 1. As the training time t increases, the convergence weight value q increases. The commonly used activation function is used to realize this relationship, that is, q=sigmoid(t). When the training time t=2, q=0.88, the calculation coefficients o4=1, o5=1 and o6=0 obtained by prior training, and the risk adjustment value e=o4-q o5 +o6=1x0.88+0=0.88.

[0087] The risk adjustment value is calculated according to the negative correlation between the prediction certainty weight value and the risk adjustment value and the positive correlation between the convergence weight value and the risk.

[0088] The risk adjustment value is calculated according to the negative correlation between the prediction certainty weight value and the risk adjustment value and the positive correlation between the convergence weight value and the risk. Specifically, the prediction certainty weight value p is calculated according to the negative correlation between the variance and / or standard deviation and / or average change value output by the supervised learning model and the prediction certainty weight value; the convergence weight value q is calculated according to the positive correlation between the reinforcement learning training time and / or training times and / or training completion degree and the convergence weight value; and the risk adjustment value e is calculated according to the negative correlation between the prediction certainty weight value p and the risk adjustment value and the positive correlation between the convergence weight value q and the risk. In one embodiment, the risk adjustment value e is calculated as e=o7-p o8 +o9-q o10 +o11, wherein o7, o8 (o7-o8<0), o9, o10 (o9-o10>0), o11 are calculation coefficients obtained by prior training. In this embodiment, the variance or standard deviation or average change value output by the supervised learning model is calculated. In this embodiment, the variance is taken as an example, and the variance s=1.5 of the output of a certain supervised learning model (Gaussian model) is calculated. According to the negative correlation between the variance and the prediction certainty weight value, the prediction certainty weight value p=0.8 (p=g1-s g2+g3, where g1, g2, and g3 are the calculation coefficients obtained through prior training and g1·g2<0, where g1=1.2, g2=-1, and g3=0); record the reinforcement learning training time, number of training sessions, and training completion. This embodiment uses training time as a parameter to measure training convergence, sets the convergence weight value q between 0 and 1, and increases with the increase of training time t. This relationship is achieved using a commonly used activation function, i.e., q=sigmoid(t). When training time t=2, q=0.88; the calculation coefficients o7=0.7, o8=1, o9=0.3, o10=1, and o11=0 obtained through prior training, and calculate the risk adjustment value e=o7·p o8 +o9·q o10 +o11 =0.7×0.8+0.3×0.88+0=0.824. In another embodiment, the risk adjustment value is calculated as e=o12·p o13 ·q o14 +o15, where o12, o13 (o12·o13<0), o14 (o13·o14<0), and o15 are calculation coefficients obtained through pre-training. In this embodiment, the variance, standard deviation, or average change value of the supervised learning model output is statistically calculated. Taking variance as an example, this embodiment calculates the variance s=1.5 of the output of a supervised learning model (Gaussian model). Based on the negative correlation between variance and prediction certainty weight value, the prediction certainty weight value p=0.8 (p=g1·s g2 +g3, where g1, g2, and g3 are the calculation coefficients obtained through prior training and g1·g2<0, where g1=1.2, g2=-1, and g3=0); record the reinforcement learning training time, number of training sessions, and training completion. This embodiment uses training time as a parameter to measure training convergence, sets the convergence weight value q between 0 and 1, and increases with the increase of training time t. This relationship is achieved using a commonly used activation function, i.e., q=sigmoid(t). When training time t=2, q=0.88; the calculation coefficients o11=1.2, o12=1, o13=1, and o14=0 obtained through prior training, and calculate the risk adjustment value e=o12·p o13 ·q o14 +o15 =1.2×0.8×0.88+0=0.845.

[0089] The risk function is calculated based on the positive correlation between the risk value and the risk adjustment value in the sequence data. In this embodiment, the risk function is calculated based on the positive correlation between the risk value and the risk adjustment value in the sequence data. The risk adjustment value e and the risk value R calculated according to any one of the embodiments A1 to A3 are t The positive correlation between is used to calculate the risk function, such as any one of formula (3) or formula (4):

[0090] (3)

[0091] (4)

[0092] Among them, f1, f2, f3, f4, f5, and f6 are calculation coefficients obtained through prior training.

[0093] The minimized risk value is calculated based on the reconstructed risk function, and the flow chart is as follows: Figure 5 Shown, including:

[0094] According to the state s t Calculate discount factors;

[0095] Calculate the minimized risk value based on the discount factor and SAC algorithm.

[0096] In this embodiment, the discount factor γ is calculated based on the state t (obtained through risk assessment training under different states), it can generally be taken as 0.95. SAC is a classic reinforcement learning algorithm. Soft Actor-Critic maximizes the entropy-enhanced reward by learning a stochastic policy. This policy maps states to actions and a Q function, which estimates the target value of the current policy and is optimized through approximate dynamic programming. In this way, the SAC algorithm maximizes the entropy-enhanced reward. In this process, SAC treats the target as absolute truth, thereby deriving a better reinforcement learning algorithm with stable performance and sufficiently high sample efficiency. Therefore, the SAC algorithm calculates the minimized risk value by minimizing the function shown in Equation (5).

[0097] (5)

[0098] The minimum risk strategy function is a combination of different states and different behaviors corresponding to the minimized risk value obtained after the minimized risk value is input into the supervised learning model. Input into the supervised learning model to minimize the risk value The corresponding state and output behavior constitute the strategy function π(a t |s t ).

[0099] In another exemplary embodiment, taking the field of autonomous driving as an example, an offline reinforcement learning-based risk management system according to an embodiment of the present invention includes:

[0100] The offline dataset generation module generates an offline dataset based on the offline data filtering strategy and the four-tuple data format. In this embodiment, offline data is collected in a real driving environment, including but not limited to driving conditions, weather conditions, traffic conditions, and building interior environments. The module selects data corresponding to the optimal strategy under different conditions, or selects data for all strategies under the same or similar conditions.

[0101] The offline data set includes four-tuple data at multiple moments (T moments in this embodiment). The four-tuple data at each moment t is: < state s t , driving behavior t , the next moment state s t+1 , Risk R t >. The definitions and specific contents of each data element are as follows:

[0102] Time t: Set every day, every half day, every hour, every minute, or every second as a time t according to the data environment requirements;

[0103] Status t : The collected driving-related environmental status at each time t. In this embodiment, the status may include road surface information, vehicle status information, driver status information, traffic status information, etc.

[0104] Behavior a t : The relevant behaviors or actions performed in the environment at each time instant t. In this embodiment, behaviors can include steering wheel / joystick operation, gear lever operation, light changes, horn sounding, rearview mirror movement, seat belt tightening / untightening, seat adjustment, airbag deployment, etc.

[0105] Risk R t : Risk is defined as the value at risk, which means the maximum loss expected under a given confidence level and a certain driving time.

[0106] Sequence data generation module: trains a supervised learning model based on the offline dataset and generates sequence data based on the supervised learning model;

[0107] Risk reconstruction module: Calculates risk adjustment value based on the change value of sequence data and training data and reconstructs the risk function based on it; the training data includes any one or a combination of training times, training time, and training completion degree;

[0108] The minimum risk strategy generation module: according to the reconstructed risk function, the minimum risk value is calculated, and the minimum risk value is input into the supervised learning model to obtain the minimum risk strategy function. In this embodiment, according to the reconstructed risk function, the minimum risk value is calculated, and the minimum risk value is input into the supervised learning model, and the state corresponding to the minimum risk value and the output behavior constitute the minimum risk strategy function under different state and different behavior combination, and the automatic driving can be performed according to the minimum risk strategy function, which can effectively improve the safety of the automatic driving behavior.

[0109] In another example embodiment, taking the field of climate disaster warning as an example, a risk management system based on offline reinforcement learning is provided, which comprises:

[0110] The offline data set generation module: the offline data is generated into an offline data set according to the screening strategy and the four-tuple data format. In this embodiment, offline data is collected in a real climate environment, which includes but is not limited to topographic environment, monsoon environment, air pressure environment, temperature and humidity environment, etc. The data corresponding to the optimal strategy under different states is selected, or the data under the same or similar state of all strategies is selected.

[0111] The offline data set includes four-tuple data of multiple time points (T time points in this embodiment), and the four-tuple data of each time point t is: <state s t , behavior a t , next time state s t+1 , risk R t >. The definition and specific content of each data element are as follows:

[0112] Time t: according to the environment demand of the data, each day or every half day or every hour or every minute or every second is set as a time t;

[0113] State s t : the collected environment state at each time t. In this embodiment, the state can include temperature, humidity, air pressure, wind power, cloud thickness, precipitation probability, thunderstorm index, etc.

[0114] Behavior a t : the related behavior or action made in the environment at each time t. In this embodiment, the behavior can include climate broadcast, disaster level determination, climate disaster warning, emergency notification, etc.

[0115] Risk R t : risk is defined as the value at risk, which means the maximum loss expected within a given confidence level and a certain time limit.

[0116] The sequence data generation module: the supervised learning model is trained according to the offline data set, and the sequence data is generated according to the supervised learning model;

[0117] Risk reconstruction module: Calculates risk adjustment value based on the change value of sequence data and training data and reconstructs the risk function based on it; the training data includes any one or a combination of training times, training time, and training completion degree;

[0118] Minimum Risk Strategy Generation Module: This module calculates the minimum risk value based on the reconstructed risk function and inputs the minimized risk value into a supervised learning model to obtain a minimum risk strategy function. In this embodiment, the minimum risk value is calculated based on the reconstructed risk function and then input into the supervised learning model. The state corresponding to the minimum risk value and the output behavior are used to construct minimum risk strategy functions for different state and behavior combinations. In this embodiment, climate disaster warnings based on the minimum risk strategy function can effectively improve their accuracy.

[0119] Of course, those skilled in the art should realize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. As long as they are within the scope of the present invention, any changes or modifications to the above embodiments will fall within the scope of protection of the present invention.

Claims

1. A risk management system based on offline reinforcement learning, characterized in that: include: Offline data set generation module: generates offline data sets according to the screening strategy and the four-tuple data format; the four-tuple data format is: <state s t 、Behavior a t , the next moment state s t+1 , Risk R t > In the field of autonomous driving, the state s t Indicates the driving-related environmental status collected at each time t, including road surface information, vehicle status information, driver status information, and traffic status information; the behavior a t Represents the relevant behaviors or actions performed at each time t in the environment, including steering wheel / joystick operation, gear lever operation, light change, horn sounding, rearview mirror movement, seat belt tightening / untightening, seat adjustment, and airbag deployment; or in the field of climate disaster early warning, the state t Represents the environmental conditions collected at each time t, including temperature, humidity, air pressure, wind speed, cloud thickness, precipitation probability, and thunderstorm index; the behavior a t Represents relevant behaviors or actions performed at each time t in the environment, including climate broadcast, disaster level determination, climate disaster warning, and emergency notification; Sequence data generation module: trains a supervised learning model based on the offline dataset and generates sequence data based on the supervised learning model; Risk reconstruction module: Calculates risk adjustment value based on the change value of sequence data and training data and reconstructs the risk function based on it; the training data includes any one or a combination of training times, training time, and training completion degree; The risk adjustment value is calculated based on the change value of the sequence data and the training data, including: calculating the prediction certainty weight value based on the variance and / or standard deviation and / or average change value of the sequence data output by the supervised learning model; calculating the convergence weight value based on the reinforcement learning training time and / or number of trainings and / or training completion; and calculating the risk adjustment value based on the negative correlation between the prediction certainty weight value and the risk and / or the positive correlation between the convergence weight value and the risk; The risk function is calculated based on the positive correlation between the risk value and the risk adjustment value in the sequence data; Minimum risk strategy generation module: Calculate the minimum risk value based on the reconstructed risk function, input the minimum risk value into the supervised learning model, and obtain the minimum risk strategy function.

2. The risk management system based on offline reinforcement learning according to claim 1, characterized in that: The screening strategy includes any one or more combinations of the strategy optimization principle and the strategy coverage principle; the strategy optimization principle is to select the data corresponding to the optimal strategy under different states, and the strategy coverage principle is to select the data of all strategies under the same or similar states.

3. The risk management system based on offline reinforcement learning according to claim 1, characterized in that: The supervised learning model is trained based on the offline data set, including: Randomly sample an initial state s0 from the offline dataset; Randomly initialize the parameters of the policy function π(·), where the input of the policy function is the state s t , the output is behavior a t ; The input of the supervised learning model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , Risk R t ; Supervised learning models are computed using deep neural networks and maximum likelihood estimation algorithms.

4. The risk management system based on offline reinforcement learning according to claim 3, characterized in that: The generating of sequence data according to the supervised learning model includes: Step 1: According to the state s t Randomly sample actions a from the policy function t ~ π(s t ); Step 2: Set the state s t With behavior a t Input into the supervised learning model to calculate the next moment state s t+1 and risk R t ; Step 3: Arrange the quadruple data in chronological order < state s t 、Behavior a t , the next moment state s t+1 , Risk R t >, forming a sequence; Step 4: Determine whether the sequence reaches the preset length. If so, the process ends and the sequence data of the preset length is obtained; otherwise, the process returns to step 1.

5. The risk management system based on offline reinforcement learning according to claim 1, characterized in that: The step of calculating the minimized risk value according to the reconstructed risk function includes: According to the state s t Calculate discount factors; Calculate the minimized risk value based on the discount factor and SAC algorithm.

6. The risk management system based on offline reinforcement learning according to claim 1, characterized in that: The minimum risk strategy function is a combination of different states and different behaviors corresponding to the minimized risk value obtained after the minimized risk value is input into the supervised learning model.

Images