Risk management method and device based on dynamic playback pool and electronic equipment

Abstract

The invention discloses a risk management method and device based on a dynamic playback pool and electronic equipment, and the device comprises a dynamic data set generation module based on a large language model, which is used for generating a dynamic data set according to an offline data set and a real-time environment state vector; the state sequence generation module based on the dynamic data set and the deep neural network is used for training a state transition model and generating a state sequence; the dynamic playback pool updating module based on model uncertainty is used for updating the dynamic playback pool by using a priority queue method; the state transition model optimization module based on the dynamic playback pool is used for updating parameters of the state transition model according to the dynamic experience playback pool and a gradient descent method; and the risk management strategy generation module is used for obtaining a real-time optimized risk strategy function according to the latest state transition model. The method solves the problem that an existing intelligent risk management method cannot continuously optimize according to the dynamic change of a decision-making environment to cause strategy failure.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a risk management method, device and electronic equipment based on a dynamic replay pool. Background Art

[0002] In the field of artificial intelligence, every strategic decision carries risk, which can be mitigated through effective risk management. As AI develops, its application within industries and its involvement in strategic decision-making deepens. Failure to effectively identify and manage risks can lead to immeasurable harm.

[0003] To address the need for risk management, a quantitative, automated, and intelligent risk management approach is proposed, applying rule-based, machine learning, deep learning, and reinforcement learning methods. However, due to the dynamic nature of the decision-making environment, including external interference, environmental changes, and strategic game-playing, strategies trained with historical data struggle to adapt to these dynamics, lack flexibility, and can become ineffective over time. Furthermore, dynamic policy updates using on-policy methods can lead to catastrophic forgetting, resulting in policy failure.

[0004] In order to solve the problem that existing intelligent risk management methods cannot continuously optimize according to the dynamic changes of the decision-making environment, resulting in strategy failure, a risk management method, device and electronic equipment based on dynamic replay pool are proposed. Summary of the Invention

[0005] The embodiments of the present invention provide a risk management method, device, and electronic device based on a dynamic replay pool to at least solve the problem that existing intelligent risk management methods cannot continuously optimize according to dynamic changes in the decision-making environment, resulting in strategy failure.

[0006] According to one embodiment of the present invention, a risk management method based on a dynamic replay pool is proposed, comprising:

[0007] Offline data is filtered based on decision accuracy and / or data coverage to form an offline dataset. A large language model and BERT model are used to output a real-time environment state vector. A dynamic dataset is generated based on the offline dataset and the real-time environment state vector.

[0008] Train a state transition model based on a dynamic data set and a deep neural network, and generate a state sequence based on the state transition model;

[0009] The priority of each data group in the state sequence is calculated according to the variance of the state transition model and the dynamic replay pool is updated using the priority queue method;

[0010] The parameters of the state transition model are updated based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0011] The risk strategy function is optimized in real time based on the latest state transition model.

[0012] In an exemplary embodiment, filtering offline data according to decision accuracy and / or data coverage to form an offline data set includes:

[0013] Calculating the decision accuracy rate based on the benefits of historical decisions and / or the growth rate of benefits of historical decisions and / or the loss rate of historical decisions;

[0014] Calculate data coverage based on the degree of historical decision differences and / or the degree of historical decision duplication;

[0015] Calculate the decision collection weight value based on the decision accuracy and / or data coverage;

[0016] The offline data is filtered according to the order of the decision collection weight values and the preset decision collection weight threshold to form an offline data set.

[0017] In an exemplary embodiment, the method of using a large language model and a BERT model to output a real-time environment state vector includes the steps of:

[0018] Collect real-time external environment data;

[0019] Input external environment data into the large language model to obtain output;

[0020] The embedding vector is calculated based on the output of the large language model and the BERT model, which is the real-time environment state vector.

[0021] In an exemplary embodiment, the dynamic data set is in a five-tuple data format, and the five-tuple data format is: <state s t 、Behavior a t , the next moment state s t+1 , risk t , Real-time environment status I t >.

[0022] In an exemplary embodiment, the training of a state transition model based on a dynamic data set and a deep neural network includes:

[0023] Randomly sample an initial state s0 from the dynamic dataset;

[0024] Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ;

[0025] The input of the state transition model is the state s at each moment in the dynamic data set t and behavior a t , the output is the next moment state s t+1 , risk t ;

[0026] The state transition model is trained using deep neural networks and maximum likelihood estimation algorithms.

[0027] In an exemplary embodiment, the step of calculating the priority of each data group in the state sequence according to the variance of the state transition model includes:

[0028] Calculate the variance of the Gaussian distribution output by the state transition model, which is the variance of the state transition model;

[0029] The mean uncertainty is calculated based on the positive correlation between the variance of the state transition model and the mean uncertainty;

[0030] The priority of each data group in the state sequence is calculated based on the positive correlation between mean uncertainty and priority.

[0031] In an exemplary embodiment, the method of updating the dynamic replay pool using a priority queue includes:

[0032] Pre-configure dynamic replay pools based on the deadline nature of decision events;

[0033] When new samples are obtained through interaction with the environment, the samples with the lowest priority are removed from the dynamic replay pool and replaced with new samples.

[0034] In an exemplary embodiment, updating the parameters of the state transition model according to the dynamic replay pool and the gradient descent method includes:

[0035] Randomly sample an initial state from the dynamic replay pool;

[0036] Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ;

[0037] The input of the state transition model is the state s at each moment in the dynamic data set t and behavior a t , the output is the next moment state s t+1 , risk t+1 ;

[0038] A deep neural network and maximum likelihood estimation algorithm are used to update the state transition model and optimize the parameters in the neural network through gradient back propagation.

[0039] In an exemplary embodiment, the risk strategy function optimized in real time according to the latest state transition model includes:

[0040] Obtain the risk function based on the latest state transition model;

[0041] Calculate the minimum risk value based on the preset calculation coefficient and SAC algorithm;

[0042] The decision function corresponding to the minimum risk value is the real-time optimized risk strategy function.

[0043] According to one embodiment of the present invention, a risk management device based on a dynamic replay pool is proposed, comprising:

[0044] The dynamic data set generation module based on the large language model is used to filter offline data according to the decision accuracy and / or data coverage to form an offline data set, use the large language model and BERT model to output the real-time environment state vector, and generate a dynamic data set based on the offline data set and the real-time environment state vector; the data format of the dynamic data set is <state s t 、Behavior a t , the next moment state s t+1 , risk t , Real-time environment status I t >

[0045] A state sequence generation module based on a dynamic data set and a deep neural network is used to train a state transition model based on the dynamic data set and the deep neural network, and generate a state sequence based on the state transition model;

[0046] A dynamic replay pool update module based on model uncertainty is used to calculate the priority of each data group in the state sequence according to the variance of the state transition model and update the dynamic replay pool using the priority queue method;

[0047] A state transition model optimization module based on the dynamic replay pool is used to update the parameters of the state transition model according to the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0048] The real-time risk management strategy generation module is used to obtain the real-time optimized risk strategy function based on the latest state transition model.

[0049] According to one embodiment of the present invention, an electronic device is provided, including:

[0050] at least one processor;

[0051] and a memory communicatively coupled to the at least one processor;

[0052] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the risk management method based on the dynamic replay pool.

[0053] The risk management method, device, and electronic device based on a dynamic replay pool of the present invention have the following advantages:

[0054] (1) Offline data is filtered based on decision accuracy and / or data coverage to form an offline data set. Compared with traditional risk management and strategy evaluation technical solutions, data with high strategy coverage and high historical returns can be selected without interacting with the actual environment, which facilitates the subsequent rapid and accurate construction of environmental models and strategy evaluation.

[0055] (2) A large language model and a BERT model are used to output the real-time environment state vector and a dynamic data set is generated based on the offline data set and the real-time environment state vector. Compared with the traditional risk management and control technical solutions that are only trained based on historical strategies, this method can consider the information of the external environment in real time and make corresponding strategy adjustments based on the latest information, effectively reducing the strategy failure rate.

[0056] (3) The priority of each data group in the state sequence is calculated based on the variance of the state transition model, and the dynamic replay pool is updated using the priority queue method. Compared with the traditional experience replay pool technical solution, the dynamic replay pool data can be selectively saved according to the credibility and importance of the data samples, which facilitates more effective learning in the strategy optimization process and improves the model accuracy.

[0057] (4) The parameters of the state transition model are updated based on the dynamic replay pool and gradient descent method, and a real-time optimized risk strategy function is obtained based on the latest state transition model. Compared with traditional risk management and control technical solutions, it can more efficiently utilize historical experience and adjust strategies according to the needs of the current environment, thereby improving strategy effectiveness and reducing failure rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 1 is a schematic structural diagram of a risk management device based on a dynamic replay pool according to an embodiment of the present invention;

[0059] Figure 2 This is a flow chart of an embodiment of the present invention for filtering offline data to form an offline data set based on decision accuracy and / or data coverage;

[0060] Figure 3 This is a flowchart of an embodiment of the present invention using a large language model and a BERT model to output a real-time environment state vector;

[0061] Figure 4 is a flow chart of an embodiment of the present invention for training a state transition model based on a dynamic data set and a deep neural network;

[0062] Figure 5 is a flow chart of calculating the priority of each data group in a state sequence according to the variance of a state transition model according to an embodiment of the present invention;

[0063] Figure 6 This is a flow chart of an embodiment of the present invention for updating a dynamic replay pool using a priority queue method;

[0064] Figure 7 is a flow chart of an embodiment of the present invention for updating parameters of a state transition model based on a dynamic replay pool and a gradient descent method;

[0065] Figure 8 This is a flow chart of a risk strategy function that is optimized in real time according to the latest state transition model in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the invention, but are not intended to limit the present invention in any form. It should be noted that those skilled in the art may make several changes and modifications without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0067] The risk management method, device and electronic device based on the dynamic replay pool of the present invention are applicable to scenarios including but not limited to financial decision-making, autonomous driving obstacle identification and warning, behavioral risk identification, climate monitoring and warning, weather forecasting and other risk strategy assessments in areas requiring observation and decision-making.

[0068] Taking financial decision making as an example, a risk management device based on a dynamic replay pool according to an embodiment of the present invention is shown in the structural diagram. Figure 1 Shown, including:

[0069] Dynamic dataset generation module based on a large language model: This module filters offline data based on decision accuracy and / or data coverage to form an offline dataset. It uses the large language model and BERT model to output a real-time environment state vector, and then generates a dynamic dataset based on the offline dataset and the real-time environment state vector.

[0070] State sequence generation module based on dynamic data sets and deep neural networks: trains a state transition model based on dynamic data sets and deep neural networks, and generates state sequences based on the state transition model;

[0071] Dynamic replay pool update module based on model uncertainty: Calculates the priority of each data group in the state sequence according to the variance of the state transition model and updates the dynamic replay pool using the priority queue method;

[0072] State transition model optimization module based on dynamic replay pool: updates the parameters of the state transition model based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0073] Real-time risk management strategy generation module: obtains real-time optimized risk strategy function based on the latest state transition model.

[0074] Taking financial decision-making as an example, a risk management method based on a dynamic replay pool in an embodiment of the present invention includes:

[0075] Offline data is filtered based on decision accuracy and / or data coverage to form an offline dataset. A large language model and BERT model are used to output a real-time environment state vector. A dynamic dataset is generated based on the offline dataset and the real-time environment state vector.

[0076] Train a state transition model based on a dynamic data set and a deep neural network, and generate a state sequence based on the state transition model;

[0077] The priority of each data group in the state sequence is calculated according to the variance of the state transition model and the dynamic replay pool is updated using the priority queue method;

[0078] The parameters of the state transition model are updated based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0079] The risk strategy function is optimized in real time based on the latest state transition model.

[0080] In a preferred embodiment, the offline data is filtered according to the decision accuracy and / or data coverage to form an offline data set. The flow chart is as follows: Figure 2 As shown, the steps include:

[0081] Calculating the decision accuracy rate based on the benefits of historical decisions and / or the growth rate of benefits of historical decisions and / or the loss rate of historical decisions;

[0082] Calculate data coverage based on the degree of historical decision differences and / or the degree of historical decision duplication;

[0083] Calculate the decision collection weight value based on the decision accuracy and / or data coverage;

[0084] The offline data is filtered according to the order of the decision collection weight values and the preset decision collection weight threshold to form an offline data set.

[0085] In this embodiment, historical decision data is collected in an environment, and the decision accuracy is calculated based on the benefits of historical decisions and / or the growth rate of benefits of historical decisions and / or the loss rate of historical decisions, which includes: calculating the decision accuracy based on the positive correlation between the positive benefits of historical decisions and the decision accuracy, calculating the decision accuracy based on the positive correlation between the benefit growth rate of historical decisions and the decision accuracy, calculating the decision accuracy based on the negative correlation between the loss rate of historical decisions and the decision accuracy, calculating the decision accuracy based on the positive correlation between the positive benefits of historical decisions and the benefit growth rate of historical decisions and the decision accuracy, calculating the decision accuracy based on the positive correlation between the positive benefits of historical decisions and the decision accuracy, and the negative correlation between the loss rate of historical decisions and the decision accuracy, calculating the decision accuracy based on the positive correlation between the benefit growth rate of historical decisions and the decision accuracy, and the negative correlation between the loss rate of historical decisions and the decision accuracy, or calculating the decision accuracy based on the positive correlation between the positive benefits of historical decisions and the benefit growth rate of historical decisions and the decision accuracy, and the negative correlation between the loss rate of historical decisions and the decision accuracy, and the decision accuracy is represented by the variable u;

[0086] The calculation of the data coverage rate based on the degree of historical decision difference and / or the degree of historical decision repetition is any one of: calculating the data coverage rate based on a positive correlation between the degree of historical decision difference and the data coverage rate, calculating the data coverage rate based on a negative correlation between the degree of historical decision repetition and the data coverage rate, or calculating the data coverage rate based on a positive correlation between the degree of historical decision difference and the data coverage rate and a negative correlation between the degree of historical decision repetition and the data coverage rate, wherein the data coverage rate is represented by a variable b;

[0087] The decision collection weight value calculated according to the decision accuracy and / or data coverage is calculated based on the positive correlation between the decision collection weight value and the decision accuracy and / or data coverage, and the decision collection weight value is represented by the variable v.

[0088] Different implementation methods for calculating decision collection weight values are shown using Examples A1 to A3 as follows:

[0089] Example A1: Calculate the decision collection weight value according to the decision accuracy rate.

[0090] Collect historical decision data in the environment, calculate the decision accuracy u based on the historical decision benefits and / or historical decision benefit growth rate and / or historical decision loss rate; calculate the decision collection weight v based on the positive correlation between the decision accuracy u and the decision collection weight v. In a preferred embodiment, the decision collection weight v is calculated as e1·u e2+e3, where e1, e2 (e1·e2>0), and e3 are calculation coefficients obtained through pre-training. In this embodiment, the profit and loss data of decisions made in a historical period of time in the market environment are collected, and the average positive profit is calculated to be 0.8 (normalized according to the preset profit threshold). Based on the positive correlation between the positive profit of historical decisions and the decision accuracy, the decision accuracy is calculated to be u=0.8. The calculation coefficients obtained through pre-training e1=1, e2=1, and e3=0 are used to calculate the decision collection weight v=e1·u e2 +e3 =1×0.8+0=0.8.

[0091] Embodiment A2: Calculating decision collection weight values according to data coverage.

[0092] Collect decision data from a period of time in the historical environment, and calculate the data coverage b based on the degree of historical decision differences and / or the degree of historical decision repetition; calculate the decision collection weight v based on the positive correlation between the data coverage b and the decision collection weight v. In a preferred embodiment, the decision collection weight v is calculated as e4·b e5 +e6, where e4, e5 (e4·e5>0), and e6 are calculation coefficients obtained through pre-training. In this embodiment, the number of repetitions of a decision within a historical period in the market environment is 5, and the repetition rate of the historical decision is calculated to be 0.5 (calculated based on the ratio of the number of repetitions to a preset decision repetition threshold). Based on the negative correlation between the historical decision repetition rate and the data coverage rate b, the specificity of the historical strategy is calculated as b=0.45 / 0.5=0.9 (where 0.45 is a calculation coefficient obtained through pre-training). The pre-trained calculation coefficients e4=1, e5=1, and e6=0, and the decision collection weight v=e4·b e5 +e6=1×0.9+0=0.9.

[0093] Example A3: Calculate the decision collection weight value based on the decision accuracy and data coverage.

[0094] Collect decision data from a historical period of time in the environment, calculate the decision accuracy u based on the historical decision benefits and / or historical decision benefit growth rate and / or historical decision loss rate; calculate the data coverage b based on the degree of historical decision differences and / or historical decision repetition; calculate the decision collection weight v based on the positive correlation between the decision accuracy u and the data coverage b and the decision collection weight v. In a preferred embodiment, the decision collection weight v is calculated as e7·u e8 +e9·b e10+e11, where e7, e8, e9, e10, and e11 are calculation coefficients obtained through pre-training. In this embodiment, the profit and loss data of decisions made within a historical period of time in the market environment are collected, and the average positive profit is calculated to be 0.8 (normalized according to a preset profit threshold). Based on the positive correlation between the positive profit of historical decisions and the decision accuracy, the decision accuracy u=0.8 is calculated. The number of repetitions of a certain decision within a historical period of time in the market environment is 5, and the repetition rate of the historical decision is calculated to be 0.5 (calculated based on the ratio of the number of repetitions to the preset decision repetition threshold). Based on the negative correlation between the historical decision repetition rate and the data coverage b, the specificity of the historical strategy is calculated to be b=0.45 / 0.5=0.9 (where 0.45 is a calculation coefficient obtained through pre-training). The pre-trained calculation coefficients e7=0.6, e8=1, e9=0.4, e10=1, and e11=0 are used to calculate the decision collection weight v=e7·u e8 +e9·b e10 +e11=0.6×0.8+0.4×0.9+0=0.84. In another preferred embodiment, the decision collection weight value v=e12·u e13 b e14 +e15, where e12 (e12>0), e13 (e13>0), e14 (e14>0), and e15 are calculation coefficients obtained through pre-training. In this embodiment, profit and loss data for decisions made within a historical period of time in the market environment are collected, and the average positive profit is calculated to be 0.8 (normalized according to a preset profit threshold). Based on the positive correlation between the positive profit of historical decisions and the decision accuracy, the decision accuracy u=0.8 is calculated. The number of repetitions of a decision within a historical period of time in the market environment is 5, and the repetition rate of the historical decision is calculated to be 0.5 (calculated based on the ratio of the number of repetitions to the preset decision repetition threshold). Based on the negative correlation between the historical decision repetition rate and the data coverage b, the specificity of the historical strategy is calculated as b=0.45 / 0.5=0.9 (where 0.45 is a calculation coefficient obtained through pre-training). The pre-trained calculation coefficients e12=1.2, e13=1, e14=1, and e15=0 are used to calculate the decision collection weight v=e12·u e13 b e14 +e15=1.2×0.8×0.9+0=0.864.

[0095] According to the market environment and the decision collection weight threshold V set in advance, the decision collection weight values calculated by the method described in any of Examples A1 to A3 are sorted, and the data corresponding to historical decisions with decision collection weight values greater than the decision collection weight threshold V are screened out as offline data to form an offline data set.

[0096] In a preferred embodiment, the large language model and BERT model are used to output a real-time environment state vector, such as Figure 3 As shown, the steps include:

[0097] Collect real-time external environment data;

[0098] Input external environment data into the large language model to obtain output;

[0099] The embedding vector is calculated based on the output of the large language model and the BERT model, which is the real-time environment state vector.

[0100] In this embodiment, external information is summarized based on a large language model, and the summarized embedding vector is obtained through the Bert model. Since external information may include a large amount of policy documents, financial reports, social media information, etc., if it is directly used as the environment state input, it may contain a large amount of irrelevant information. Due to the large amount of information, it may be difficult for the intelligent agent to fully and effectively utilize the information. In this invention, the power of the large language model is utilized to obtain an effective summary and representation of the external environment through the large language model. The prompt used is:

[0101] Question: Based on "xxx Policy" and "xxx News", please evaluate the impact on xxx.

[0102] It should be noted that it is necessary to obtain information from the Internet in real time through relevant interfaces or crawlers, and the original text of news or policies is also used as input to the language model. Based on the results of the large language model response, the embedding vector is obtained through BERT, which is the real-time environment state vector I t .

[0103] In a preferred embodiment, the five-tuple data format is: <status s t 、Behavior a t , the next moment state s t+1 , risk t , Real-time environment status I t >.

[0104] In this embodiment, the offline data set includes a total of four-tuple data at multiple time points (T time points in this embodiment). The definition and specific content of each data element are as follows:

[0105] Time t: According to the market environment requirements of the data, set every day, every hour, every minute, or every second as a time t;

[0106] Status t: The collected environmental state at each time t. The state can include both on-market and off-market data for the day. On-market data includes but is not limited to opening price, closing price, highest price, lowest price, fundamentals, and financial reports. Off-market data includes but is not limited to weather, policy information, bank interest rates, gold prices, oil prices, etc.

[0107] Behavior a t : The relevant behavior or action taken at each time instant t in the environment. The behavior can include a combination of the stock traded, the transaction price, and the transaction volume.

[0108] Risk t : Risk is defined as Value at Risk (VaR), which means the maximum expected loss within a given confidence level and a certain time limit, as shown in formula (1):

[0109] (1)

[0110] Where α is the quantile point, X=(s t ,a t ) is the current state-behavior combination. The risk value means the maximum expected loss within a given confidence level α and a certain market period.

[0111] Real-time environment status I t : Output real-time market environment state vector based on the large language model and BERT model.

[0112] In a preferred embodiment, the state transition model is trained based on the dynamic data set and the deep neural network, and the flow chart is as follows: Figure 4 As shown, the steps include:

[0113] Randomly sample an initial state s0 from the dynamic dataset;

[0114] Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ;

[0115] The input of the state transition model is the state s at each moment in the dynamic data set t and behavior a t , the output is the next moment state s t+1 , risk t ;

[0116] The state transition model is trained using deep neural networks and maximum likelihood estimation algorithms.

[0117] In this embodiment, an initial state s0 is randomly sampled from the dynamic data set; at the same time, the parameters of the decision function π(·) are randomly initialized. Here, the decision function is modeled by a neural network, and the parameters represent all the parameters of the neural network. The input of the decision function is the state s t , the output is a continuous vector a t .

[0118] According to the collected dynamic data set, the state transition model T is trained. The input of the model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , risk t , according to the maximum likelihood estimation, the model is optimized using the formula shown in formula (2):

[0119] (2)

[0120] In actual situations, a deep neural network can be used as a supervised learning model, and a Gaussian model can be used as the last layer of the network. The mean and variance of the prediction results are output at the same time, and the parameters in the neural network are optimized through gradient backpropagation.

[0121] In a preferred embodiment, the priority of each data group in the state sequence is calculated based on the variance of the state transition model, and the flow chart is as follows: Figure 5 Shown, including:

[0122] Calculate the variance of the Gaussian distribution output by the state transition model, which is the variance of the state transition model;

[0123] The mean uncertainty is calculated based on the positive correlation between the variance of the state transition model and the mean uncertainty;

[0124] The priority of each data group in the state sequence is calculated based on the positive correlation between mean uncertainty and priority.

[0125] In this embodiment, the state transition model output is a Gaussian distribution. The variance of the Gaussian distribution is the variance of the state transition model. The variance of the state transition model is represented by the variable m, and the mean uncertainty is represented by the variable q. The mean uncertainty is calculated based on the positive correlation between the variance of the state transition model and the mean uncertainty, that is, mean uncertainty q = g1·m g2 +g3, where g1, g2 (g1·g2>0), and g3 are calculation coefficients obtained through prior training;

[0126] The priority is represented by the variable n. The priority of each data group in the state sequence is calculated based on the positive correlation between the mean uncertainty and the priority, that is, priority n=g4·q g5+g6, where g4, g5 (g4·g5>0), and g6 are calculation coefficients obtained through prior training.

[0127] In a preferred embodiment, the method of using the priority queue to update the dynamic playback pool is as shown in the flow chart. Figure 6 Shown, including:

[0128] Pre-configure dynamic replay pools based on the deadline nature of decision events;

[0129] When new samples are obtained through interaction with the environment, the samples with the lowest priority are removed from the dynamic replay pool and replaced with new samples.

[0130] In this embodiment, a dynamic replay pool is pre-configured according to the deadline nature of the decision event. The dynamic replay pool is a fixed-size cache, for example, a size of 100,000, and each sample stored therein is (current state, current action, state at next moment, risk at next moment, real-time environment state).

[0131] Based on this dynamic replay pool, a sample is generated each time the system interacts with the environment. Traditional methods directly place samples into the dynamic replay pool and use the replay pool as a queue to replace previous samples. In this embodiment, a priority queue is used. That is, when a new sample is generated by interacting with the environment, the sample with the lowest priority in the dynamic replay pool is removed and replaced by the new sample.

[0132] In a preferred embodiment, the parameters of the state transition model are updated according to the dynamic replay pool and gradient descent method, as shown in the flow chart. Figure 7 Shown, including:

[0133] Randomly sample an initial state from the dynamic replay pool;

[0134] Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ;

[0135] The input of the state transition model is the state s at each moment in the dynamic data set t and behavior a t , the output is the next moment state s t+1 , risk t+1 ;

[0136] A deep neural network and maximum likelihood estimation algorithm are used to update the state transition model and optimize the parameters in the neural network through gradient back propagation.

[0137] In this embodiment, an initial state s0 is randomly sampled from the dynamic replay pool; at the same time, the parameters of the decision function π(·) are randomly initialized. Here, the decision function is modeled by a neural network, and the parameters represent all the parameters of the neural network. The input of the decision function is the state s t , the output is a continuous vector a t .

[0138] Update the state transition model. The input of the updated state transition model is the state s at each moment t and behavior a t , the output is the next moment state s t+1 , risk t+1 , according to the maximum likelihood estimation, the model is optimized using the formula shown in formula (3):

[0139] (3)

[0140] In a preferred embodiment, the risk strategy function is optimized in real time according to the latest state transition model, as shown in the flow chart. Figure 8 As shown, the steps include:

[0141] Obtain the risk function based on the latest state transition model;

[0142] Calculate the minimum risk value based on the preset calculation coefficient and SAC algorithm;

[0143] The decision function corresponding to the minimum risk value is the real-time optimized risk strategy function.

[0144] In this embodiment, the risk function r is obtained based on the latest state transition model. t ; The calculation coefficient γ is obtained based on the risk assessment training under different conditions t , in this embodiment, it is taken as 0.95.

[0145] SAC is a classic reinforcement learning algorithm. Soft Actor-Critic maximizes the entropy-enhanced reward by learning a stochastic policy that maps states to actions and a Q-function. This function estimates the target value of the current policy and optimizes it through approximate dynamic programming. In this way, the SAC algorithm maximizes the entropy-enhanced reward. In this process, SAC treats the target as the absolute truth, thereby deriving a better reinforcement learning algorithm with stable performance and sufficiently high sample efficiency. Therefore, according to the preset calculation coefficient γ t The minimum value of risk is calculated by the SAC algorithm, as shown in formula (4):

[0146] (4)

[0147] The decision function π(a t |s t ) is the risk strategy function optimized in real time.

[0148] In another exemplary embodiment, taking the field of autonomous driving as an example, a risk management device based on a dynamic replay pool according to an embodiment of the present invention includes:

[0149] Dynamic dataset generation module based on a large language model: This module filters offline data based on decision accuracy and / or data coverage to form an offline dataset. It uses the large language model and BERT model to output a real-time environment state vector, and then generates a dynamic dataset based on the offline dataset and the real-time environment state vector.

[0150] State sequence generation module based on dynamic data sets and deep neural networks: trains a state transition model based on dynamic data sets and deep neural networks, and generates state sequences based on the state transition model;

[0151] Dynamic replay pool update module based on model uncertainty: Calculates the priority of each data group in the state sequence according to the variance of the state transition model and updates the dynamic replay pool using the priority queue method;

[0152] State transition model optimization module based on dynamic replay pool: updates the parameters of the state transition model based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0153] Real-time risk management strategy generation module: obtains real-time optimized risk strategy function based on the latest state transition model.

[0154] In this embodiment, offline data is collected in a real driving environment.

[0155] Based on a large language model, external information is summarized and embedded using the Bert model. Since external information may include a large amount of road traffic information, autonomous driving news, and other information, if it is directly used as the environmental state input, it may contain a large amount of irrelevant information. Furthermore, due to the large amount of information, it may be difficult for the agent to fully and effectively utilize the information. In this invention, the capabilities of the large language model are utilized to obtain an effective summary and representation of the external environment. The prompt used is:

[0156] Question: Based on the "xxx Road Traffic Safety Law" and "xxx Traffic News", please evaluate the impact on autonomous driving.

[0157] It should be noted that it is necessary to obtain real-time information from the Internet through relevant interfaces or crawlers, and the original text of news or traffic information is also used as input to the language model. Based on the results of the large language model response, the embedding vector is obtained through BERT, which is the real-time driving environment state vector I t .

[0158] A large language model and BERT model are used to output a real-time driving environment state vector. A dynamic dataset is generated based on the offline dataset and the real-time driving environment state vector. The real driving environment includes, but is not limited to, the driving environment, weather conditions, traffic conditions, and building interior environments. The dynamic dataset includes five-tuple data at multiple time points (T time points in this embodiment). The five-tuple data at each time point t is: < state s t 、Behavior a t , the next moment state s t+1 , risk t , Real-time environment status I t >. The definitions and specific contents of each data element are as follows:

[0159] Time t: Set every day, every half day, every hour, every minute, or every second as a time t according to the data environment requirements;

[0160] Status t : At each moment t, the collected driving-related environmental states include road surface information, vehicle status information, driver status information, traffic status information, etc.

[0161] Behavior a t : The relevant behaviors or actions performed at each time t in the environment can include steering wheel / joystick operation, gear lever operation, light change, horn sounding, rearview mirror movement, seat belt tightening / untightening, seat adjustment, airbag deployment, etc.

[0162] Risk t : Risk is defined as the value at risk, which means the maximum loss expected under a given confidence level and a certain driving time.

[0163] Real-time environment status I t : Output real-time driving environment state vector based on the large language model and BERT model.

[0164] The state transition model is trained based on a dynamic data set, and is updated in real time based on a prioritized dynamic replay pool. A real-time optimized risk strategy function is obtained based on the latest state transition model, allowing the autonomous driving system or the driver to adjust the optimal driving strategy based on real-time driving environment changes, thereby reducing risks and improving safety.

[0165] In another exemplary embodiment, taking the field of climate disaster early warning as an example, a risk management device based on a dynamic replay pool in an embodiment of the present invention includes:

[0166] Dynamic dataset generation module based on a large language model: This module filters offline data based on decision accuracy and / or data coverage to form an offline dataset. It uses the large language model and BERT model to output a real-time environment state vector, and then generates a dynamic dataset based on the offline dataset and the real-time environment state vector.

[0167] State sequence generation module based on dynamic data sets and deep neural networks: trains a state transition model based on dynamic data sets and deep neural networks, and generates state sequences based on the state transition model;

[0168] Dynamic replay pool update module based on model uncertainty: Calculates the priority of each data group in the state sequence according to the variance of the state transition model and updates the dynamic replay pool using the priority queue method;

[0169] State transition model optimization module based on dynamic replay pool: updates the parameters of the state transition model based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time;

[0170] Real-time risk management strategy generation module: obtains real-time optimized risk strategy function based on the latest state transition model.

[0171] In this embodiment, offline data is collected in a real climate environment.

[0172] Based on a large language model, external information is summarized and embedded using the Bert model. Since external information may include a large amount of climate news, weather information, climate warning news, and climate-related media, if it is directly used as environmental state input, it may contain a large amount of irrelevant information. Furthermore, due to the large amount of information, it may be difficult for the agent to fully and effectively utilize the information. In this invention, the power of a large language model is utilized to effectively summarize and represent the external environment. The prompt used is:

[0173] Question: Based on the "xxx Weather Forecast" and "xxx Climate Disaster Alert", please evaluate the impact on disaster warning.

[0174] It should be noted that it is necessary to obtain real-time information from the Internet through relevant interfaces or crawlers, and the original text of news or traffic information is also used as input to the language model. Based on the results of the large language model response, the embedding vector is obtained through BERT, which is the real-time climate environment state vector I t .

[0175] A large language model and BERT model are used to output the real-time driving climate environment state vector. A dynamic dataset is generated based on the offline dataset and the real-time climate environment state vector. The real climate environment includes but is not limited to topographic environment, monsoon environment, air pressure environment, temperature and humidity environment, etc. The dynamic dataset includes a total of five-tuple data at multiple time points (T time points in this embodiment). The five-tuple data at each time point t is: < state s t 、Behavior a t , the next moment state s t+1 , risk t , Real-time environment status I t >. The definitions and specific contents of each data element are as follows:

[0176] Time t: Set every day, every half day, every hour, every minute, or every second as a time t according to the data environment requirements;

[0177] Status t : At each time t, the collected environmental conditions include temperature, humidity, air pressure, wind speed, cloud thickness, precipitation probability, thunderstorm index, etc.

[0178] Behavior a t : Related behaviors or actions performed at each moment t in the environment, including climate broadcasting, disaster level determination, climate disaster warning, emergency notification, etc.

[0179] Risk t : Risk is defined as the value at risk, which means the maximum expected loss within a given confidence level and a certain time limit.

[0180] Real-time environment status I t : Output real-time driving environment state vector based on the large language model and BERT model.

[0181] The state transition model is trained according to the dynamic data set, and the state transition model is updated in real time based on the priority dynamic replay pool. The real-time optimized risk strategy function is obtained according to the latest state transition model, which facilitates the disaster warning system or warning personnel to adjust the optimal climate disaster warning strategy according to the real-time climate and environmental changes, thereby improving the accuracy of the warning.

[0182] According to one embodiment of the present invention, an electronic device is provided, including:

[0183] at least one processor;

[0184] and a memory communicatively coupled to the at least one processor;

[0185] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the risk management method based on the dynamic replay pool.

[0186] The electronic device may include a processor, a memory, a communication interface and a bus.

[0187] A computer program stored in the memory and executable on the processor.

[0188] The memory includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory may be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory may include both an internal storage unit of the electronic device and an external storage device. The memory can be used not only to store application software installed in the electronic device and various types of data, such as various types of environmental data and the code of risk management programs, but also to temporarily store data that has been output or is about to be output.

[0189] In some embodiments, the processor may be comprised of an integrated circuit, such as a single packaged integrated circuit or a combination of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor is the control core (control unit) of the electronic device, connecting the various components of the electronic device using various interfaces and circuits. It executes programs or modules stored in the memory and accesses data stored in the memory to perform various functions of the electronic device and process data.

[0190] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WIFI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0191] The bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory and at least one processor.

[0192] The above description of the structure of the electronic device does not constitute a limitation on the electronic device. The electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0193] For example, although not shown, the electronic device may further include a power source (e.g., a battery) to power various components. Preferably, the power source may be logically connected to the at least one processor via a power management device, thereby enabling functions such as charge management, discharge management, and power consumption management via the power management device. The power source may further include any of one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like. The electronic device may also include various sensors, Bluetooth modules, WiFi modules, and the like, which are not further detailed here.

[0194] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WIFI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0195] Optionally, the electronic device may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or display unit, and is used to display information processed in the electronic device and to display a visual user interface.

[0196] Of course, those skilled in the art should realize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. As long as they are within the scope of the present invention, any changes or modifications to the above embodiments will fall within the scope of protection of the present invention.

Claims

1. A risk management method based on a dynamic replay pool, characterized in that: include: Offline data is filtered according to decision accuracy and / or data coverage to form an offline data set, a large language model and a BERT model are used to output a real-time environment state vector, and a dynamic data set is generated based on the offline data set and the real-time environment state vector; the data format of the dynamic data set is <state s t ,Behavior a t , next moment status s t+1 ,risk r t , real-time environmental status I t > Train a state transition model based on a dynamic data set and a deep neural network, and generate a state sequence based on the state transition model; The priority of each data group in the state sequence is calculated according to the variance of the state transition model and the dynamic replay pool is updated using the priority queue method; The parameters of the state transition model are updated based on the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time; The risk strategy function is optimized in real time based on the latest state transition model.

2. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The offline data is filtered according to the decision accuracy and / or data coverage to form an offline data set, including: Calculating the decision accuracy rate based on the benefits of historical decisions and / or the growth rate of benefits of historical decisions and / or the loss rate of historical decisions; Calculate data coverage based on the degree of historical decision differences and / or the degree of historical decision duplication; Calculate the decision collection weight value based on the decision accuracy and / or data coverage; The offline data is filtered according to the order of the decision collection weight values and the preset decision collection weight threshold to form an offline data set.

3. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The method of using a large language model and a BERT model to output a real-time environment state vector includes the following steps: Collect real-time external environment data; Input external environment data into the large language model to obtain output; The embedding vector is calculated based on the output of the large language model and the BERT model, which is the real-time environment state vector.

4. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The state transition model is trained based on a dynamic data set and a deep neural network, including: Randomly sample an initial state from the dynamic dataset s 0 ; Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ; The input of the state transition model is the state at each moment in the dynamic data set s t and behavior a t , the output is the state at the next moment s t+1 ,risk r t ; The state transition model is trained using deep neural networks and maximum likelihood estimation algorithms.

5. The risk management method based on dynamic replay pool according to claim 1, characterized in that: Calculating the priority of each data group in the state sequence according to the variance of the state transition model includes: Calculate the variance of the Gaussian distribution output by the state transition model, which is the variance of the state transition model; The mean uncertainty is calculated based on the positive correlation between the variance of the state transition model and the mean uncertainty; The priority of each data group in the state sequence is calculated based on the positive correlation between mean uncertainty and priority.

6. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The method of using a priority queue to update the dynamic replay pool includes: Pre-configure dynamic replay pools based on the deadline nature of decision events; When new samples are obtained through interaction with the environment, the samples with the lowest priority are removed from the dynamic replay pool and replaced with new samples.

7. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The updating of the parameters of the state transition model according to the dynamic replay pool and the gradient descent method includes: Randomly sample an initial state from the dynamic replay pool; Randomly initialize the parameters of the decision function π(·), where the input of the decision function is the state s t , the output is behavior a t ; The input of the state transition model is the state at each moment in the dynamic data set s t and behavior a t , the output is the state at the next moment s t+1 ,risk r t+1 ; A deep neural network and maximum likelihood estimation algorithm are used to update the state transition model and optimize the parameters in the neural network through gradient back propagation.

8. The risk management method based on dynamic replay pool according to claim 1, characterized in that: The risk strategy function optimized in real time according to the latest state transition model includes: Obtain the risk function based on the latest state transition model; Calculate the minimum risk value based on the preset calculation coefficient and SAC algorithm; The decision function corresponding to the minimum risk value is the real-time optimized risk strategy function.

9. A risk management device based on a dynamic replay pool, characterized in that: include: The dynamic data set generation module based on the large language model is used to filter offline data according to the decision accuracy and / or data coverage to form an offline data set, use the large language model and BERT model to output the real-time environment state vector, and generate a dynamic data set based on the offline data set and the real-time environment state vector; the data format of the dynamic data set is <state s t ,Behavior a t , next moment status s t+1 ,risk r t , real-time environmental status I t > A state sequence generation module based on a dynamic data set and a deep neural network is used to train a state transition model based on the dynamic data set and the deep neural network, and generate a state sequence based on the state transition model; A dynamic replay pool update module based on model uncertainty is used to calculate the priority of each data group in the state sequence according to the variance of the state transition model and update the dynamic replay pool using the priority queue method; A state transition model optimization module based on the dynamic replay pool is used to update the parameters of the state transition model according to the dynamic replay pool and gradient descent method, so that the state transition model reflects the latest dynamic replay pool in real time; The real-time risk management strategy generation module is used to obtain the real-time optimized risk strategy function based on the latest state transition model.

10. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the risk management method based on the dynamic replay pool as described in any one of claims 1-8.

Images