Reinforcement learning-based risk prediction method, device and equipment, and storage medium

Abstract

The application relates to the fields of risk prediction and digital medicine, and specifically discloses a risk prediction method and device based on reinforcement learning, equipment and a storage medium. Risk influencing factors and corresponding historical data of a prediction object are acquired; a network topology model is obtained by processing based on a self-attention network model and a gated recurrent unit; dynamic characteristics and static characteristics of the historical data are acquired based on the network topology model to generate space-time characteristics; and a risk prediction result is obtained based on a risk prediction model. The risk prediction method based on reinforcement learning can process risk influencing factors and historical data of a prediction object to generate a network topology model, and then extract space-time characteristics from data in the network topology model, realize the combination of characteristics in time and space, effectively improve the accuracy of risk prediction, process space-time characteristics through a risk prediction model, and output a risk prediction result, effectively improving the efficiency of risk prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of risk prediction and digital medical technology, and particularly relates to a risk prediction method and device based on reinforcement learning, equipment and a storage medium. BACKGROUND

[0002] With the development of society and the progress of technology, risk management plays an increasingly important role in various fields. In the medical field, infectious disease prevention has become a necessary task. For example, in infectious disease prevention, risk prediction can help doctors better analyze personnel flow groups and social relationships, better predict disease risks, and thus better perform treatment and prevention. However, traditional risk prediction methods mainly include risk prediction technology based on machine learning and risk prediction technology based on behavior analysis, which mainly rely on historical data and statistical analysis, and predict future risks that may occur by analyzing events that have already occurred. However, excessive reliance on historical data sets makes the quality and quantity of data sets have a greater impact on prediction accuracy. In actual application, the integrity and quality of data are often not guaranteed, so the accuracy of traditional prediction methods is low. For example, some important data may not be collected due to technical limitations or policy reasons, or the data quality is poor, and there are errors and uncertainties. Moreover, for new prediction samples, the model needs to be retrained, which consumes a lot of time and resources; traditional risk prediction methods ignore the impact of emerging risks and uncertainty factors.

[0003] Traditional risk prediction methods are mainly based on historical data and past experience, but for emerging risks and uncertainty factors, there is often not enough data and experience to analyze and predict them; complex mathematical models and algorithms often need to be established, which requires a lot of calculation and analysis work, and it is difficult to apply to actual risk management, and there are problems of model complexity and application difficulty. These problems reduce the accuracy and efficiency of risk prediction.

[0004] Therefore, how to improve the accuracy and efficiency of risk prediction has become a problem to be solved. SUMMARY

[0005] The present application provides a risk prediction method, device, equipment and storage medium based on reinforcement learning to improve the accuracy and efficiency of risk prediction.

[0006] In a first aspect, the present application provides a risk prediction method based on reinforcement learning, the method comprising:

[0007] obtaining at least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor;

[0008] The risk influencing factor and the historical data are processed based on a self-attention network model and a gated recurrent unit to obtain a network topology model of the prediction object.

[0009] Based on the network topology model, dynamic characteristics and static characteristics of the historical data are obtained, and based on the dynamic characteristics and the static characteristics, spatiotemporal characteristics of the historical data are generated.

[0010] Based on a preset risk prediction model and the spatiotemporal characteristics, the risk of the prediction object is predicted to obtain a risk prediction result of the prediction object.

[0011] In a second aspect, the present application further provides a risk prediction device based on reinforcement learning, the device comprising:

[0012] A historical data acquisition module is configured to acquire at least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor.

[0013] A network topology model acquisition module is configured to process the risk influencing factor and the historical data based on a self-attention network model and a gated recurrent unit to obtain a network topology model of the prediction object.

[0014] A spatiotemporal characteristic generation module is configured to acquire dynamic characteristics and static characteristics of the historical data based on the network topology model, and generate spatiotemporal characteristics of the historical data based on the dynamic characteristics and the static characteristics.

[0015] A risk prediction result acquisition module is configured to predict the risk of the prediction object based on a preset risk prediction model and the spatiotemporal characteristics to obtain a risk prediction result of the prediction object.

[0016] In a third aspect, the present application further provides a computer device, which comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program and realize the risk prediction method based on reinforcement learning as described above when executing the computer program.

[0017] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program; the computer program is executed by a processor to make the processor realize the risk prediction method based on reinforcement learning as described above.

[0018] The application discloses a risk prediction method and device based on reinforcement learning, equipment and a storage medium. At least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor are acquired. The risk influencing factor and the historical data are processed based on a self-attention network model and a gated recurrent unit, and a network topology model of the prediction object is acquired. Dynamic characteristics and static characteristics of the historical data are acquired based on the network topology model, and space-time characteristics of the historical data are generated based on the dynamic characteristics and the static characteristics. The risk of the prediction object is predicted based on a preset risk prediction model and the space-time characteristics, and a risk prediction result of the prediction object is acquired. Compared with a machine learning method, the risk prediction method based on reinforcement learning can process the risk influencing factor of the prediction object and historical data thereof, generate a network topology model, and then extract space-time characteristics of data in the network topology model, realize the combination of characteristics of data in time and space, process the space-time characteristics through a risk prediction model, output a risk prediction result, and effectively improve the accuracy and efficiency of risk prediction. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a first embodiment schematic flow chart of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0021] Figure 2 is a space-time feature extraction schematic diagram of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0022] Figure 3 is a second embodiment schematic flow chart of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0023] Figure 4 is a third embodiment schematic flow chart of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0024] Figure 5 is a fourth embodiment schematic flow chart of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0025] Figure 6 is a model training schematic diagram of a risk prediction method based on reinforcement learning provided by an embodiment of the application;

[0026] Figure 7 A schematic block diagram of a risk prediction device based on reinforcement learning provided for an embodiment of the present application;

[0027] Figure 8 A structural schematic block diagram of a computer device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] The flowcharts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0030] It should be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0031] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0032] Embodiments of the present application provide a risk prediction method, device and equipment based on reinforcement learning and a storage medium. The risk prediction method based on reinforcement learning can be applied to a server. By obtaining the spatio-temporal features of data, the risk prediction model is used to process the spatio-temporal features to obtain the risk prediction result, which effectively improves the accuracy and efficiency of risk prediction. The server can be an independent server or a server cluster.

[0033] Some embodiments of the present application will be described in detail below in combination with the drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1is a schematic flowchart of a risk prediction method based on reinforcement learning provided by an embodiment of the present application. The risk prediction method based on reinforcement learning can be applied in a server, for obtaining spatio-temporal features of data, processing the spatio-temporal features using a risk prediction model, and obtaining a risk prediction result, thereby effectively improving the accuracy and efficiency of risk prediction.

[0035] As shown in Figure 1 The risk prediction method based on reinforcement learning specifically includes steps S101 to S104.

[0036] S101, at least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor are obtained.

[0037] In one embodiment, the prediction object can be the risk of a patient suffering from a certain disease, and the risk influencing factors can be the age, health condition, genetic condition, and history of the patient, for example, the incidence of diabetes is related to age, living environment, blood pressure, and blood lipids, etc., so that the age, living environment, blood pressure, and blood lipids can be used as the risk influencing factors of the patient suffering from diabetes, and the corresponding data is obtained as the historical data. The prediction object can also be the risk prediction of infectious diseases, and the infectious disease risk prediction can use the virus infectivity, virus lethality, and personnel activity as different influencing factors, and the financial risk prediction can use economic indicators, market sentiment, and policy changes as different risk influencing factors.

[0038] In one embodiment, the prediction object can not only be the disease risk and infectious disease risk in the medical field, but also the internet risk and financial risk. The risk influencing factors are determined according to the prediction object, and the historical data of each risk influencing factor is obtained.

[0039] In one embodiment, after the risk influencing factors of the prediction object are determined, the historical data of each influencing factor is obtained, for example, the virus infectivity and lethality in the infectious disease risk prediction can gradually decrease with the spread of the virus, and the personnel activity can be the flow data of personnel in a certain period of time.

[0040] S102, the risk influencing factors and the historical data are processed based on a self-attention network model and a gated recurrent unit, to obtain a network topology model of the prediction object.

[0041] In one embodiment, the network topology model is divided into a network topology structure representation module and a time series data representation module.

[0042] In a specific embodiment, the graph attention network (GAT) model, which is a type of self-attention network model, is used to model the network topology of each entity (i.e., risk influencing factors). Each entity is represented as a node in the network topology, and the relationship between entities is represented as an edge in the network topology, thus obtaining the network topology representation module of the network topology model.

[0043] In a specific embodiment, a gated recurrent unit (GRU) is used to model the data of each entity, representing the data of each entity at different time points as a time-series data sequence, thereby obtaining the time-series data representation module of the network topology model.

[0044] In a specific embodiment, the network topology structure is combined with the time-series data sequence of historical data to obtain the network topology model.

[0045] S103. Based on the network topology model, obtain the dynamic and static features of the historical data, and generate the spatiotemporal features of the historical data based on the dynamic and static features.

[0046] In one embodiment, such as Figure 2 As shown, extracting the temporal and spatial characteristics of historical data of each entity based on the network topology model is beneficial for solving the problem of the high correlation and complexity of entity data in time and space.

[0047] In one embodiment, GRU is used to encode the data of each entity at different points in time to obtain the dynamic characteristics of the data of each entity at different times.

[0048] In one embodiment, a self-attention network is used to select features from dynamic features to obtain static features. The self-attention network can adaptively select features and attention weights, compressing and filtering dynamic features to obtain static features.

[0049] In one embodiment, dynamic features and static features are concatenated using tensors to obtain the spatiotemporal features of the entity's historical data.

[0050] S104. Based on the preset risk prediction model and the spatiotemporal characteristics, the risk of the prediction object is predicted to obtain the risk prediction result of the prediction object.

[0051] The risk of the prediction object is predicted based on the preset risk prediction model and the spatiotemporal features, and a risk prediction result of the prediction object is obtained, including: processing the spatiotemporal features based on the risk prediction model to obtain a predicted risk of each risk influencing factor; and generating the risk prediction result of the prediction object based on the predicted risk of each risk influencing factor.

[0052] After the risk prediction result of the prediction object is obtained by predicting the risk of the prediction object based on the preset risk prediction model and the spatiotemporal features, the method further includes: generating a risk management strategy based on the risk type and the occurrence probability, for reference by a user.

[0053] In one embodiment, each node of the network topology is taken as an agent, and the agent and the spatiotemporal features of historical data of each agent are input into the risk prediction model to first generate a predicted risk of each agent, and then obtain the risk prediction result of the prediction object in combination with the predicted risk of each agent.

[0054] In one embodiment, the risk prediction result includes a risk type and an occurrence probability corresponding to the risk type.

[0055] In one embodiment, a corresponding risk management strategy can be generated according to the risk type and the risk prediction result, for reference by a user. For example, a hospital physical examination center can generate and save a health record of a patient according to data of each visit or physical examination of the patient, generate a disease risk of the patient according to data in the health record and latest physical examination data of the patient, and generate a disease prevention measure according to the type of the disease, and the patient can perform corresponding disease prevention according to the disease risk and the prevention measure in the physical examination report.

[0056] In one embodiment, before predicting the future risk of the prediction object using the risk prediction model, a multi-agent reinforcement learning decision model is trained according to historical data and historical risk records of the prediction object to obtain a risk prediction model that can be used.

[0057] In one embodiment, the traditional risk prediction over-reliance on historical data sets and historical experience, but the collection of historical data may not be complete and cannot predict the newly generated risk with historical experience, for example, the patient's examination data in a hospital may not be recorded by another hospital, which will cause the hospital to make errors in predicting the risk of the patient's disease; when a new disease occurs, it is impossible to judge the risk of the disease according to historical experience. The risk prediction method based on reinforcement learning of the application can accurately obtain the spatio-temporal characteristics between data in the historical data, and obtain the most possible predicted risk through reinforcement learning. The spatio-temporal characteristics of the data can be accurately extracted from the historical examination data of the patient, and then the disease prediction result and the disease prevention measure can be obtained according to the risk prediction model.

[0058] Please refer to Figure 3 , Figure 3 is a schematic flow chart of a risk prediction method based on reinforcement learning provided by an embodiment of the application. The risk prediction method based on reinforcement learning can be applied to a server, for obtaining spatio-temporal characteristics of data, processing the spatio-temporal characteristics by using a risk prediction model, and obtaining a risk prediction result, thereby effectively improving the accuracy and efficiency of risk prediction.

[0059] As shown in Figure 3 , the step S102 of the risk prediction method based on reinforcement learning specifically includes steps S201 to S203.

[0060] S201, based on the graph attention network model, obtaining the correlation between the risk influencing factors, and based on the correlation, constructing a network topology structure of the risk influencing factors;

[0061] S202, based on the gating recurrent unit, processing the historical data of each risk influencing factor to obtain a time series data sequence corresponding to the historical data;

[0062] S203, based on the network topology structure and the time series data sequence, generating the network topology model.

[0063] In one embodiment, when a risk prediction task is involved, constructing a network model based on space-time graph can improve the modeling capability of network topology structure and data analysis. This model can use GAT to model the entity network topology structure and use GRU to model the entity data. Specifically, this model can be divided into two parts: an entity network topology structure representation module and a time series data representation module.

[0064] In one embodiment, the risk influencing factors of the prediction object are taken as entities in the network topology structure, and the historical data of each risk influencing factor is taken as entity data corresponding to each entity.

[0065] In specific embodiments, in the entity network topology representation module, the entity network topology is modeled using a graph attention network (GAT), denoted as G=(V, E). Each entity is represented as a node V of the network topology, and the relationship between entities is represented as an edge E of the network topology. Each node V has its own feature vector x i i , and the feature information of all nodes constitutes a feature matrix X∈R N×P , where P represents the number of node attribute features. This modeling can be applied to risk prediction tasks to better capture anomalies in the entity network topology. GAT is a self-attention network model that can adaptively select features and attention weights, thereby improving the modeling capability of the entity network topology. At the same time, GAT can use a multi-head attention mechanism to extract higher-level network features to better analyze the entity network topology.

[0066] In specific embodiments, in the time series data representation module, the entity data is modeled using a gated recurrent unit (GRU). GRU can represent the data of each entity at different time points as a time series data sequence. GRU can adaptively select features and attention weights, thereby improving the modeling capability of the data. At the same time, GRU can use a gating mechanism to model long-term dependencies in the entity data.

[0067] In one embodiment, the network topology and the time series data sequence are combined to generate a network topology model.

[0068] Referring to Figure 4 , Figure 4 is a schematic flowchart of a risk prediction method based on reinforcement learning provided by an embodiment of the present application. The risk prediction method based on reinforcement learning can be applied in a server, for obtaining spatio-temporal features of data, processing the spatio-temporal features using a risk prediction model, and obtaining a risk prediction result, thereby effectively improving the accuracy and efficiency of risk prediction.

[0069] As shown in Figure 4 , the step S103 of the risk prediction method based on reinforcement learning specifically includes steps S301 to S303.

[0070] S301, based on the gated recurrent unit, encoding historical data in the network topology model to obtain dynamic features of the historical data;

[0071] S302, based on the self-attention network model, filtering the dynamic features according to attention weights to obtain static features of the historical data;

[0072] ​S303, tensor splicing the dynamic feature and the static feature to obtain the spatio-temporal feature.

[0073] In one embodiment, based on the constructed network topology model, the data of the entity is subjected to spatio-temporal feature extraction.

[0074] In specific embodiments, the data of the entity is encoded. The data of each entity at different time points is encoded using GRU to obtain the dynamic feature of the data. Because GRU is a recurrent neural network, it can adaptively select features and attention weights and can model long-term dependencies in the data. By encoding the data of each entity using GRU, the dynamic feature of each entity at different time points is obtained.

[0075] In one embodiment, the dynamic feature is subjected to feature selection. The dynamic feature is subjected to feature selection using a self-attention network to obtain the static feature of the data of the entity. The self-attention network is a method that can adaptively select features and attention weights, reducing the dependence on external information and being better at capturing the internal correlation of data or features, and can better capture anomalies in the data of the entity. By using the self-attention network, the dynamic feature can be compressed and screened to obtain the static feature of the data of the entity.

[0076] In one embodiment, the dynamic feature and the static feature are subjected to tensor splicing to obtain the extracted spatio-temporal feature.

[0077] In one embodiment, by using GRU and the self-attention network, features and attention weights can be adaptively selected, thereby improving the modeling capability of the data of the entity.

[0078] In specific embodiments, a target feature is selected from the dynamic feature as the static feature, and the dynamic feature and the static feature are subjected to tensor splicing to enhance the weight of the target feature. For example, the dynamic feature is 10, 9, and 8, and the target feature 9 is selected as the static feature. The dynamic feature and the static feature are then spliced to obtain 10, 9, 8, and 9, wherein the weight of the feature 9 is enhanced.

[0079] Please refer to Figure 5 , Figure 5 is a schematic flowchart of a risk prediction method based on reinforcement learning provided by an embodiment of the present application. The risk prediction method based on reinforcement learning can be applied in a server, for obtaining spatio-temporal features of data, processing the spatio-temporal features using a risk prediction model, and obtaining a risk prediction result, thereby effectively improving the accuracy and efficiency of risk prediction.

[0080] As shown in Figure 5 , before the step S104 of the risk prediction method based on reinforcement learning, the method specifically further includes steps S401 to S403.

[0081] S401, acquire training samples based on the risk influencing factors and the historical data;

[0082] S402, train a reinforcement learning decision model based on the historical risk records of the prediction object and the training samples, and record the number of training times;

[0083] S403, when the number of training times reaches a preset maximum number of training times, determine the reinforcement learning decision model as the risk prediction model.

[0084] Training the reinforcement learning decision model based on the historical risk records of the prediction object and the training samples includes: predicting the historical risk of the prediction object based on the training samples and the pre-trained model to obtain a historical prediction result; comparing the historical risk records with the historical prediction result to obtain a comparison result, and updating parameters of the reinforcement learning decision model based on a reward function and the comparison result to correct the reinforcement learning decision model.

[0085] In one embodiment, a multi-agent reinforcement learning decision model is trained based on historical data and historical risk records of a prediction object to obtain a risk prediction model that can be used. For example, a patient's personal health record can be used to obtain the patient's historical risk records of a certain disease, historical data of risk influencing factors of the disease in the health record is processed and input into a multi-agent reinforcement learning decision model, and an output result, i.e., a risk predicted by the model, is obtained. The predicted risk is compared with the historical risk records. When the predicted risk is the same as the historical risk, a positive reward is given based on a reward function, and if they are not the same, a negative reward is given to correct the model and train the model.

[0086] In one embodiment, as shown in Figure 6 each node (i.e., each entity) of the constructed network topology corresponds to an agent, and the agents can communicate and cooperate with each other.

[0087] In specific embodiments, a state factor of the model is defined: each agent also has a state vector that contains current state information and state information of surrounding agents. For example, in the risk prediction task in the Internet field, each host can be regarded as an agent. The state of each agent can be represented by network traffic data and running state information of the subnet where the agent is located, and can include the following information: network traffic data: including the size, type, source and target of network traffic; network running state: including the running state of the host, CPU utilization, memory utilization and the like.

[0088] In specific embodiments, the action selection of the model is defined: each agent can perform different actions specified, and the current state information needs to be considered when performing the action.

[0089] In specific embodiments, the reward function of the model is defined: the reward function is a very important part of the RP-MARL (multi-agent reinforcement learning decision model) algorithm, which directly affects the performance of the algorithm. In the RP-MARL algorithm, the reward function is divided into two parts: immediate reward and delayed reward. The immediate reward refers to the reward obtained by the agent performing an action in the current time step, which is related to the definition of entities, relationships, and features. The delayed reward refers to the reward obtained by the agent in the future time step. For example, the reward function can be defined as the detection and defense effect of network viruses, such as the detection rate and defense success rate of network viruses, which can be divided into the following two parts: detection effect: when the network virus is detected, give a positive reward; otherwise, give a negative reward; defense effect: when the network virus is successfully defended, give a positive reward; otherwise, give a negative reward.

[0090] In one embodiment, the policy selection of the model is defined: the RP-MARL algorithm uses the method of "optimal long-term value" (i.e. Figure 6 the value network in ) to select the action policy of the agent, that is, to predict the risk.

[0091] In specific embodiments, the RP-MARL is trained, first, the network weight is initialized, the new training is started, and the training data is the historical data of each agent of the network topology structure; second, the training data is input into the model to generate the strategy corresponding to each agent as argmax a∈A Q * (S t , a t , θ - ), t is the time, S is the state information of the agent, a is the strategy, that is, the risk that may occur to each agent, a has a certain probability to become other strategies, A is the set of all actions, θ is the probability, that is, the probability of the risk occurring, Q * is the optimal long-term value function, which is used to describe the maximum expected return of an action, that is, the strategy is always to select the action with the maximum optimal long-term value in the current state, and γ is the discount factor with a value of 0-1; each agent changes the system state through the generated strategy and produces an immediate reward r; the initial network weight is updated according to the generated new data According to the new network weight, a new strategy of each agent is generated, and the training frequency is recorded. After reaching the preset frequency, the training is stopped, and the multi-agent reinforcement learning decision model at this time is used as the risk prediction model.

[0092] Referring to Figure 7 , Figure 7 An embodiment of the present application provides a schematic block diagram of a risk prediction device based on reinforcement learning, which is used for executing the aforementioned risk prediction method based on reinforcement learning. Wherein, the risk prediction device based on reinforcement learning can be configured in a server.

[0093] As Figure 7 shown, the risk prediction device based on reinforcement learning 500 comprises:

[0094] A historical data acquisition module 501 is configured to acquire at least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor;

[0095] A network topology model obtaining module 502 is configured to process the risk influencing factor and the historical data based on a self-attention network model and a gated recurrent unit, and obtain a network topology model of the prediction object;

[0096] A space-time feature generation module 503 is configured to acquire dynamic features and static features of the historical data based on the network topology model, and generate space-time features of the historical data based on the dynamic features and the static features;

[0097] A risk prediction result obtaining module 504 is configured to predict the risk of the prediction object based on a preset risk prediction model and the space-time features, and obtain a risk prediction result of the prediction object.

[0098] In an embodiment, the network topology model obtaining module 502 comprises:

[0099] A network topology structure construction unit is configured to acquire a correlation between the risk influencing factors based on the graph attention network model, and construct a network topology structure of the risk influencing factors based on the correlation;

[0100] A time series data sequence obtaining unit is configured to process the historical data of each risk influencing factor based on the gated recurrent unit, and obtain a time series data sequence corresponding to the historical data;

[0101] A network topology model generation unit is configured to generate the network topology model based on the network topology structure and the time series data sequence.

[0102] In an embodiment, the space-time feature generation module 503 comprises:

[0103] The dynamic feature obtaining unit is configured to encode historical data in the network topology model based on the gating cycle unit, and obtain dynamic features of the historical data.

[0104] The static feature obtaining unit is configured to filter the dynamic features according to attention weights based on the self-attention network model, and obtain static features of the historical data.

[0105] The spatio-temporal feature obtaining unit is configured to concatenate the dynamic features and the static features to obtain spatio-temporal features.

[0106] In an embodiment, the risk prediction device 500 based on reinforcement learning further includes a risk prediction model obtaining module, and the risk prediction model obtaining module includes:

[0107] The training sample obtaining unit is configured to obtain training samples based on the risk influence factors and the historical data.

[0108] The model training unit is configured to train a reinforcement learning decision model based on historical risk records of the prediction object and the training samples, and record a training number.

[0109] The risk prediction model determining unit is configured to determine the reinforcement learning decision model as the risk prediction model when the training number reaches a preset maximum training number.

[0110] In an embodiment, the model training unit includes:

[0111] The historical prediction result obtaining sub-unit is configured to predict historical risks of the prediction object based on the training samples and the pre-training model to obtain historical prediction results.

[0112] The model parameter updating sub-unit is configured to compare the historical risk records and the historical prediction results to obtain a comparison result, and update parameters of the reinforcement learning decision model based on a reward function and the comparison result to correct the reinforcement learning decision model.

[0113] In an embodiment, the risk prediction result obtaining module 504 includes:

[0114] The prediction risk obtaining unit is configured to process the spatio-temporal features based on the risk prediction model to obtain prediction risks of the risk influence factors.

[0115] The risk prediction result obtaining unit is configured to generate a risk prediction result of the prediction object based on the prediction risks of the risk influence factors.

[0116] In an embodiment, the risk prediction apparatus 500 based on reinforcement learning further comprises:

[0117] a risk management strategy generation unit configured to generate a risk management strategy based on the risk type and the occurrence probability for reference by a user.

[0118] It should be noted that, for the convenience and brevity of description, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0119] The apparatus described above can be implemented in the form of a computer program, which can run on a computer device such as the computer device shown in Figure 8 .

[0120] Please refer to Figure 8 , Figure 8 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device can be a server.

[0121] Please refer to Figure 8 , the computer device comprises a processor, a memory and a network interface connected through a system bus, wherein the memory can comprise a non-volatile storage medium and an internal memory.

[0122] The non-volatile storage medium can store an operating system and a computer program. The computer program comprises program instructions which, when executed, can cause the processor to execute any kind of risk prediction method based on reinforcement learning.

[0123] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0124] The internal memory provides an environment for the execution of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to execute any kind of risk prediction method based on reinforcement learning.

[0125] The network interface is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure shown in Figure 8 , is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] It should be appreciated that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0127] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0128] obtain at least one risk influencing factor of a prediction object and historical data corresponding to the risk influencing factor;

[0129] process the risk influencing factor and the historical data based on a self-attention network model and a gated recurrent unit, to obtain a network topology model of the prediction object;

[0130] obtain dynamic features and static features of the historical data based on the network topology model, and generate spatio-temporal features of the historical data based on the dynamic features and the static features;

[0131] predict a risk of the prediction object based on a preset risk prediction model and the spatio-temporal features, to obtain a risk prediction result of the prediction object.

[0132] In one embodiment, when implementing the processing of the risk influencing factor and the historical data based on the self-attention network model and the gated recurrent unit to obtain the network topology model of the prediction object, the processor is configured to implement:

[0133] obtain a correlation between the risk influencing factors based on the graph attention network model, and construct a network topology structure of the risk influencing factors based on the correlation;

[0134] process the historical data of each risk influencing factor based on the gated recurrent unit, to obtain a time series data sequence corresponding to the historical data;

[0135] generate the network topology model based on the network topology structure and the time series data sequence.

[0136] In an embodiment, the processor, when implementing obtaining dynamic features and static features of the historical data based on the network topology model, and generating spatio-temporal features of the historical data based on the dynamic features and the static features, is configured to:

[0137] encoding the historical data in the network topology model based on the gating recurrent unit to obtain the dynamic features of the historical data;

[0138] filtering the dynamic features according to attention weights based on the self-attention network model to obtain the static features of the historical data;

[0139] tensor splicing the dynamic features and the static features to obtain the spatio-temporal features.

[0140] In an embodiment, before the processor implements predicting the risk of the prediction object based on the preset risk prediction model and the spatio-temporal features to obtain the risk prediction result of the prediction object, the processor is further configured to:

[0141] obtaining training samples based on the risk influencing factors and the historical data;

[0142] training a reinforcement learning decision model based on historical risk records of the prediction object and the training samples, and recording a training number;

[0143] when the training number reaches a preset maximum training number, determining the reinforcement learning decision model as the risk prediction model.

[0144] In an embodiment, when the processor implements training a reinforcement learning decision model based on historical risk records of the prediction object and the training samples, the processor is configured to:

[0145] predicting historical risks of the prediction object based on the training samples and the pre-training model to obtain historical prediction results;

[0146] comparing the historical risk records with the historical prediction results to obtain a comparison result, and updating parameters of the reinforcement learning decision model based on a reward function and the comparison result to correct the reinforcement learning decision model.

[0147] In an embodiment, when the processor implements predicting the risk of the prediction object based on the preset risk prediction model and the spatio-temporal features to obtain the risk prediction result of the prediction object, the processor is configured to

[0148] Based on the risk prediction model, the spatio-temporal features are processed to obtain a predicted risk of each risk influencing factor;

[0149] Based on the predicted risk of each risk influencing factor, a risk prediction result of the prediction object is generated.

[0150] In one embodiment, the processor, after implementing the preset risk prediction model and the spatio-temporal features to predict the risk of the prediction object and obtain the risk prediction result of the prediction object, is further configured to implement:

[0151] Based on the risk type and the occurrence probability, a risk management strategy is generated for reference by a user.

[0152] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any one of the risk prediction methods based on reinforcement learning provided by the embodiments of the present application.

[0153] The computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0154] The above merely illustrates the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A risk prediction method based on reinforcement learning, characterized in that, include: Obtain at least one risk factor affecting the prediction object and the historical data corresponding to the risk factor; Based on a self-attention network model and a gated recurrent unit, the risk influencing factors and the historical data are processed to obtain the network topology model of the prediction object. The self-attention network model includes a graph attention network model. The process of processing the risk influencing factors and historical data based on the self-attention network model and gated recurrent units to obtain the network topology model of the prediction object includes: Based on the graph attention network model, the correlation between the risk influencing factors is obtained, and based on the correlation, a network topology structure of the risk influencing factors is constructed, wherein each node of the network topology structure is an agent; Based on the gated loop unit, the historical data of each of the risk influencing factors are processed to obtain the time-series data sequence corresponding to the historical data; The network topology model is generated based on the network topology and the time-series data sequence. Based on the network topology model, the dynamic and static features of the historical data are obtained, and the spatiotemporal features of the historical data are generated based on the dynamic and static features. Based on the preset risk prediction model and the spatiotemporal characteristics, the risk of the prediction object is predicted, the predicted risk of each intelligent agent is obtained, and the predicted risks of each intelligent agent are aggregated to obtain the risk prediction result of the prediction object.

2. The risk prediction method based on reinforcement learning according to claim 1, characterized in that, The step of obtaining dynamic and static features of the historical data based on the network topology model, and generating spatiotemporal features of the historical data based on the dynamic and static features, includes: Based on the gated loop unit, the historical data in the network topology model is encoded to obtain the dynamic characteristics of the historical data; Based on the self-attention network model, the dynamic features are filtered according to attention weights to obtain the static features of the historical data; The spatiotemporal features are obtained by tensor concatenation of the dynamic features and the static features.

3. The risk prediction method based on reinforcement learning according to claim 1, characterized in that, Before obtaining the risk prediction result of the predicted object based on the preset risk prediction model and the spatiotemporal characteristics, the method further includes: Based on the aforementioned risk factors and the historical data, training samples were obtained; Based on the historical risk records of the predicted object and the training samples, the reinforcement learning decision model is trained, and the number of training times is recorded. When the number of training iterations reaches the preset maximum number of training iterations, the reinforcement learning decision model is determined as the risk prediction model.

4. The risk prediction method based on reinforcement learning according to claim 3, characterized in that, The training of the reinforcement learning decision model based on the historical risk records of the predicted object and the training samples includes: Based on the training samples and the pre-trained model, the historical risk of the prediction object is predicted to obtain historical prediction results. The historical risk records are compared with the historical prediction results to obtain the comparison results. Based on the reward function and the comparison results, the parameters of the reinforcement learning decision model are updated to correct the reinforcement learning decision model.

5. The risk prediction method based on reinforcement learning according to claim 1, characterized in that, The method of predicting the risk of the predicted object based on a preset risk prediction model and the spatiotemporal characteristics, and obtaining the risk prediction result of the predicted object, includes: Based on the risk prediction model, the spatiotemporal characteristics are processed to obtain the predicted risk of each of the risk influencing factors; Based on the predicted risks of each of the aforementioned risk influencing factors, a risk prediction result for the predicted object is generated.

6. The risk prediction method based on reinforcement learning according to any one of claims 1-5, characterized in that, The risk prediction result includes the risk type and the probability of occurrence corresponding to the risk type. After predicting the risk of the prediction object based on the preset risk prediction model and the spatiotemporal characteristics, and obtaining the risk prediction result of the prediction object, the method further includes: Based on the risk type and the probability of occurrence, a risk management strategy is generated for user reference.

7. A risk prediction device based on reinforcement learning, characterized in that, include: The historical data acquisition module is used to acquire at least one risk influencing factor of the prediction object and the historical data corresponding to the risk influencing factor. The network topology model acquisition module is used to process the risk influencing factors and the historical data based on a self-attention network model and a gated recurrent unit to obtain the network topology model of the prediction object, wherein the self-attention network model includes a graph attention network model. The network topology model acquisition module includes: A network topology construction unit is used to obtain the correlation between the risk influencing factors based on the graph attention network model, and to construct the network topology of the risk influencing factors based on the correlation, wherein each node of the network topology acts as an agent. The time-series data sequence acquisition unit is used to process the historical data of each of the risk influencing factors based on the gated loop unit to obtain the time-series data sequence corresponding to the historical data; A network topology model generation unit is used to generate the network topology model based on the network topology and the time-series data sequence. The spatiotemporal feature generation module is used to obtain the dynamic and static features of the historical data based on the network topology model, and generate the spatiotemporal features of the historical data based on the dynamic and static features. The risk prediction result acquisition module is used to predict the risk of the prediction object based on a preset risk prediction model and the spatiotemporal characteristics, obtain the predicted risk of each of the intelligent agents, and aggregate the predicted risks of each of the intelligent agents to obtain the risk prediction result of the prediction object.

8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the risk prediction method based on reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the risk prediction method based on reinforcement learning as described in any one of claims 1 to 6.

Images