Method and device for reward prediction model training and control action recommendation model optimization

Abstract

The embodiment of the invention discloses a reward prediction model training and control action recommendation model optimization method and device, and the method comprises the steps: building reward training data based on pre-collected transaction feature data and a pre-trained initial control action recommendation model; performing supervised learning on a pre-constructed reward prediction model according to the reward training data to obtain an initial reward prediction model; preset auxiliary learning labels are obtained, the auxiliary learning labels correspond to different learning tasks, multi-task learning is conducted on the initial reward prediction model according to the auxiliary learning labels, and a target reward prediction model is obtained. Through the target reward prediction model obtained through multi-task learning, the prediction of the reward signal in the reinforcement learning process can be realized, the labor cost is effectively reduced, the prediction efficiency of the reward signal is improved, meanwhile, the reasonability and effectiveness of the predicted reward signal are ensured, and the interpretability of the reward signal is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for training a reward prediction model and optimizing a control action recommendation model. Background Art

[0002] Internet finance is the product of the combination of the Internet and the financial industry, including third-party payment, big data finance and other forms. Internet finance can achieve symmetry of new market information, and also has convenient payment functions, so that the supply and demand sides of funds can trade directly. Risk control is an important part of payment security management, which can determine what control actions to output for risky transaction events, such as directly letting them go, performing SMS, face verification and other verifications on transaction events. In the transaction process, various recommendation models are usually used to recommend control actions that are more in line with user expectations and have risk control capabilities to users, so as to ensure user payment security.

[0003] At present, when the control action recommendation model is trained through reinforcement learning, the reward signal output efficiency of the control action recommendation model is low, the interpretability is poor, and the rationality and effectiveness are low. Summary of the invention

[0004] In a first aspect of the embodiments of the specification, a reward prediction model training method is provided, which can effectively reduce labor costs, improve the prediction efficiency of reward signals, and at the same time ensure the rationality and effectiveness of the predicted reward signals, and improve the interpretability of the reward signals. The method includes:

[0005] Construct reward training data based on pre-collected transaction feature data and pre-trained initial control action recommendation model;

[0006] Performing supervised learning on a pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model;

[0007] Preset auxiliary learning labels are obtained, where the auxiliary learning labels correspond to different learning tasks, and multi-task learning is performed on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

[0008] Further, in some embodiments of the present specification, based on the above solution, the reward training data is constructed based on the pre-collected transaction feature data and the pre-trained initial control action recommendation model, including:

[0009] Inputting the transaction feature data into the initial control action recommendation model to determine the control action data corresponding to the transaction feature data;

[0010] Reward training data is constructed based on the binary feature data consisting of the transaction feature data and the management and control action data.

[0011] Further, in some embodiments of the present specification, based on the aforementioned solution, the pre-constructed reward prediction model is supervised for learning according to the reward training data to obtain an initial reward prediction model, including:

[0012] matching a sample reward signal in a business-specified scoring database based on the reward training data;

[0013] The reward training data is input into the pre-built reward prediction model, and the sample reward signal is used as a supervised training label to perform supervised learning on the reward prediction model to obtain an initial reward prediction model.

[0014] Further, in some embodiments of the present specification, based on the aforementioned solution, the initial reward prediction model includes at least one expert network, at least one gating network and at least one combiner, and the number of the combiners is the same as the number of the learning tasks;

[0015] The performing multi-task learning on the initial reward prediction model according to the auxiliary learning label to obtain a target reward prediction model includes:

[0016] Using each of the expert networks to extract features from the reward training data to obtain a feature embedding representation vector;

[0017] Inputting the reward training data into each of the gating networks for model training, so that the gating network determines the probability distribution weights of each of the expert networks in different learning tasks;

[0018] The feature embedding representation vectors and the probability distribution weights are combined by the combiners corresponding to different tasks to obtain a reward signal prediction value corresponding to the reward training data;

[0019] The loss of each learning task is determined according to the reward signal prediction value and the auxiliary learning label, and the model parameters of the initial reward prediction model are updated based on the loss until the loss of the initial reward prediction model converges or reaches a predetermined number of iterative training times, thereby obtaining a target reward prediction model.

[0020] Further, in some embodiments of the present specification, based on the aforementioned solution, the expert network includes a dynamic embedding layer, a first deep neural network layer, and a second deep neural network layer;

[0021] The step of extracting features from the reward training data using each of the expert networks to obtain a feature embedding representation vector includes:

[0022] Performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation;

[0023] Performing a nonlinear transformation on the underlying feature embedding representation through each of the first deep neural network layers to obtain a shared feature embedding representation across learning tasks in the underlying features;

[0024] The independent features of each learning task are extracted from the shared feature embedding representation through each of the second deep neural network layers, and the independent features are integrated to obtain a corresponding feature embedding representation vector.

[0025] Further, in some embodiments of the present specification, based on the aforementioned solution, the dynamic embedding layer includes a multi-embedding layer and a third deep neural network layer;

[0026] The step of performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation includes:

[0027] Performing binning processing on the reward training data to obtain binning features;

[0028] Performing quantile division processing on the reward training data to obtain quantile features;

[0029] The binning feature is converted into a first intermediate feature through the multi-embedding layer, and the quantile feature is nonlinearly transformed through the third deep neural network layer to obtain a second intermediate feature;

[0030] The first intermediate feature and the second intermediate feature are fused to obtain the underlying feature embedding representation.

[0031] Furthermore, in some embodiments of the present specification, based on the aforementioned scheme, the auxiliary learning labels include fraud risk identification task labels, financial risk identification task labels, help identification task labels, and restriction release task labels.

[0032] In a second aspect of the embodiments of the specification, a method for optimizing a management and control action recommendation model is also proposed, the method comprising:

[0033] Acquire transaction feature data, and input the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action;

[0034] Inputting the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by the reward prediction model training method in the first aspect;

[0035] The initial control action recommendation model is subjected to proximal strategy optimization according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

[0036] In a third aspect of the embodiments of the specification, a reward prediction model training device is also proposed, including:

[0037] A reward training data acquisition module is used to construct reward training data based on pre-collected transaction feature data and a pre-trained initial control action recommendation model;

[0038] A supervised learning module, used to perform supervised learning on a pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model;

[0039] The multi-task learning module is used to obtain preset auxiliary learning labels, where the auxiliary learning labels correspond to different learning tasks, and perform multi-task learning on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

[0040] In a fourth aspect of the embodiments of the specification, a control action recommendation model optimization device is also proposed, including:

[0041] A control action generation module is used to obtain transaction feature data and input the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action;

[0042] A reward signal prediction module, used for inputting the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by the reward prediction model training method in one aspect;

[0043] A proximal strategy optimization module is used to perform proximal strategy optimization on the initial control action recommendation model according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

[0044] According to a fifth aspect of the embodiments of this specification, a computer program product is provided. The computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the method steps in the first aspect or the second aspect.

[0045] In a sixth aspect of the embodiments of this specification, a storage medium is further provided, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the method in the first aspect or the second aspect.

[0046] According to a seventh aspect of the embodiments of this specification, there is also provided an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method in the first aspect or the second aspect.

[0047] In an embodiment of the present specification, reward training data is constructed through pre-collected transaction feature data and a pre-trained initial management and control action recommendation model, and supervised learning is performed on the pre-constructed reward prediction model based on the reward training data to obtain an initial reward prediction model; auxiliary learning labels corresponding to different learning tasks are obtained, and then multi-task learning is performed on the initial reward prediction model based on the auxiliary learning labels to obtain a target reward prediction model. On the one hand, the target reward prediction model obtained through multi-task learning can realize the prediction of reward signals in the reinforcement learning process of the control action recommendation model. Compared with outputting the reward signal of the control action recommendation model through expert experience, it can effectively reduce labor costs and improve the prediction output efficiency of reward signals. On the other hand, through the auxiliary learning labels of different learning tasks, multi-task learning of the target reward prediction model is realized, the rationality and effectiveness of the predicted output reward signal are improved, and the interpretability of the reward signal is improved. And by setting different auxiliary learning labels, the generalization of the target reward prediction model can be improved, and the scope of application of the target reward prediction model can be guaranteed. On the other hand, supervised learning followed by multi-task learning can effectively improve the convergence speed of the target reward prediction model, reduce the training and construction cost of the target reward prediction model, and effectively ensure the accuracy of the predicted reward signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the system architecture of an exemplary application environment in which a reward prediction model training method, a control action recommendation model optimization method, and an apparatus according to an embodiment of this specification can be applied is shown.

[0049] Figure 2 A flowchart of a reward prediction model training method provided in an embodiment of this specification.

[0050] Figure 3 A flowchart of a target reward prediction model obtained through multi-task learning training is provided in an embodiment of this specification.

[0051] Figure 4 A schematic diagram of the network structure of a target reward prediction model provided in an embodiment of this specification.

[0052] Figure 5 A schematic diagram of a process for extracting a feature embedding representation vector of reward training data provided in an embodiment of this specification.

[0053] Figure 6A schematic diagram of a process for extracting underlying feature embedding representations of reward training data provided in an embodiment of this specification.

[0054] Figure 7 A schematic diagram of the network structure of an expert network provided in an embodiment of this specification.

[0055] Figure 8 A flowchart of a control action recommendation model optimization method provided in an embodiment of this specification.

[0056] Fig. 9 A flowchart of a target control action recommendation model obtained through proximal strategy optimization is provided in an embodiment of this specification.

[0057] Fig.10 Another flowchart of obtaining a target control action recommendation model through proximal strategy optimization is provided in an embodiment of this specification.

[0058] Fig.11 A schematic diagram of the structure of a reward prediction model training device provided in an embodiment of this specification.

[0059] Fig.12 A schematic diagram of the structure of a control action recommendation model optimization device provided in an embodiment of this specification.

[0060] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0062] Figure 1 A schematic diagram of the system architecture of an exemplary application environment in which a reward prediction model training method, a control action recommendation model optimization method, and an apparatus according to an embodiment of this specification can be applied is shown.

[0063] like Figure 1As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with artificial intelligence (AI) computing capabilities, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0064] The reward prediction model training method or control action recommendation model optimization method provided in the embodiments of this specification is generally executed by the terminal devices 101, 102, and 103, and accordingly, the reward prediction model training device or the control action recommendation model optimization device is generally set in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the reward prediction model training method or the control action recommendation model optimization method provided in the embodiments of this specification can also be executed by the server 105, and accordingly, the reward prediction model training device or the control action recommendation model optimization device can also be set in the server 105, which is not particularly limited in this exemplary embodiment.

[0065] See also Figure 2 , is a flowchart of a reward prediction model training method provided in an embodiment of this specification. In the embodiment of this specification, the reward prediction model training method can be applied to a terminal device or a server, and this exemplary embodiment does not make any special restrictions on this. Figure 2 The process shown is described in detail, and the reward prediction model training method in the embodiment of this specification may specifically include the following steps:

[0066] Step S210, constructing reward training data based on the pre-collected transaction feature data and the pre-trained initial management and control action recommendation model;

[0067] Step S220, performing supervised learning on the pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model;

[0068] Step S230, obtaining preset auxiliary learning labels, wherein the auxiliary learning labels correspond to different learning tasks, and performing multi-task learning on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

[0069] According to the reward prediction model training method in the embodiments of this specification, on the one hand, the target reward prediction model obtained through multi-task learning can realize the prediction of reward signals in the reinforcement learning process of the control action recommendation model. Compared with outputting the reward signal of the control action recommendation model through expert experience, it can effectively reduce labor costs and improve the prediction output efficiency of reward signals; on the other hand, through the auxiliary learning labels of different learning tasks, multi-task learning of the target reward prediction model is realized, the rationality and effectiveness of the predicted output reward signal are improved, and the interpretability of the reward signal is improved; and by setting different auxiliary learning labels, the generalization of the target reward prediction model can be improved, and the scope of application of the target reward prediction model can be guaranteed; on the other hand, supervised learning followed by multi-task learning can effectively improve the convergence speed of the target reward prediction model, reduce the training and construction cost of the target reward prediction model, and effectively ensure the accuracy of the predicted reward signal.

[0070] The following is a detailed description of the reward prediction model training method provided in the embodiments of this specification.

[0071] In step S210, reward training data is constructed based on the pre-collected transaction feature data and the pre-trained initial management and control action recommendation model.

[0072] In an example embodiment of the present specification, transaction feature data refers to feature data extracted from the user's transaction data. For example, the transaction feature data may be feature data such as the transaction user and transaction amount, or may be feature data such as the transaction event and transaction location. This example embodiment does not specifically limit the type of transaction feature data.

[0073] The initial control action recommendation model refers to an artificial intelligence model that outputs predicted control actions based on input transaction feature data. For example, the initial control action recommendation model can be a model based on a Transformer network or a model based on a Convolutional Neural Network (CNN). The model type adopted by the initial control action recommendation model can be selected based on actual conditions, and there is no special limitation on this.

[0074] Reward training data refers to sample data used to train a reward prediction model. For example, the reward training data may include transaction feature data and management and control action data. Specifically, the transaction feature data may be input into a pre-trained initial management and control action recommendation model to obtain output management and control action data, and the reward training data may be constructed based on the transaction feature data and the management and control action data.

[0075] Transaction data can be collected in advance, its features can be extracted through an embedded representation network to obtain transaction feature data, and the transaction feature data can be stored in a database. When executing the reward prediction model training method provided in the embodiment of this specification, the transaction feature data can be obtained from the database; of course, transaction data can also be obtained in real time and converted into transaction feature data. This example embodiment does not specifically limit the method or source of obtaining the transaction feature data.

[0076] In step S220, supervised learning is performed on the pre-constructed reward prediction model according to the reward training data to obtain an initial reward prediction model.

[0077] In an example embodiment of the present specification, a reward prediction model refers to an artificial intelligence model that gives a predicted reward signal based on reward training data. For example, the reward prediction model can be a neural network model based on a multi-head self-attention mechanism (Multi-HeadAttention), or a deep learning model based on a multi-task learning network structure. For example, a deep learning model based on a multi-task learning (MTL) network structure can be a mixed expert model (Multi-gate Mixture-of-Experts, MMoE). This example embodiment does not specifically limit the network type of the reward prediction model.

[0078] The pre-built reward prediction model refers to a reward prediction model having model parameters in an initial state, and the initial reward prediction model refers to a reward prediction model obtained by adjusting and updating the model parameters of the pre-built reward prediction model through supervised learning.

[0079] The reward signal corresponding to the reward training data can be first determined through a mechanism or rule for management and control business assessment. For example, the reward signal label corresponding to the reward training data can be determined through expert experience, or the reward signal label corresponding to the reward training data can be matched in the form of a regular database or a vector database. This example embodiment does not specifically limit the method of determining the reward signal label for supervised learning.

[0080] It is easy for those skilled in the art to understand that the reward training data and the obtained reward signal can be used to construct a sample data set, and the sample data set can be divided into a training set, a validation set and a test set. The parameters of the pre-constructed reward prediction model are adjusted on the training set, and the loss function is minimized by the back propagation algorithm. The hyperparameters of the pre-constructed reward prediction model are adjusted using the validation set to prevent overfitting. The loss function can be a mean square error, a cross entropy loss function, etc. The model performance of the reward prediction model is evaluated on the test set, and based on the performance of the reward prediction model on the test set, the model structure or model parameters of the reward prediction model are continuously adjusted to achieve supervised learning of the pre-constructed reward prediction model, so as to further improve the model performance, thereby constructing an initial reward prediction model.

[0081] In step S230, a preset auxiliary learning label is obtained, where the auxiliary learning label corresponds to different learning tasks, and multi-task learning is performed on the initial reward prediction model according to the auxiliary learning label to obtain a target reward prediction model.

[0082] In an example embodiment of the present specification, an auxiliary learning label refers to a label that contains different learning tasks to guide the initial reward prediction model to achieve multi-task learning. For example, the auxiliary learning label can be a fraud risk identification task label, a financial risk identification task label, a help-seeking identification task label, a limit-release task label, an extreme negative emotion identification task label, etc. Of course, the auxiliary learning label can also be other types of risk and experience labels used to assist the initial reward prediction model in multi-task learning, and this embodiment is not limited to this.

[0083] The target reward prediction model obtained through multi-task learning can realize the prediction of reward signals in the reinforcement learning process of the control action recommendation model. Compared with outputting the reward signal of the control action recommendation model through expert experience, it can effectively reduce labor costs and improve the prediction output efficiency of reward signals.

[0084] Through the auxiliary learning labels of different learning tasks, multi-task learning of the target reward prediction model is realized, the rationality and effectiveness of the predicted output reward signal are improved, and the interpretability of the reward signal is improved; and by setting different auxiliary learning labels, the generalization of the target reward prediction model can be improved to ensure the scope of application of the target reward prediction model.

[0085] By first performing supervised learning and then multi-task learning on the pre-built reward prediction model, the convergence speed of the trained target reward prediction model can be effectively improved, the training and construction cost of the target reward prediction model can be reduced, and the accuracy of the predicted reward signal can be effectively guaranteed.

[0086] Next, step S210 and step S230 are described in detail.

[0087] In an example embodiment of the present specification, a reinforcement learning framework can be constructed to implement the training of the target reward prediction model. For example, the initial control action recommendation model can be used as an agent in the reinforcement learning framework, and reasonable control actions can be recommended based on input transaction event, user and other transaction feature data; the control actions output by the initial control action recommendation model can be used as actions in the reinforcement learning framework; a complex combination of control systems (business needs), transaction users, etc. can be used as the environment of the reinforcement learning framework, and the environment can output a feedback signal after receiving the control action; the target reward prediction model can be used to generate an incentive signal in the reinforcement learning framework, which is used to quantify the quality of the control actions output by the initial control action recommendation model, thereby guiding the initial control action recommendation model to perform reinforcement learning.

[0088] Optionally, the following steps may be used to construct reward training data based on pre-collected transaction feature data and a pre-trained initial control action recommendation model, including:

[0089] The transaction feature data can be input into the initial management and control action recommendation model to determine the management and control action data corresponding to the transaction feature data; and the reward training data can be constructed based on the binary feature data consisting of the transaction feature data and the management and control action data.

[0090] Among them, the management and control action data refers to the service actions provided to users in response to their operations in transaction events, which are used to optimize the risks, compliance and efficiency issues that may be involved in transaction events. For example, the management and control action data can be directly provided through actions, or can be SMS reminder actions for transaction events, or can be SMS verification or face verification actions for transaction events. This example embodiment does not specifically limit the management and control action data.

[0091] Reward training data can be constructed based on binary feature data consisting of transaction feature data and control action data output by the initial control action recommendation model, so as to implement model training of the pre-constructed reward prediction model according to the reward training data.

[0092] Optionally, supervised learning of a pre-built reward prediction model can be performed based on reward training data to obtain an initial reward prediction model through the following steps, which may specifically include:

[0093] The sample reward signal can be matched in the business-specified scoring database based on the reward training data; then the reward training data can be input into the pre-built reward prediction model, and the sample reward signal can be used as a supervised training label to perform supervised learning on the reward prediction model to obtain an initial reward prediction model.

[0094] Among them, the business-specified scoring database refers to a pre-constructed database for determining the reward signal or reward points corresponding to the reward training data. For example, a regular expression can be constructed based on the reward training data, and the reward signal or reward points corresponding to the reward training data can be matched in the business-specified scoring database based on the regular expression; of course, the reward training data can also be embedded as a feature embedding vector, and the reward signal or reward points corresponding to the reward training data can be matched in the business-specified scoring database based on the feature embedding vector. This example embodiment does not specifically limit the method of matching sample reward signals in the business-specified scoring database.

[0095] After obtaining the sample reward signal corresponding to the reward training data, the reward training data can be used as the input of the pre-constructed reward prediction model, and the matched sample reward signal can be used as a supervised training label to achieve supervised learning of the pre-constructed reward prediction model and obtain an initial reward prediction model.

[0096] In an optional embodiment of the present specification, the initial reward prediction model may include at least one expert network, at least one gating network and at least one combiner, and the number of combiners is the same as the number of learning tasks.

[0097] Optionally, you can Figure 3 The steps in the implementation of multi-task learning of the initial reward prediction model based on auxiliary learning labels, refer to Figure 3 As shown, it may specifically include:

[0098] Step S310, using each of the expert networks to extract features from the reward training data to obtain a feature embedding representation vector;

[0099] Step S320, inputting the reward training data into each of the gating networks for model training, so that the gating network determines the probability distribution weights of each of the expert networks in different learning tasks;

[0100] Step S330, combining each of the feature embedding representation vectors and the probability distribution weights through the combiners corresponding to different tasks to obtain a reward signal prediction value corresponding to the reward training data;

[0101] Step S340, determining the loss of each learning task according to the reward signal prediction value and the auxiliary learning label, and updating the model parameters of the initial reward prediction model based on the loss, until the loss of the initial reward prediction model converges or reaches a predetermined number of iterative training times, to obtain a target reward prediction model.

[0102] Among them, the expert network refers to a neural network used to extract and share the underlying features of feature data. Multiple expert networks are parallel networks, and each network learns for the entire data set. However, as the multi-task learning training proceeds, each expert network will usually be specialized and become a dedicated network for processing specific types of data or specific learning tasks.

[0103] The task of the gated network is to dynamically assign the weight of the expert network to each data point. The probability distribution of the output of the expert network can be determined based on the input data, and the initial reward prediction model can have multiple gated networks, each of which corresponds to a specific learning task. In this embodiment, the data point refers to a set of sample data consisting of sample reward signals, reward training data, and auxiliary learning labels. For example, assuming that there are three expert networks E1, E2, and E3 and a gated network G, for task T1, when a data point is input into the model, the gated network G calculates a set of weights w1, w2, and w3 based on the characteristics of the data point based on the attention mechanism, corresponding to E1, E2, and E3 respectively. This set of weights determines the contribution of the feature vectors output by each expert network at this data point to the final prediction.

[0104] The task of the combiner is to combine the output data of the expert network with the probability distribution weights of the gated network to generate the output network. Each learning task will have a combiner to generate the output related to the learning task. For example, the combiner can be a multi-layer perceptron, a recurrent layer, an attention layer, and other network structures, which are not limited in this specification.

[0105] Figure 4 A schematic diagram of the network structure of a target reward prediction model provided in an embodiment of this specification.

[0106] refer to Figure 4 As shown, the initial reward prediction model may include an expert network A 402, an expert network B 403 and an expert network C 404, a gating network A 405 and a gating network B 406, a combiner A 407 and a combiner B 408, where the combiner A 407 is used to output data related to the learning task A, and the combiner B 408 is used to output data related to the learning task B.

[0107] Expert network A 402, expert network B 403 and expert network C 404 can be used to extract features from reward training data 401 to obtain feature embedding representation vectors; then the reward training data 401 can be input into gated network A 405 and gated network B 406 for model training, so that gated network A 405 and gated network B 406 determine the probability distribution weights of expert network A 402, expert network B 403 and expert network C 404 in learning task A and learning task B respectively; then combiner A 407 and combiner B 408 corresponding to learning task A and learning task B respectively combine each feature embedding representation vector and probability distribution weight to obtain a reward signal prediction value corresponding to reward training data 401; then the loss of each learning task can be determined according to the reward signal prediction value and the auxiliary learning label, and the model parameters of the initial reward prediction model are updated based on the loss until the loss of the initial reward prediction model converges or reaches a predetermined number of iterative training times to obtain a target reward prediction model.

[0108] Optionally, the expert network includes a dynamic embedding layer, a first deep neural network layer, and a second deep neural network layer; Figure 5 The content in this paper realizes the use of each expert network to extract features from reward training data and obtain feature embedding representation vectors. Figure 5 As shown, it may specifically include:

[0109] Step S510, performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation;

[0110] Step S520, performing a nonlinear transformation on the underlying feature embedding representation through each of the first deep neural network layers to obtain a shared feature embedding representation across learning tasks in the underlying features;

[0111] Step S530: extracting independent features of each learning task from the shared feature embedding representation through each of the second deep neural network layers, and integrating the independent features to obtain a corresponding feature embedding representation vector.

[0112] The dynamic embedding layer is used to perform preliminary feature conversion on the input reward training data, and the parameters of the dynamic embedding layer can be dynamically adjusted according to the different input data, so as to generate more accurate feature representations for different input data. For example, the underlying features can be the embedding vectors obtained by mapping the discrete or continuous features in the reward training data to a high-dimensional continuous space, and the complex relationships between features can be captured through the embedding vectors.

[0113] After obtaining the underlying features, it is necessary to further explore the common information in the underlying features. The first deep neural network layer is used to extract shared features across learning tasks from the underlying features. Shared features refer to features that are important for all or most learning tasks, which can help the model better understand the common patterns and rules between different learning tasks.

[0114] The second deep neural network layer focuses on extracting the independent features required for each specific learning task from the shared features, so that the model can optimize the performance of each learning task in a targeted manner without being interfered with by other learning tasks. Among them, independent features refer to features that are important for a specific learning task but unimportant relative to other learning tasks.

[0115] By extracting shared features from the expert network, the model can learn more general representations from multiple related learning tasks, thereby improving the generalization ability on new learning tasks. Moreover, the existence of shared features allows the model to learn together on multiple learning tasks, avoiding the problem of overfitting on a single learning task. The extraction of independent features helps the model to effectively transfer knowledge between similar learning tasks, and can quickly adapt to new learning tasks even when the amount of data is limited. By cleverly combining the extraction of shared features and independent features, both global shared information and specific details of the learning task are taken into account, achieving high efficiency and high performance in multi-task learning scenarios.

[0116] Optionally, the dynamic embedding layer may include a multi-embedding layer and a third deep neural network layer; Figure 6 The steps in the above code are used to transform the reward training data into features through each dynamic embedding layer to obtain the underlying feature embedding representation. Figure 6 As shown, it may specifically include:

[0117] Step S610, binning the reward training data to obtain bin features;

[0118] Step S620, performing quantile division processing on the reward training data to obtain quantile features;

[0119] Step S630, converting the binning feature into a first intermediate feature through the multi-embedding layer, and performing a nonlinear transformation on the quantile feature through the third deep neural network layer to obtain a second intermediate feature;

[0120] Step S640: fusing the first intermediate feature and the second intermediate feature to obtain the underlying feature embedding representation.

[0121] Among them, binning refers to the process of classifying different types of data in reward training data. For example, reward training data can include different types of data such as numerical type and categorical type. Binning can divide the continuous numerical features in the reward training data into several intervals, each interval corresponds to a discrete value, so as to convert the numerical features into categorical features, which can better handle nonlinear values ​​and outliers, and facilitate subsequent embedding layer processing.

[0122] The quantile partitioning process divides the continuous numerical features in the reward training data into certain proportions, such as quartiles, deciles, etc. It can also convert continuous features into categorical form, while also taking into account the characteristics of data distribution.

[0123] Figure 7 A schematic diagram of the network structure of an expert network provided in an embodiment of this specification.

[0124] refer to Figure 7 As shown, the expert network 700 may include a dynamic embedding layer 710, a first deep neural network layer 720 and a second deep neural network layer 730; the reward training data may be subjected to feature conversion through the dynamic embedding layer 710 to obtain an underlying feature embedding representation; the underlying feature embedding representation may be subjected to nonlinear transformation through each first deep neural network layer 720 to obtain a shared feature embedding representation of the underlying features across learning tasks; the independent features of each learning task may be extracted from the shared feature embedding representation through each second deep neural network layer 730, and the independent features may be integrated through the integration network layer 740 to obtain a corresponding feature embedding representation vector.

[0125] Optionally, the dynamic embedding layer may include a multi-embedding layer 711 and a third deep neural network layer 712; the reward training data may be binned to obtain bin features; the reward training data may be quantile divided to obtain quantile features; the bin features may be converted into first intermediate features through the multi-embedding layer 711, and the quantile features may be nonlinearly transformed through the third deep neural network layer 712 to obtain second intermediate features; the first intermediate features and the second intermediate features may be fused to obtain the underlying feature embedding representation.

[0126] To summarize, the reward prediction model training method provided in the embodiments of this specification constructs reward training data through pre-collected transaction feature data and a pre-trained initial control action recommendation model, and performs supervised learning on the pre-constructed reward prediction model based on the reward training data to obtain an initial reward prediction model; obtains auxiliary learning labels corresponding to different learning tasks, and then performs multi-task learning on the initial reward prediction model based on the auxiliary learning labels to obtain a target reward prediction model. On the one hand, the target reward prediction model obtained through multi-task learning can realize the prediction of reward signals in the reinforcement learning process of the control action recommendation model. Compared with outputting the reward signal of the control action recommendation model through expert experience, it can effectively reduce labor costs and improve the prediction output efficiency of reward signals. On the other hand, through the auxiliary learning labels of different learning tasks, multi-task learning of the target reward prediction model is realized, the rationality and effectiveness of the predicted output reward signal are improved, and the interpretability of the reward signal is improved. And by setting different auxiliary learning labels, the generalization of the target reward prediction model can be improved, and the scope of application of the target reward prediction model can be guaranteed. On the other hand, supervised learning followed by multi-task learning can effectively improve the convergence speed of the target reward prediction model, reduce the training and construction cost of the target reward prediction model, and effectively ensure the accuracy of the predicted reward signal.

[0127] See also Figure 8 , is a flow chart of a method for optimizing a control action recommendation model provided in an embodiment of this specification. In the embodiment of this specification, the method for optimizing a control action recommendation model can be applied to a terminal device or a server, and this exemplary embodiment does not specifically limit this. Figure 8 The process shown is described in detail, and the control action recommendation model optimization method in the embodiment of this specification may specifically include the following steps:

[0128] Step S810, acquiring transaction feature data, and inputting the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action;

[0129] Step S820, inputting the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by a reward prediction model training method;

[0130] Step S830, performing proximal strategy optimization on the initial control action recommendation model according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

[0131] Among them, transaction feature data refers to feature data extracted from the user's transaction data. For example, transaction feature data can be feature data such as transaction users and transaction amounts, or feature data such as transaction events and transaction locations. This example embodiment does not specifically limit the type of transaction feature data.

[0132] Predicted control actions refer to control actions recommended by the initial control action recommendation model based on transaction feature data. For example, the predicted control action can be a direct action, a text message reminder action for a transaction event, or a text message verification or face verification action for a transaction event. This example embodiment does not impose any special limitations on the control action data.

[0133] The target reward prediction model can be obtained by training the reward prediction model training method provided in the present embodiment, and the transaction feature data and the predicted control action can be input into the target reward prediction model as binary data to obtain a predicted reward signal. For example, the predicted reward signal can be a positive reward or a negative reward.

[0134] The goal of proximal policy optimization (PPO) is to maximize the expected cumulative reward by continuously adjusting the policy. The main idea is to limit the magnitude of policy changes each time the policy is updated to ensure the stability of the policy update.

[0135] After obtaining the transaction feature data, predicted management and control actions, and predicted reward signals, the initial management and control action recommendation model can be subjected to proximal strategy optimization. For example, the proximal strategy optimization of the initial management and control action recommendation model can be achieved by setting a clipping function, or by setting a penalty item. This example embodiment does not impose any special limitation on the method of proximal strategy optimization.

[0136] In an alternative embodiment, the Fig. 9 The steps in the above code are used to optimize the proximal strategy of the initial control action recommendation model and obtain the target control action recommendation model. Fig. 9 As shown, it may specifically include:

[0137] Step S910, determining an advantage estimate value according to the transaction feature data, the predicted control action and the predicted reward signal;

[0138] Step S920, constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a policy loss and a clipping function, so that the clipping function suppresses policy updates that exceed an update step range;

[0139] Step S930, with the goal of maximizing the cumulative reward, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain a target control action recommendation model.

[0140] Among them, the advantage estimate refers to the estimate used to evaluate the relative merits of the state-action pair, which measures the merits of executing the control action relative to the average level, that is, the effect of taking the control action relative to the average action in a given state. For example, the advantage estimate can be determined by constructing an advantage function and combining the advantage function with transaction feature data, predicted control actions, and predicted reward signals. For example, the advantage function can be a GAE (Generalized Advantage Estimation) function.

[0141] The policy loss can be calculated by the ratio between the new policy and the old policy and the advantage estimate. The clipping function can ensure that the update step between the new policy and the old policy is within the update step range, ensuring that the new policy will not differ too much from the old policy.

[0142] In an optional embodiment, it is also possible to Fig.10 The steps in the above code are used to optimize the proximal strategy of the initial control action recommendation model and obtain the target control action recommendation model. Fig.10 As shown, it may specifically include:

[0143] Step S1010, determining an advantage estimation value according to the transaction feature data, the predicted control action and the predicted reward signal;

[0144] Step S1020, constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a strategy loss with a penalty term set;

[0145] Step S1030, with the goal of maximizing the cumulative reward and minimizing the penalty term, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain a target control action recommendation model.

[0146] The penalty term is a function term used to suppress excessive policy updates, and the penalty term can be proportional to the KL divergence between the new and old policies. By adding the penalty term to the policy loss, when the new policy differs greatly from the old policy, the penalty term will increase, thereby suppressing excessive policy updates.

[0147] Proximal policy optimization can effectively avoid the problems of training instability and performance degradation caused by excessive policy updates. The stability of policy updates can be controlled to improve the learning optimization efficiency and reliability of the target control action recommendation model, so that the target control action recommendation model maintains good performance and robustness.

[0148] In summary, the control action recommendation model optimization method provided in the embodiments of this specification can input transaction feature data into a pre-trained initial control action recommendation model to obtain predicted control actions, and input transaction feature data and predicted control actions into a target reward prediction model to obtain predicted reward signals, and then the initial control action recommendation model can be subjected to proximal strategy optimization based on the transaction feature data, predicted control actions, and predicted reward signals to obtain a target control action recommendation model. On the one hand, the predicted reward signal is determined by the target reward prediction model trained by the reward prediction model training method in the embodiments of this specification, and is used for the optimization of the target reward prediction model, which can effectively improve the performance of the target reward prediction model and the accuracy of the output; on the other hand, the learning optimization efficiency and reliability of the target control action recommendation model can be improved through proximal strategy optimization, so that the target control action recommendation model maintains good performance and robustness.

[0149] See also Fig.11 , is a schematic diagram of the structure of a reward prediction model training device provided in an embodiment of this specification. Fig.11 As shown, the reward prediction model training device 1100 can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the reward prediction model training device 1100 includes a reward training data acquisition module 1110, a supervised learning module 1120, and a multi-task learning module 1130, which may specifically include:

[0150] The reward training data acquisition module 1110 is used to construct reward training data based on the pre-collected transaction feature data and the pre-trained initial management and control action recommendation model;

[0151] The supervised learning module 1120 is used to perform supervised learning on the pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model;

[0152] The multi-task learning module 1130 is used to obtain preset auxiliary learning labels, where the auxiliary learning labels correspond to different learning tasks, and perform multi-task learning on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

[0153] Optionally, in some implementations of this specification, the reward training data acquisition module 1110 is configured to:

[0154] Inputting the transaction feature data into the initial control action recommendation model to determine the control action data corresponding to the transaction feature data;

[0155] Reward training data is constructed based on the binary feature data consisting of the transaction feature data and the management and control action data.

[0156] Optionally, in some implementations of this specification, the supervised learning module 1120 is configured to:

[0157] matching a sample reward signal in a business-specified scoring database based on the reward training data;

[0158] The reward training data is input into the pre-built reward prediction model, and the sample reward signal is used as a supervised training label to perform supervised learning on the reward prediction model to obtain an initial reward prediction model.

[0159] Optionally, in some embodiments of the present specification, the initial reward prediction model includes at least one expert network, at least one gating network and at least one combiner, and the number of the combiners is the same as the number of the learning tasks; the multi-task learning module 1130 is configured as follows:

[0160] Using each of the expert networks to extract features from the reward training data to obtain a feature embedding representation vector;

[0161] Inputting the reward training data into each of the gating networks for model training, so that the gating network determines the probability distribution weights of each of the expert networks in different learning tasks;

[0162] The feature embedding representation vectors and the probability distribution weights are combined by the combiners corresponding to different tasks to obtain a reward signal prediction value corresponding to the reward training data;

[0163] The loss of each learning task is determined according to the reward signal prediction value and the auxiliary learning label, and the model parameters of the initial reward prediction model are updated based on the loss until the loss of the initial reward prediction model converges or reaches a predetermined number of iterative training times, thereby obtaining a target reward prediction model.

[0164] Optionally, in some implementations of this specification, the expert network includes a dynamic embedding layer, a first deep neural network layer, and a second deep neural network layer; the multi-task learning module 1130 is configured as follows:

[0165] Performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation;

[0166] Performing a nonlinear transformation on the underlying feature embedding representation through each of the first deep neural network layers to obtain a shared feature embedding representation across learning tasks in the underlying features;

[0167] The independent features of each learning task are extracted from the shared feature embedding representation through each of the second deep neural network layers, and the independent features are integrated to obtain a corresponding feature embedding representation vector.

[0168] Optionally, in some implementations of this specification, the dynamic embedding layer includes a multi-embedding layer and a third deep neural network layer; the multi-task learning module 1130 is configured as follows:

[0169] Performing binning processing on the reward training data to obtain binning features;

[0170] Performing quantile division processing on the reward training data to obtain quantile features;

[0171] The binning feature is converted into a first intermediate feature through the multi-embedding layer, and the quantile feature is nonlinearly transformed through the third deep neural network layer to obtain a second intermediate feature;

[0172] The first intermediate feature and the second intermediate feature are fused to obtain the underlying feature embedding representation.

[0173] Optionally, in some embodiments of the present specification, the auxiliary learning labels include fraud risk identification task labels, financial risk identification task labels, help identification task labels, and restriction release task labels.

[0174] See also Fig.12 , is a schematic diagram of the structure of a control action recommendation model optimization device provided in an embodiment of this specification. Fig.12 As shown, the control action recommendation model optimization device 1200 can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the control action recommendation model optimization device 1200 includes a control action generation module 1210, a reward signal prediction module 1220, and a proximal strategy optimization module 1230, which may specifically include:

[0175] A control action generation module 1210 is used to obtain transaction feature data and input the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action;

[0176] A reward signal prediction module 1220, which is used to input the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by the reward prediction model training method in the embodiment of this specification;

[0177] The proximal strategy optimization module 1230 is used to perform proximal strategy optimization on the initial control action recommendation model according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

[0178] Optionally, in some implementations of this specification, the proximal strategy optimization module 1230 is configured to:

[0179] determining an advantage estimate based on the transaction characteristic data, the predicted control action, and the predicted reward signal;

[0180] Constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a policy loss and a clipping function, so that the clipping function suppresses policy updates that exceed an update step range;

[0181] With the goal of maximizing the cumulative reward, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain the target control action recommendation model.

[0182] Optionally, in some implementations of this specification, the proximal strategy optimization module 1230 is configured to:

[0183] determining an advantage estimate based on the transaction characteristic data, the predicted control action, and the predicted reward signal;

[0184] Constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a policy loss with a penalty term;

[0185] With the goal of maximizing the cumulative reward and minimizing the penalty term, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain a target control action recommendation model.

[0186] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0187] The embodiments of this specification also provide a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the method as described in the embodiments of this specification. The specific execution process can be found in the specific description of the embodiments of this specification, and will not be repeated here.

[0188] This specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor and executes the method as described in the embodiments of this specification. The specific execution process can be found in the specific description of the embodiments of this specification, and will not be repeated here.

[0189] The embodiments of this specification also provide Fig.13 The structural diagram of the electronic device shown in FIG. Fig.13 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0190] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0191] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0192] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0193] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0194] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0195] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0196] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0200] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0201] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0202] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0203] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0205] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0206] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A reward prediction model training method, the method comprising: Construct reward training data based on pre-collected transaction feature data and pre-trained initial control action recommendation model; Performing supervised learning on a pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model; Preset auxiliary learning labels are obtained, where the auxiliary learning labels correspond to different learning tasks, and multi-task learning is performed on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

2. The reward prediction model training method according to claim 1, wherein the reward training data is constructed based on the pre-collected transaction feature data and the pre-trained initial control action recommendation model, comprising: Inputting the transaction feature data into the initial control action recommendation model to determine the control action data corresponding to the transaction feature data; Reward training data is constructed based on the binary feature data consisting of the transaction feature data and the management and control action data.

3. The reward prediction model training method according to claim 2, wherein the pre-constructed reward prediction model is subjected to supervised learning according to the reward training data to obtain an initial reward prediction model, comprising: matching a sample reward signal in a business-specified scoring database based on the reward training data; The reward training data is input into the pre-built reward prediction model, and the sample reward signal is used as a supervised training label to perform supervised learning on the reward prediction model to obtain an initial reward prediction model.

4. The reward prediction model training method according to claim 1, wherein the initial reward prediction model comprises at least one expert network, at least one gating network and at least one combiner, and the number of the combiners is the same as the number of the learning tasks; The performing multi-task learning on the initial reward prediction model according to the auxiliary learning label to obtain a target reward prediction model includes: Using each of the expert networks to extract features from the reward training data to obtain a feature embedding representation vector; Inputting the reward training data into each of the gating networks for model training, so that the gating network determines the probability distribution weights of each of the expert networks in different learning tasks; The feature embedding representation vectors and the probability distribution weights are combined by the combiners corresponding to different tasks to obtain a reward signal prediction value corresponding to the reward training data; The loss of each learning task is determined according to the reward signal prediction value and the auxiliary learning label, and the model parameters of the initial reward prediction model are updated based on the loss until the loss of the initial reward prediction model converges or reaches a predetermined number of iterative training times, thereby obtaining a target reward prediction model.

5. The reward prediction model training method according to claim 4, wherein the expert network comprises a dynamic embedding layer, a first deep neural network layer, and a second deep neural network layer; The step of extracting features from the reward training data using each of the expert networks to obtain a feature embedding representation vector includes: Performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation; Performing a nonlinear transformation on the underlying feature embedding representation through each of the first deep neural network layers to obtain a shared feature embedding representation across learning tasks in the underlying features; The independent features of each learning task are extracted from the shared feature embedding representation through each of the second deep neural network layers, and the independent features are integrated to obtain a corresponding feature embedding representation vector.

6. The reward prediction model training method according to claim 5, wherein the dynamic embedding layer comprises a multi-embedding layer and a third deep neural network layer; The step of performing feature conversion on the reward training data through each of the dynamic embedding layers to obtain an underlying feature embedding representation includes: Performing binning processing on the reward training data to obtain binning features; Performing quantile division processing on the reward training data to obtain quantile features; The binning feature is converted into a first intermediate feature through the multi-embedding layer, and the quantile feature is nonlinearly transformed through the third deep neural network layer to obtain a second intermediate feature; The first intermediate feature and the second intermediate feature are fused to obtain the underlying feature embedding representation.

7. According to the reward prediction model training method according to claim 1, the auxiliary learning labels include fraud risk identification task labels, financial risk identification task labels, help identification task labels and limit release task labels.

8. A method for optimizing a management and control action recommendation model, the method comprising: Acquire transaction feature data, and input the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action; Inputting the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by the reward prediction model training method according to any one of claims 1 to 7; The initial control action recommendation model is subjected to proximal strategy optimization according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

9. The method for optimizing the control action recommendation model according to claim 8, wherein the proximal strategy optimization of the initial control action recommendation model is performed according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model, comprising: determining an advantage estimate based on the transaction characteristic data, the predicted control action, and the predicted reward signal; Constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a policy loss and a clipping function, so that the clipping function suppresses policy updates that exceed an update step range; With the goal of maximizing the cumulative reward, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain the target control action recommendation model.

10. The method for optimizing the control action recommendation model according to claim 8, wherein the proximal strategy optimization of the initial control action recommendation model is performed according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model, comprising: determining an advantage estimate based on the transaction characteristic data, the predicted control action, and the predicted reward signal; Constructing a clipping loss function based on the advantage estimate, wherein the clipping loss function includes a policy loss with a penalty term; With the goal of maximizing the cumulative reward and minimizing the penalty term, the initial control action recommendation model is optimized in combination with the clipping loss function to obtain a target control action recommendation model.

11. A reward prediction model training device, the device comprising: A reward training data acquisition module is used to construct reward training data based on pre-collected transaction feature data and a pre-trained initial control action recommendation model; A supervised learning module, used to perform supervised learning on a pre-built reward prediction model according to the reward training data to obtain an initial reward prediction model; The multi-task learning module is used to obtain preset auxiliary learning labels, where the auxiliary learning labels correspond to different learning tasks, and perform multi-task learning on the initial reward prediction model according to the auxiliary learning labels to obtain a target reward prediction model.

12. A control action recommendation model optimization device, the device comprising: A control action generation module is used to obtain transaction feature data and input the transaction feature data into a pre-trained initial control action recommendation model to obtain a predicted control action; A reward signal prediction module, used for inputting the transaction feature data and the predicted control action into a target reward prediction model to obtain a predicted reward signal, wherein the target reward prediction model is trained by the reward prediction model training method according to any one of claims 1 to 7; A proximal strategy optimization module is used to perform proximal strategy optimization on the initial control action recommendation model according to the transaction feature data, the predicted control action and the predicted reward signal to obtain a target control action recommendation model.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 or 8 to 10 are implemented.

14. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 7 or 8 to 10.

15. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 7 or 8 to 10 are implemented.

Images