Behavior prediction method and device, model training method and device and storage medium

Abstract

The embodiment of the invention provides a behavior prediction method and device, a model training method and device and a storage medium. In the behavior prediction method, when a behavior result of a target user is predicted, a task type can be used as prompt information to be input into a behavior prediction model. According to the behavior prediction model, conditional attention coding can be performed on the to-be-predicted data based on the task type and the importance of the pre-learned user data of different dimensions in different tasks, so that the concerned features of the task type can be accurately extracted from the to-be-predicted data. Based on the extracted features, the behavior prediction model can accurately predict the behavior result of the to-be-predicted data in the behavior stage corresponding to the task type.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a behavior prediction, model training method, device and storage medium. Background Art

[0002] In the field of financial credit, loan activities mainly include application, disbursement and repayment. In the application stage, users need to submit loan applications. If the loan application is approved, it will enter the disbursement stage. Users can withdraw funds in the disbursement stage. In the repayment stage after the withdrawal is completed, some users may default. Therefore, the financial credit field faces credit risk issues.

[0003] In order to avoid credit risk, a risk behavior prediction model can be trained and used to detect credit risk during the application process. However, in practice, the number of users who have overdue payments is relatively small, which means that the number of samples for training the risk behavior prediction model is insufficient, resulting in a low accuracy of the trained risk behavior prediction model and an inability to accurately output risk detection results. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Multiple aspects of the present application provide a behavior prediction, model training method, device and storage medium in a multi-stage service scenario, so as to accurately predict user behavior and improve the accuracy of behavior prediction results.

[0005] An embodiment of the present application provides a behavior prediction method in a multi-stage service scenario, including: responding to a behavior prediction request for a target user, obtaining data to be predicted corresponding to the behavior prediction request; the data to be predicted includes: a task type corresponding to the prediction request and multi-dimensional user data of the target user; the task type corresponds to a target behavior stage in the multi-stage service scenario; using a behavior prediction model, taking the task type as a condition, performing feature extraction on the data to be predicted based on a conditional attention mechanism, and obtaining a conditional attention feature of the data to be predicted; based on the conditional attention feature, determining a prediction label of the data to be predicted under the task type, so as to obtain a behavior prediction result of the target user at the target behavior stage.

[0006] Optionally, taking the task type as a condition, feature extraction is performed on the data to be predicted based on a conditional attention mechanism to obtain conditional attention features of the data to be predicted, including: vectorizing the multi-dimensional user data and the task type of the data to be predicted respectively to obtain a user feature vector and a task feature vector; inputting the user feature vector and the task feature vector into at least one transformation network respectively; utilizing the at least one transformation network, using a first weight parameter corresponding to the task type and a second attention weight corresponding to the multi-dimensional user data, to perform conditional attention-based feature encoding on the task feature vector and the user feature vector respectively to obtain at least one encoding result; wherein, in two adjacent transformation networks, the encoding result output by the previous transformation network is input into the next transformation network together with the user feature vector and the task feature vector; obtaining the encoding result output by the last transformation network in the at least one transformation network as the conditional attention feature.

[0007] Optionally, the first weight parameter corresponding to the task type and the second weight parameter corresponding to the multi-dimensional user data are used to perform conditional attention-based feature encoding on the task feature vector and the user feature vector, respectively, to obtain at least one encoding result, including: in any transformation network, using an embedding layer to embed the identification information of the transformation network into the task feature vector to obtain an updated task feature vector; using a linear transformation layer, using the first weight parameter and the second weight parameter, respectively performing linear transformation on the updated task feature vector and the user feature vector to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector; concatenating the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector to obtain a conditional coding matrix; using a multi-head attention layer, performing multi-head attention calculation on the conditional coding matrix to obtain a first conditional attention distribution vector of the data to be predicted; using a coding output layer, outputting the conditional attention feature corresponding to the transformation network according to the first conditional attention distribution vector and the residual information input into the transformation network as the encoding result of the transformation network.

[0008] Optionally, the first weight parameter and the second weight parameter are used to perform linear transformation on the updated task feature vector and the user feature vector, respectively, to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector, including: using the first weight parameter to perform linear transformation calculation on the task feature vector to obtain a key matrix and a value matrix of the task feature vector; using the second weight parameter to perform linear transformation calculation on the user feature vector to obtain a key matrix, a query matrix and a value matrix of the user feature vector; splicing the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector to obtain a conditional coding matrix, including: splicing the key matrix corresponding to the task feature vector with the key matrix corresponding to the user feature vector to obtain a target key matrix containing the task type; splicing the value matrix corresponding to the task feature vector with the value matrix corresponding to the user feature vector to obtain a target value matrix containing the task type; using the target key matrix, the target value matrix and the query matrix as the conditional coding matrix.

[0009] Optionally, based on the first conditional attention distribution vector and the residual information input into the transformation network, the conditional attention feature of the data to be predicted is output, including: residually connecting the first conditional attention distribution vector and the target residual information; normalizing the first conditional attention distribution vector after the residual connection to obtain a second conditional attention distribution vector; performing feedforward calculation on the second conditional attention distribution vector to obtain a third conditional attention distribution vector; residually connecting the second conditional attention distribution vector and the third conditional attention distribution vector; normalizing the third conditional attention distribution vector after the residual connection to obtain the conditional attention feature.

[0010] Optionally, before adding the target residual information to the first conditional attention distribution vector of the data to be predicted, it also includes: if the transformation network is the first transformation network among the at least one transformation network, determining the user feature vector as the target residual information; or, if the transformation network is not the first transformation network among the at least one transformation network, determining the conditional attention feature output by the previous transformation network of the transformation network as the target residual information.

[0011] Optionally, the multi-stage service scenario includes: a credit scenario; the task type includes: an application approval result prediction task, a spending behavior prediction task, or a repayment behavior prediction task.

[0012] An embodiment of the present application also provides a method for training a behavior prediction model in a multi-stage service scenario, including: obtaining sample data in the multi-stage service scenario, any sample data including multi-dimensional user data of a single user, a task type, and a true label of the sample data under the task type; wherein the task type is determined according to the behavior stage corresponding to the sample data in the multi-stage service scenario; the multiple sample data correspond to multiple behavior stages; using a behavior prediction model, taking the task types of each of the multiple sample data as conditions, and extracting features from the multiple sample data based on a conditional attention mechanism to obtain conditional attention features of the multiple sample data; determining predicted labels of the multiple sample data under their respective task types based on the conditional attention features of the multiple sample data; and training the behavior prediction model with the goal of converging the gap between the true labels and predicted labels of the multiple sample data under their respective task types to a specified range.

[0013] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided in the embodiment of the present application.

[0014] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, which can implement the steps in the method provided in the embodiment of the present application when the computer program is executed by a processor.

[0015] In an embodiment of the present application, when predicting the behavior results of the target user, the task type can be input into the behavior prediction model as prompt information. The behavior prediction model can perform conditional attention encoding on the data to be predicted based on the task type and the importance of user data of different dimensions learned in advance in different tasks, so as to accurately extract the features that the task type focuses on from the data to be predicted. Based on the extracted features, the behavior prediction model can accurately predict the behavior results of the data to be predicted at the behavior stage corresponding to the task type. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a behavior prediction model training method in a multi-stage service scenario provided by an exemplary embodiment of the present application;

[0018] Figure 2 A schematic diagram of the structure of a behavior prediction model provided by an exemplary embodiment of the present application;

[0019] Figure 3 A flowchart of a behavior prediction method in a multi-stage service scenario provided by an exemplary embodiment of the present application;

[0020] Figure 4 A schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0023] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0024] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0025] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] In order to solve the technical problem in the prior art that the risk behavior prediction model has low accuracy due to sample data selection bias, a solution is provided in some embodiments of the present application. The technical solutions provided in each embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0027] Figure 1 is a flow chart of a behavior prediction model training method in a multi-stage service scenario provided by an exemplary embodiment of the present application. The method may include: Figure 1 Steps shown:

[0028] Step 101: Acquire multiple sample data in a multi-stage service scenario, where any sample data includes multi-dimensional user data of a single user, a task type, and a true label of the sample data under the task type; wherein the task type is determined based on the behavior stage corresponding to the sample data in the multi-stage service scenario; and the multiple sample data correspond to multiple behavior stages.

[0029] Step 102: Utilize the behavior prediction model, take the task type of each of the plurality of sample data as a condition, perform feature extraction on the plurality of sample data based on the conditional attention mechanism, and obtain conditional attention features of the plurality of sample data.

[0030] Step 103: Determine the predicted labels of the sample data under the respective task types according to the conditional attention features.

[0031] Step 104 : training the behavior prediction model with the goal of converging the gap between the real labels and the predicted labels of the plurality of sample data under the respective task types to a specified range.

[0032] The training method provided in this embodiment is used to train a behavior prediction model, which is used to predict the behavior results of users in different behavior stages in a multi-stage service scenario. Among them, the multi-stage service scenario may include multiple behavior stages, and there is a certain correlation between the multiple behavior stages. For example, the correlation relationship can be expressed as: among multiple behavior stages, the behavior results of the previous behavior stage can be used as a prerequisite for whether to enter the next behavior stage. Among them, the multi-stage service scenario may include: credit scenarios, advertising and marketing scenarios, online education scenarios, etc.

[0033] For example, in a credit scenario, the behavior stages of the credit scenario can be divided into an application stage, a disbursement stage, or a repayment stage according to the process links of the credit scenario. Among them, the behavior results in the application stage may include: approval passed or approval failed. The behavior results in the disbursement stage may include: funds disbursed or funds not disbursed. The behavior results in the repayment stage may include: overdue repayment or not overdue repayment. The sample data in the credit scenario is usually composed of user data of loan application users in the historical stage. Based on user data from multiple stages, a behavior prediction model for the credit scenario can be trained to predict the user's behavior results at different behavior stages based on the user's user data when receiving the user's credit application, so that the credit risk is within a controllable and perceptible range.

[0034] For example, in an advertising marketing scenario, the behavior stage can be divided into an ad browsing stage and a purchase stage. Among them, the behavior results in the ad clicking stage may include: clicking on the ad link or not clicking on the ad link. The behavior results in the purchase stage may include: purchase or not purchase. Among them, the user behavior in the ad browsing stage and the user behavior in the purchase stage have a certain correlation, that is, after the user clicks the ad link in the ad browsing stage, he can enter the purchase stage. When training the behavior detection model for the advertising marketing scenario, joint modeling can be performed based on the relevant prediction tasks of multiple behavior stages.

[0035] For another example, in an online education scenario, the behavior stages can be divided into: entry stage, primary stage, intermediate stage, and advanced stage. Among them, the behavior results of the entry stage may include: passing the entry test or failing the entry test. The behavior results of the primary stage may include: passing the primary test or failing the primary test; and so on. Among them, the behaviors of multiple stages have a certain correlation, that is, after passing the entry test, the user can enter the primary stage; after passing the primary test, the user can enter the intermediate stage; after passing the intermediate test, the user can enter the advanced stage. When training the behavior detection model for the online education scenario, joint modeling can be performed based on the related prediction tasks of multiple behavior stages.

[0036] The sample data in the multi-stage service scenario includes sample data corresponding to multiple different behavior stages. For example, in the credit scenario, some users enter the application stage but fail to pass the approval, so they cannot enter the subsequent disbursement stage. Another part of the users enter the disbursement stage after passing the approval, but have not yet disbursed funds, so they cannot enter the subsequent repayment stage. The sample data generated by the credit scenario may include: sample data in the application stage and sample data in the disbursement stage. Based on the sample data corresponding to different behavior stages, the behavior detection model can be trained to learn the intrinsic connection between different user data and the behavior results of different behavior stages. Among them, the sample data of different behavior stages are annotated with the real labels corresponding to the behavior stages. For example, in the credit scenario, the sample data that stops at the application stage is annotated with the real label of whether the approval is passed. The sample data that stops at the disbursement stage is annotated with the real label of whether the disbursement is made. The sample data that stops at the repayment stage is annotated with the real label of whether it is overdue.

[0037] The following will take any sample data as an example to illustrate the training process of the behavior prediction model. When there is a batch of sample data, the training process of each sample data in the batch of sample data can be performed with reference to the training method provided in this embodiment. Among them, any sample data includes multi-dimensional user data and task type of a single user. Among them, multi-dimensional user data refers to data that describes the characteristics of loan application users from multiple dimensions. The multiple dimensions may include: monthly sales, average daily number of orders, liability information, etc. Among them, user data of different dimensions contribute differently to prediction tasks at different behavior stages. Among them, the task type is determined according to the behavior stage corresponding to the sample data in the multi-stage service scenario. Sample data corresponding to different behavior stages correspond to different task types. For example, when the user corresponding to the sample data stops at the application stage, the task type corresponding to the sample data is the application approval result prediction task. When the user corresponding to the sample data stops at the withdrawal stage, the task type corresponding to the sample data is the withdrawal behavior prediction task. When the user corresponding to the sample data stops at the repayment stage, the task type corresponding to the sample data is the repayment behavior prediction task. The task type can be used as a sample data feature of the task dimension to provide task prompts for the behavior prediction model.

[0038] After obtaining the sample data in the multi-stage service scenario, the sample data can be input into the behavior prediction model. In this embodiment, the behavior prediction model can be a transformation model based on the attention mechanism. In step 202, the behavior prediction model can be used to extract features of the sample data based on the conditional attention mechanism, taking the task type as a condition, and obtaining the conditional attention features of the sample data. Among them, the conditional attention mechanism refers to a feature extraction mechanism that selectively extracts information that the condition focuses on from the input data under specific conditions. Based on the conditional attention mechanism, the preference of different task types for knowledge can be captured, so that when a given task type is used as a prompt word, the knowledge information preferred by the task type can be accurately extracted, and the prediction result under the task type can be accurately output. In this embodiment, taking the task type as a condition, when the sample data is feature extracted based on the attention mechanism, the information that the task focuses on can be selectively extracted from the sample data to obtain the conditional attention features of the sample data.

[0039] In some exemplary embodiments, the behavior prediction model includes at least one transformation network, and any transformation network can take the task type as a condition and extract features of the sample data based on the conditional attention mechanism to obtain the conditional attention features of the sample data.

[0040] Before the sample data is input into the at least one transformation network, the multi-dimensional user data and the task type of the sample data may be vectorized to obtain a user feature vector and a task feature vector. Through the vectorization process, the multi-dimensional user data and the task type can be mapped to the vector space, so as to facilitate feature extraction in the vector space. For example, the vectorization result of the sample data a may include: a task feature vector eT and a user feature vector Ex, Ex = [e1, e2, ..., em]. Wherein, e1, e2, ..., em are respectively the user vectorization results of m dimensions, and m is a positive integer. Optionally, the user feature vector and the task feature vector may be input into the at least one transformation network. Wherein, any transformation network includes a weight parameter corresponding to the task type and a weight parameter corresponding to the multi-dimensional user data. For the convenience of description and distinction, the weight parameter corresponding to the task type is described as the first weight parameter, and the weight parameter corresponding to the user multi-dimensional data is described as the second weight parameter. The "first and second" are only used to distinguish similar or identical description objects, and do not limit the number of parameters. The first weight parameter corresponding to the task type and the second weight parameter corresponding to the multi-dimensional user data are used to extract the attention features of the task type from the sample data. During the training process, the first weight parameter and the second weight parameter in the transformation network are the parameters to be learned.

[0041] In the behavior prediction model, the at least one transformation network can be used to adopt the first weight parameter corresponding to the task type and the second weight parameter corresponding to the multi-dimensional user data to perform feature encoding on the task feature vector corresponding to the task type and the user feature vector corresponding to the multi-dimensional user data, respectively, to obtain at least one encoding result. Among them, in two adjacent transformation networks, the encoding result output by the previous transformation network is used as residual information, and is input into the next transformation network together with the user feature vector and the task feature vector. That is, when there are multiple transformation networks, in adjacent transformation networks, the output layer of the previous transformation network is connected to the residual connection and normalization layer of the next transformation network, thereby deepening the depth of the behavior prediction model through multiple transformation networks and improving the performance of the behavior prediction model. Based on the above structure, the encoding result output by the last transformation network in the at least one transformation network can be obtained as the conditional attention feature of the sample data.

[0042] The following will take any transformation network in the at least one transformation network as an example to further exemplify the structure of the transformation network and the data processing method.

[0043] Optionally, any transformation network may include at least: an embedding layer, a linear transformation layer, a multi-head attention layer, and an encoding output layer.

[0044] In any transformation network, the identification information of the transformation network can be embedded into the task feature vector using the embedding layer to obtain an updated task feature vector. The identification information of the transformation network can be randomly initialized. When there are multiple transformation networks, the identification information of different transformation networks is different. After the same task feature vector is input into different transformation networks, the identification information of different transformation networks is respectively embedded into the input task feature vector, and random interference can be added to the task vector to differentiate the processing objects of different transformation networks, thereby improving the prediction effect of the model.

[0045] Optionally, an optional method for embedding the identification information of the transformation network into the task feature vector may include: initializing the layer coding vector corresponding to the identification information of the transformation network; concatenating the layer coding vector and the task feature vector to obtain an updated task feature vector. For example, the behavior prediction model includes three transformation networks, namely: transformation network L1, transformation network L2, and transformation network L3. Among them, the identification information of transformation network L1, transformation network L2, and transformation network L3 are L1, L2, and L3, respectively. L1, L2, and L3 are randomly initialized, respectively, to obtain layer coding vectors eL1, eL2, and eL3. After the task feature vector eT of the sample data a is input into the transformation network L1, in the embedding layer of the transformation network L1, the layer coding vector eL1 and the task feature vector eT can be concatenated to obtain an updated task feature vector eTL1. Similarly, after the task feature vector eT of the sample data a is input into the transformation network L2, in the embedding layer of the transformation network L2, the layer coding vector eL2 and the task feature vector eT can be concatenated to obtain an updated task feature vector eTL2, which will not be repeated here.

[0046] The user feature vector output by the embedding layer and the updated task feature vector are input into the linear transformation layer. In the linear transformation layer, the first weight parameter corresponding to the task type and the second weight parameter corresponding to the multi-dimensional user data are used to perform linear transformation on the input updated task feature vector and the user feature vector, respectively, to obtain the transformation matrix of the updated task feature vector and the transformation matrix of the user feature vector. Among them, the linear transformation is used to generate a query (Query), a key (Key) and a value (Value) according to the input vector. The implementation method of the linear transformation will be exemplified below. Optionally, in the linear transformation layer, the first weight parameter may be used to perform a linear transformation calculation on the updated task feature vector to obtain a key matrix and a value matrix of the updated task feature vector. Among them, the first weight parameter may include two weight parameter matrices, one weight parameter matrix is ​​used to perform a linear transformation calculation on the updated task feature vector to obtain a key matrix; the other weight parameter matrix is ​​used to perform a linear transformation calculation on the updated task feature vector to obtain a value matrix. For example, the weight parameter matrix included in the first weight parameter can be expressed as: Wkl and Wvl. The process of linear transformation of the updated task feature vector eTL can be expressed as: key matrix Pk = eTL*Wkl, value matrix Pv = eTL*Wvl.

[0047] In the linear transformation layer, the second weight parameter can be used to perform a linear transformation calculation on the user feature vector to obtain a key matrix, a query matrix and a value matrix of the user feature vector. Among them, the second weight parameter may include three weight parameter matrices, one weight parameter matrix is ​​used to perform a linear transformation on the user feature vector to obtain a key matrix; another weight parameter matrix is ​​used to perform a linear transformation on the user feature vector to obtain a query matrix; and another weight parameter matrix is ​​used to perform a linear transformation on the user feature vector to obtain a value matrix. For example, the weight parameter matrix included in the second weight parameter can be expressed as: Wk2, Wq, Wv2. The process of linear transformation of the user feature vector Ex can be expressed as: key matrix K = Ex*Wk2, query matrix Q = Ex*Wq, value matrix V = Ex*Wv2.

[0048] Among them, the first weight parameter and the second weight parameter are parameters to be learned during the training process, and can be continuously optimized with the iterative training process of the model to learn the features that different task types focus on.

[0049] After obtaining the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector based on the above implementation, the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector can be spliced ​​to obtain a conditional coding matrix. Optionally, the key matrix corresponding to the task feature vector and the key matrix corresponding to the user feature vector can be spliced ​​to obtain a target key matrix containing the task type; that is, the key matrix Pk and the key matrix K are spliced ​​to obtain a target key matrix KT. The value matrix corresponding to the task feature vector and the value matrix corresponding to the user feature vector are spliced ​​to obtain a target value matrix containing the task type; that is, the value matrix Pv and the value matrix V are spliced ​​to obtain a target value matrix VT. The above-mentioned target key matrix KT, target value matrix VT and query matrix Q can be used as conditional coding matrices.

[0050] After obtaining the conditional coding matrix based on the above implementation, the multi-head attention layer can be used to perform multi-head attention calculation on the conditional coding matrix to obtain the first conditional attention distribution vector of the data to be predicted. Optionally, the target key matrix, target value matrix and query matrix obtained in the above steps can be input into the multi-head attention layer. In the multi-head attention layer, the target key matrix, target value matrix and query matrix are multi-headed attention calculated to obtain the first conditional attention distribution vector of the sample data.

[0051] The first conditional attention distribution vector is input into the encoding output layer. In the encoding output layer, the conditional attention feature corresponding to the transformation network can be output according to the first conditional attention distribution vector and the residual information input into the transformation network. Optionally, the encoding output layer may include at least a first residual connection and a normalization layer (Add&Normlayer), a feed forward layer (Feed Forward), and a second residual connection and a normalization layer connected in sequence.

[0052] Optionally, after the first conditional attention distribution vector is input into the encoding output layer, the first residual connection and the normalization layer can be used to perform a residual connection between the first conditional attention distribution vector and the target residual information. The residual connection refers to adding the input of the network (i.e., the first conditional attention distribution vector) and the target residual information. In the case of a deep network structure, adding residual information to the input of the network is beneficial to reduce the probability of the gradient vanishing problem during the gradient back propagation to update the model parameters.

[0053] Optionally, for any transformation network, if the transformation network is the first transformation network in the at least one transformation network, the user feature vector is determined as the target residual information. That is, the first residual connection and normalization layer of the first transformation network can be connected to the embedding layer to obtain the user feature vector from the embedding layer as the target residual information. Optionally, if the transformation network is not the first transformation network in the at least one transformation network, the conditional attention feature output by the previous transformation network of the transformation network is determined as the target residual information. That is, the first residual connection and normalization layer of the non-first transformation network can be connected to the encoding output layer of the previous transformation network to obtain the conditional attention feature from the encoding output layer of the previous transformation network as the target residual information. The first residual connection and normalization layer can normalize the first conditional attention distribution vector after the residual connection to obtain the second conditional attention distribution vector.

[0054] The second conditional attention distribution vector is input into the feedforward layer. In the feedforward layer, the second conditional attention distribution vector can be feedforwarded to obtain a third conditional attention distribution vector. The third conditional attention distribution vector can be input into the second residual connection and normalization layer.

[0055] In the residual connection and normalization layer, the second conditional attention distribution vector and the third conditional attention distribution vector can be residually connected, and the third conditional attention distribution vector after the residual connection can be normalized to obtain the conditional attention feature.

[0056] After obtaining the conditional attention features, the conditional attention features can be used to calculate the predicted label of the sample data at the behavior stage corresponding to the task type. After obtaining the predicted label, the gap between the true label and the predicted label of the sample data at the behavior stage can be calculated as the training loss of the model, and the behavior prediction model is trained with the goal of converging to a specified range between the true label and the predicted label of the sample data at the behavior stage. Among them, when calculating the training loss of the model, a binary classification loss function can be used, such as a cross entropy loss function or a mean square error loss function, etc. This embodiment includes but is not limited to this.

[0057] In this implementation, by adding task types to sample data, explicit modeling of behavior prediction tasks at different behavior stages in multi-stage service scenarios is achieved. The behavior prediction model can learn the importance of user data of different dimensions in sample data in different tasks based on the task type, thereby reducing the learning error caused by sample bias. On the one hand, there is no need to set up a multi-task branch network for behavior prediction tasks at different behavior stages, and different prediction tasks can be achieved using a unified model structure, thereby simplifying the model structure. On the other hand, based on the learned importance of user data of different dimensions in different tasks, the behavior results of the input data at the behavior stage corresponding to the task type can be accurately predicted according to the task type and input data during the inference stage.

[0058] The following will be combined Figure 2 , taking the multi-stage service scenario implemented as a credit scenario as an example, the training method of the behavior prediction model provided in the embodiment of the present application is further illustrated.

[0059] During the training process, sample data can be obtained from multiple behavioral stages included in the credit scenario. Among them, the multiple stages include: application stage, expenditure stage and repayment stage. Assume that the number of sample data in the application stage, expenditure stage and repayment stage are N1, N2, and N3 respectively, and any sample data includes feature data of m dimensions, and N1, N2, N3, and m are positive integers. Among them, a dimension is used to describe the characteristics of the sample data from a specific aspect. For example, in the application stage, user data can be obtained from m dimensions to obtain multiple sample data, each of which is m dimensions. Any sample data may include m user feature columns, and the m user feature columns can be described in the following format: {x1=xzx, x2=xxx,…, xm=xxx}. For example, in this embodiment, the m dimensions may include: the user's monthly sales, average daily number of orders, debt information, etc.

[0060] Among them, each sample data is marked with a scenario label. The label of N1 sample data in the application stage is whether the application is approved, the label of N2 sample data in the expenditure stage is whether there is expenditure behavior, and the label of N3 sample data in the repayment stage is whether overdue occurs.

[0061] In this embodiment, in addition to the above m dimensions, a task feature dimension is added to the sample data according to the stage to which the sample data belongs. Among them, the task feature dimension uses different tasks corresponding to the sample data as prompt words to train the risk behavior prediction model's ability to capture sample data knowledge under different task types. Among them, the N1 sample data in the application stage are mainly used to train the risk behavior prediction model's ability to predict application results, the N2 sample data in the expenditure stage are mainly used to train the risk behavior prediction model's ability to predict expenditure behavior, and the N3 sample data in the repayment stage are mainly used to train the risk detection module's ability to predict overdue behavior. Based on this, a task feature column can be added to the N1 sample data in the application stage, and the value of the task feature column is "application"; a task feature column can be added to the N2 sample data in the expenditure stage, and the value of the task feature column is "withdrawal"; a task feature column can be added to the N3 sample data in the repayment stage, and the value of the task feature column is "risk". Taking the repayment stage as an example, after adding the task feature column, the description of any sample data can be updated as: {task feature column = risk, x1 = xzx, x2 = xxx, …, xm = xxx}.

[0062] After the sample data of each stage is determined based on the above implementation, the sample data of each stage may be mixed to obtain N sample data and labels corresponding to the N sample data, where N=N1+N2+N3.

[0063] Input N sample data into the network model. The following will take any sample data among the N sample data as an example to illustrate the processing process of the network model.

[0064] In the network model, in the embedding vector layer (Embedding Layer), the sample data can be encoded to obtain m+1 encoding vectors corresponding to the sample data. In this embodiment, in order to facilitate subsequent calculations, the encoding vector corresponding to the sample data is divided into two parts, namely the encoding vector eT of the task feature column and the encoding vector Ex=[e1,e2,…,em] of the user feature column. Among them, the dimension of each encoding vector e is 1×d. Among them, the encoding vector eT of the task feature column can be obtained by random initialization. When the task values ​​corresponding to the task feature column are different, different task values ​​can correspond to different initialized encoding vectors eT to distinguish different task values.

[0065] Based on the encoding vectors of the task features and the encoding vectors of the user features of the sample data obtained in the above steps, a multi-layer task-conditioned transformer network (Task-Conditioned Transformer), hereinafter referred to as a transformer network, can be input. The structures of the multiple transformer networks are similar, and any one of the transformer networks will be used as an example for illustration.

[0066] In the transformation network, a vector eL can be used to represent the transformation network, and eL can be 1×d dimensional. After the encoding vector corresponding to the sample data is input into the transformation network, the encoding vector eT of the task feature column and the vector eL corresponding to the transformation network can be concatenated to obtain a concatenated vector eTL. The value of the vector eL is obtained by initialization. When the prediction model includes multiple transformation networks, different transformation networks can correspond to different vectors eL, so that random noise is added to the encoding vectors of the feature columns input by different transformation networks based on different vectors eL, thereby improving the generalization performance of the transformation network.

[0067] Among them, the spliced ​​eTLs are input as follows Figure 2 The linear transformation layer shown in the figure obtains the key matrix Pk and value matrix Pv of the task feature column in the layer; wherein the dimensions of Pk and Pv can be 1×d. The linear transformation layer may include two multilayer perceptron (MLP) networks. In MLP, the calculation process of Pk and Pv can be expressed as: Pkl = eTL*Wkl, Pvl = eTL*Wvl. Wherein Wkl and Wvl are the model parameters to be learned.

[0068] Among them, the encoding vector Ex of the user feature column can be input as follows Figure 2 The linear transformation layer shown is used to calculate the key matrix K, query matrix K and value matrix K of the user feature column. Among them, K = Ex*Wk2, Q = Ex*Wq, V = Ex*Wv2, where Wk, Wq, Wv are the parameters to be learned with a dimension of d×d. In the first round of training, the values ​​of Wk2, Wq, Wv2 are obtained by initialization. In each subsequent round of iterative training, the values ​​of Wk2, Wq, Wv2 are continuously optimized according to the loss of the network model.

[0069] The matrix Pk and the matrix K can be concatenated to obtain the matrix KT containing the task type, whose dimension is (m+1)×d; the matrix Pv and the matrix V can be concatenated to obtain the matrix VT containing the task type, whose dimension is (m+1)×d. Based on this step, the task features can be added to the user features, so as to prompt the task type corresponding to the user features.

[0070] After obtaining vector Q, vector KT and vector VT based on the above steps, the attention value is calculated based on the multi-head attention mechanism. Among them, vector Q, vector KT and vector VT can be input into Figure 2 The multi-head attention layer shown. After performing weighted calculations on vector Q, vector KT, and vector VT, the multi-head attention layer outputs a conditional attention distribution vector Ax,L, where the dimension of the conditional attention distribution vector Ax,L is m×d.

[0071] The conditional attention distribution vector Ax,L is input as Figure 2 The first residual connection and normalization layer shown in FIG. 1 is used to obtain Axn,L. In the residual connection and normalization layer, residual information can be added to the conditional attention distribution vector Ax,L, and normalization calculation can be performed after adding the residual information, that is, Axn,L = LayerNorm(Ax + Ax,L). LayerNorm() represents a normalization operation. Ax is the residual information. When the network model contains multiple transformation networks, for the first transformation network, the residual information is the encoding vector of the user feature column, that is, Ax = Ex; for non-first transformation networks, the residual information Ax = Axo,L, where Axo,L is the output of the previous transformation network.

[0072] Next, the normalized matrix Axn,L is input as Figure 2 The feedforward layer (FeedForward) shown in Figure 1 obtains the matrix Axf,L.

[0073] The matrix Axf,L output by the feedforward layer and the matrix Axn,L output by the first residual connection and normalization layer can be input into the second residual connection and normalization layer to obtain the output matrix Axo,L, where Axo,L = LayerNorm(Axn,L + Axf,L).

[0074] Based on the above calculation process, the matrix Axo,L of the last transformation network output in the network model can be obtained, and the matrix Axo,L is input as follows Figure 2 The classification layer (Sigmoid) shown in Figure 1. In the classification layer, the matrix Axo,L can be calculated to obtain the predicted label of the sample data. Based on the predicted label and the true label of the sample data, the binary classification loss function can be calculated.

[0075] Based on the above method, the binary classification loss can be calculated for each of the N sample data, and the total classification loss of the N sample data can be obtained. Among them, the binary classification loss L of the N sample data can be expressed by the following function:

[0076]

[0077] Among them, i represents the sequence number of the sample data, Represents the predicted label of the i-th sample data, y i Represents the true label of the i-th sample data. Based on the total classification loss, the gradient can be calculated and the network can be updated in reverse until the model converges.

[0078] In this embodiment, tasks such as application, expenditure, and risk detection are modeled into different query tasks according to the scenario of sample data. During the training process, the model learns the weights of different feature dimensions in different tasks, so that when a prompt word (i.e., task type) is given, the detection result can be output according to the learned weights.

[0079] In addition to the training method of the behavior prediction model provided in the above embodiment, the embodiment of the present application also provides a behavior prediction method in a multi-stage service scenario, such as Figure 3 As shown, the method mainly includes the following steps:

[0080] Step 301, respond to a behavior prediction request for a target user, and obtain the data to be predicted corresponding to the behavior prediction request; the data to be predicted includes: the task type corresponding to the prediction request and multi-dimensional user data of the target user; the task type corresponds to the target behavior stage in a multi-stage service scenario.

[0081] Step 302: Utilize the behavior prediction model, take the task type as a condition, perform feature extraction on the data to be predicted based on the conditional attention mechanism, and obtain the conditional attention feature of the data to be predicted.

[0082] Step 303: Determine a prediction label of the data to be predicted under the task type according to the conditional attention feature, so as to obtain a behavior prediction result of the target user at the target behavior stage.

[0083] In some exemplary embodiments, the multi-stage service scenarios include: credit scenarios, advertising marketing scenarios, online education scenarios, etc. Taking the credit scenario as an example, the task types include: application approval result prediction task, expenditure behavior prediction task, or repayment behavior prediction task. Taking the advertising marketing scenario as an example, the task types include: advertisement click result prediction task or purchase result prediction task.

[0084] Taking the credit scenario as an example, the behavior prediction request can be initiated by an administrator user related to the credit service. When receiving a loan request from a credit applicant, the administrator user can initiate a behavior prediction request for the credit applicant to the electronic device. The electronic device can perform a behavior prediction operation based on a behavior prediction model. Among them, the behavior prediction model is trained based on the model training method recorded in the aforementioned embodiments, which will not be repeated here. Among them, the behavior prediction request can be used to predict the behavior results of the credit applicant at different behavior stages. Based on the records of the aforementioned embodiments, it can be known that according to the process links of the credit scenario, the behavior stage of the credit scenario can be divided into an application stage, a spending stage or a repayment stage. The task types of different behavior stages are different. When predicting behavior, the electronic device can obtain the task type as prompt information to prompt the behavior prediction model to perform the corresponding behavior prediction task, thereby obtaining the behavior prediction result of the credit applicant at the behavior stage corresponding to the task type.

[0085] The target behavior stage can be any behavior stage in a multi-stage scenario.

[0086] In some optional embodiments, the administrator user may provide task types corresponding to prediction requests according to the behavior prediction requirements for different behavior stages. For example, an application approval result prediction task at the application stage (i.e., the target behavior stage), a spending behavior prediction task at the spending stage, or a repayment behavior prediction task at the repayment stage may be provided.

[0087] In other optional embodiments, the electronic device may determine the target behavior stage according to the current behavior stage of the credit applicant to automatically determine the task type. In this embodiment, if the current behavior stage of the credit applicant is the application stage (i.e., the target behavior stage), the task type may be determined as an application approval result prediction task to predict whether the credit applicant has passed the approval. If the current behavior stage of the credit applicant is the expenditure stage, the task type may be determined as an expenditure behavior prediction task to predict whether the credit applicant will spend funds. If the current behavior stage of the credit applicant is the repayment stage, the task type may be determined as a repayment behavior prediction task to predict whether the credit applicant will have overdue repayments.

[0088] In some exemplary embodiments, the task type is used as a condition, and features of the data to be predicted are extracted based on a conditional attention mechanism to obtain conditional attention features of the data to be predicted, including: vectorizing the multi-dimensional user data and the task type of the data to be predicted respectively to obtain a user feature vector and a task feature vector; inputting the user feature vector and the task feature vector into at least one transformation network respectively; utilizing the at least one transformation network, using a first weight parameter corresponding to the task type and a second attention weight corresponding to the multi-dimensional user data, respectively performing conditional attention-based feature encoding on the task feature vector and the user feature vector to obtain at least one encoding result; wherein, in two adjacent transformation networks, the encoding result output by the previous transformation network is input into the next transformation network together with the user feature vector and the task feature vector; obtaining the encoding result output by the last transformation network in the at least one transformation network as the conditional attention feature.

[0089] In some exemplary embodiments, a first weight parameter corresponding to the task type and a second weight parameter corresponding to the multi-dimensional user data are used to perform conditional attention-based feature encoding on the task feature vector and the user feature vector, respectively, to obtain at least one encoding result, including: in any transformation network, using an embedding layer to embed the identification information of the transformation network into the task feature vector to obtain an updated task feature vector; using a linear transformation layer, using the first weight parameter and the second weight parameter, respectively, to perform linear transformation on the updated task feature vector and the user feature vector, to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector; concatenating the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector to obtain a conditional coding matrix; using a multi-head attention layer, performing multi-head attention calculation on the conditional coding matrix to obtain a first conditional attention distribution vector of the data to be predicted; using an encoding output layer, outputting the conditional attention feature corresponding to the transformation network according to the first conditional attention distribution vector and the residual information input into the transformation network as the encoding result of the transformation network.

[0090] In some exemplary embodiments, the first weight parameter and the second weight parameter are used to perform linear transformation on the updated task feature vector and the user feature vector, respectively, to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector, including: using the first weight parameter to perform linear transformation calculation on the task feature vector to obtain a key matrix and a value matrix of the task feature vector; using the second weight parameter to perform linear transformation calculation on the user feature vector to obtain a key matrix, a query matrix and a value matrix of the user feature vector; concatenating the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector to obtain a conditional coding matrix, including: concatenating the key matrix corresponding to the task feature vector with the key matrix corresponding to the user feature vector to obtain a target key matrix containing the task type; concatenating the value matrix corresponding to the task feature vector with the value matrix corresponding to the user feature vector to obtain a target value matrix containing the task type; using the target key matrix, the target value matrix and the query matrix as the conditional coding matrix.

[0091] In some exemplary embodiments, the conditional attention feature of the data to be predicted is output according to the first conditional attention distribution vector and the residual information input into the transformation network, including: residually connecting the first conditional attention distribution vector and the target residual information; normalizing the first conditional attention distribution vector after the residual connection to obtain a second conditional attention distribution vector; performing feedforward calculation on the second conditional attention distribution vector to obtain a third conditional attention distribution vector; residually connecting the second conditional attention distribution vector and the third conditional attention distribution vector; and normalizing the third conditional attention distribution vector after the residual connection to obtain the conditional attention feature.

[0092] In some exemplary embodiments, before adding the target residual information to the first conditional attention distribution vector of the data to be predicted, it also includes: if the transformation network is the first transformation network among the at least one transformation network, determining the user feature vector as the target residual information; or, if the transformation network is not the first transformation network among the at least one transformation network, determining the conditional attention feature output by the previous transformation network of the transformation network as the target residual information.

[0093] In this embodiment, when predicting the behavior results of the target user, the task type can be input into the behavior prediction model as prompt information. The behavior prediction model can perform conditional attention encoding on the data to be predicted based on the task type and the importance of user data of different dimensions learned in advance in different tasks, so as to accurately extract the features that the task type focuses on from the data to be predicted. Based on the extracted features, the behavior prediction model can accurately predict the behavior results of the data to be predicted at the behavior stage corresponding to the task type.

[0094] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0095] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0096] Figure 4 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown. Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402 .

[0097] The memory 401 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0098] In some embodiments, Figure 4The illustrated electronic device can be used to execute a behavior prediction method in a multi-stage service scenario, wherein a processor 402 is coupled to a memory 401 and is used to execute a computer program in the memory 401, so as to: respond to a behavior prediction request for a target user, and obtain data to be predicted corresponding to the behavior prediction request; the data to be predicted includes: a task type corresponding to the prediction request and multi-dimensional user data of the target user; the task type corresponds to a target behavior stage in the multi-stage service scenario; using a behavior prediction model, taking the task type as a condition, and performing feature extraction on the data to be predicted based on a conditional attention mechanism to obtain conditional attention features of the data to be predicted; based on the conditional attention features, determining a prediction label of the data to be predicted under the task type to obtain a behavior prediction result of the target user at the target behavior stage.

[0099] Optionally, when the processor 402 takes the task type as a condition and extracts features of the data to be predicted based on the conditional attention mechanism to obtain the conditional attention features of the data to be predicted, it is specifically used to: respectively vectorize the multi-dimensional user data and the task type of the data to be predicted to obtain a user feature vector and a task feature vector; respectively input the user feature vector and the task feature vector into at least one transformation network; utilize the at least one transformation network, use the first weight parameter corresponding to the task type and the second attention weight corresponding to the multi-dimensional user data, respectively perform conditional attention-based feature encoding on the task feature vector and the user feature vector to obtain at least one encoding result; wherein, in two adjacent transformation networks, the encoding result output by the previous transformation network is input into the next transformation network together with the user feature vector and the task feature vector; obtain the encoding result output by the last transformation network in the at least one transformation network as the conditional attention feature.

[0100] Optionally, when the processor 402 uses the first weight parameter corresponding to the task type and the second weight parameter corresponding to the multi-dimensional user data to perform conditional attention-based feature encoding on the task feature vector and the user feature vector, respectively, to obtain at least one encoding result, it is specifically used to: in any transformation network, use the embedding layer to embed the identification information of the transformation network into the task feature vector to obtain an updated task feature vector; use the linear transformation layer, use the first weight parameter and the second weight parameter to perform linear transformation on the updated task feature vector and the user feature vector, respectively, to obtain the transformation matrix of the updated task feature vector and the transformation matrix of the user feature vector; concatenate the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector to obtain a conditional coding matrix; use the multi-head attention layer to perform multi-head attention calculation on the conditional coding matrix to obtain the first conditional attention distribution vector of the data to be predicted; use the encoding output layer to output the conditional attention feature corresponding to the transformation network as the encoding result of the transformation network according to the first conditional attention distribution vector and the residual information input to the transformation network.

[0101] Optionally, when the processor 402 uses the first weight parameter and the second weight parameter to perform linear transformation on the updated task feature vector and the user feature vector respectively to obtain the transformation matrix of the updated task feature vector and the transformation matrix of the user feature vector, the processor 402 is specifically used to: use the first weight parameter to perform linear transformation calculation on the task feature vector to obtain a key matrix and a value matrix of the task feature vector; when the processor 402 uses the second weight parameter to perform linear transformation calculation on the user feature vector to obtain a key matrix, a query matrix and a value matrix of the user feature vector; when the transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector are spliced ​​to obtain a conditional coding matrix, the processor 402 is specifically used to: splice the key matrix corresponding to the task feature vector with the key matrix corresponding to the user feature vector to obtain a target key matrix containing the task type; splice the value matrix corresponding to the task feature vector with the value matrix corresponding to the user feature vector to obtain a target value matrix containing the task type; and use the target key matrix, the target value matrix and the query matrix as the conditional coding matrix.

[0102] Optionally, when the processor 402 outputs the conditional attention feature of the data to be predicted based on the first conditional attention distribution vector and the residual information input into the transformation network, it is specifically used to: residually connect the first conditional attention distribution vector and the target residual information; normalize the first conditional attention distribution vector after the residual connection to obtain a second conditional attention distribution vector; perform feedforward calculation on the second conditional attention distribution vector to obtain a third conditional attention distribution vector; residually connect the second conditional attention distribution vector and the third conditional attention distribution vector; normalize the third conditional attention distribution vector after the residual connection to obtain the conditional attention feature.

[0103] Optionally, before adding the target residual information to the first conditional attention distribution vector of the data to be predicted, the processor 402 is also used to: if the transformation network is the first transformation network among the at least one transformation network, determine the user feature vector as the target residual information; or, if the transformation network is not the first transformation network among the at least one transformation network, determine the conditional attention feature output by the previous transformation network of the transformation network as the target residual information.

[0104] Optionally, the multi-stage service scenario includes: a credit scenario; the task type includes: an application approval result prediction task, a spending behavior prediction task, or a repayment behavior prediction task.

[0105] In this embodiment, when predicting the behavior results of the target user, the task type can be input into the behavior prediction model as prompt information. The behavior prediction model can perform conditional attention encoding on the data to be predicted based on the task type and the importance of user data of different dimensions learned in advance in different tasks, so as to accurately extract the features that the task type focuses on from the data to be predicted. Based on the extracted features, the behavior prediction model can accurately predict the behavior results of the data to be predicted at the behavior stage corresponding to the task type.

[0106] In some embodiments, Figure 4The illustrated electronic device can be used to execute a behavior prediction model training method in a multi-stage service scenario, wherein a processor 402 is coupled to a memory 401 and is used to execute a computer program in the memory 401, so as to: obtain multiple sample data in a multi-stage service scenario, any sample data including multi-dimensional user data of a single user, a task type, and a true label of the sample data under the task type; wherein the task type is determined according to the behavior stage corresponding to the sample data in the multi-stage service scenario; the multiple sample data correspond to multiple behavior stages; using a behavior prediction model, taking the task type of each of the multiple sample data as a condition, and extracting features of the multiple sample data based on a conditional attention mechanism to obtain conditional attention features of the multiple sample data; determining the predicted labels of the multiple sample data under their respective task types based on the conditional attention features of the multiple sample data; and training the behavior prediction model with the goal of converging the gap between the true labels and the predicted labels of the multiple sample data under their respective task types to a specified range.

[0107] In this embodiment, by adding task types to sample data, explicit modeling of behavior prediction tasks at different behavior stages in multi-stage service scenarios is achieved. The behavior prediction model can learn the importance of user data of different dimensions in sample data in different tasks based on the task type, thereby reducing the learning error caused by sample bias. On the one hand, there is no need to set up a multi-task branch network for behavior prediction tasks at different behavior stages, and different prediction tasks can be achieved using a unified model structure, thereby simplifying the model structure. On the other hand, based on the learned importance of user data of different dimensions in different tasks, the behavior results of the input data at the behavior stage corresponding to the task type can be accurately predicted according to the task type and input data during the inference stage.

[0108] Further, if Figure 4 As shown, the electronic device also includes: a communication component 403, a power component 404, a display component 405, an audio component 406 and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.

[0109] Among them, the memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0110] The communication component 403 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM)), 3G (such as Wideband Code Division Multiple Access (WCDMA), 4G (such as Long Term Evolution (LTE)), 4G+ (such as upgraded Long Term Evolution (LTE-Advanced, LTE-A)) or 5G (Fifth Generation Mobile Communication Technology), or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0111] The power supply component 404 is used to provide power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power to the device where the power supply component is located.

[0112] Among them, the display component 405 includes a screen, and the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (Touch panel, TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0113] Among them, the audio component 406 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0114] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0115] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory, read-only optical disc read-only memory), optical storage, etc.) containing computer-usable program codes.

[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0120] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0121] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Parallel Random Access Machine (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A behavior prediction method in a multi-stage service scenario, characterized in that: include: In response to a behavior prediction request for a target user, obtaining to-be-predicted data corresponding to the behavior prediction request; The data to be predicted includes: the task type corresponding to the prediction request and the multi-dimensional user data of the target user; the task type corresponds to the target behavior stage in the multi-stage service scenario; Using the behavior prediction model, taking the task type as a condition, and extracting features of the data to be predicted based on a conditional attention mechanism, to obtain conditional attention features of the data to be predicted; According to the conditional attention feature, a prediction label of the data to be predicted under the task type is determined to obtain a behavior prediction result of the target user at the target behavior stage.

2. The method according to claim 1, characterized in that Taking the task type as a condition, extracting features of the data to be predicted based on a conditional attention mechanism to obtain conditional attention features of the data to be predicted, including: Vectorizing the multi-dimensional user data and task type of the data to be predicted respectively to obtain a user feature vector and a task feature vector; Inputting the user feature vector and the task feature vector into at least one transformation network respectively; Using the at least one transformation network, using a first weight parameter corresponding to the task type and a second weight parameter corresponding to the multi-dimensional user data, respectively perform conditional attention-based feature encoding on the task feature vector and the user feature vector to obtain at least one encoding result; wherein, in two adjacent transformation networks, the encoding result output by the previous transformation network is input into the next transformation network together with the user feature vector and the task feature vector; Obtain an encoding result of a last transformation network output in the at least one transformation network as the conditional attention feature.

3. The method according to claim 2, characterized in that Using the at least one transformation network, using a first weight parameter corresponding to the task type and a second weight parameter corresponding to the multi-dimensional user data, respectively perform conditional attention-based feature encoding on the task feature vector and the user feature vector to obtain at least one encoding result, including: In any transformation network, using an embedding layer to embed the identification information of the transformation network into the task feature vector to obtain an updated task feature vector; Using a linear transformation layer, using the first weight parameter and the second weight parameter, respectively, to perform a linear transformation on the updated task feature vector and the user feature vector to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector; The transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector are concatenated to obtain a conditional coding matrix; a multi-head attention layer is used to perform multi-head attention calculation on the conditional coding matrix to obtain a first conditional attention distribution vector of the data to be predicted; Utilizing the encoding output layer, according to the first conditional attention distribution vector and the residual information input into the transformation network, the conditional attention features corresponding to the transformation network are output as the encoding result of the transformation network.

4. The method according to claim 3, characterized in that The first weight parameter and the second weight parameter are used to perform linear transformation on the updated task feature vector and the user feature vector respectively, to obtain a transformation matrix of the updated task feature vector and a transformation matrix of the user feature vector, including: Using the first weight parameter to perform a linear transformation calculation on the updated task feature vector to obtain a key matrix and a value matrix of the task feature vector; Using the second weight parameter to perform linear transformation calculation on the user feature vector to obtain a key matrix, a query matrix, and a value matrix of the user feature vector; The transformation matrix of the user feature vector and the transformation matrix of the updated task feature vector are concatenated to obtain a conditional coding matrix, including: Concatenate the key matrix corresponding to the task feature vector and the key matrix corresponding to the user feature vector to obtain a target key matrix containing the task type; Concatenate the value matrix corresponding to the task feature vector and the value matrix corresponding to the user feature vector to obtain a target value matrix containing the task type; The target key matrix, the target value matrix and the query matrix are used as the conditional coding matrix.

5. The method according to claim 3, characterized in that: Outputting the conditional attention features of the data to be predicted according to the first conditional attention distribution vector and the residual information input into the transformation network includes: Perform a residual connection between the first conditional attention distribution vector and the target residual information; Normalize the first conditional attention distribution vector after the residual connection to obtain the second conditional attention distribution vector; Performing feed-forward calculation on the second conditional attention distribution vector to obtain a third conditional attention distribution vector; Performing a residual connection between the second conditional attention distribution vector and the third conditional attention distribution vector; The third conditional attention distribution vector after the residual connection is normalized to obtain the conditional attention feature.

6. The method according to claim 5, characterized in that Before adding the target residual information to the first conditional attention distribution vector of the data to be predicted, the method further includes: If the transformation network is the first transformation network in the at least one transformation network, determining the user feature vector as the target residual information; or, If the transformation network is not the first transformation network in the at least one transformation network, a conditional attention feature output by a previous transformation network of the transformation network is determined as the target residual information.

7. The method according to any one of claims 1 to 6, characterized in that: The multi-stage service scenario includes: a credit scenario; the task types include: an application approval result prediction task, a spending behavior prediction task, or a repayment behavior prediction task in a credit scenario.

8. A behavior prediction model training method in a multi-stage service scenario, characterized in that: include: Acquire multiple sample data in a multi-stage service scenario, where any sample data includes multi-dimensional user data of a single user, a task type, and a true label of the sample data under the task type; wherein the task type is determined according to the behavior stage corresponding to the sample data in the multi-stage service scenario; and the multiple sample data correspond to multiple behavior stages; Using the behavior prediction model, taking the task type of each of the multiple sample data as a condition, respectively extracting features of the multiple sample data based on the conditional attention mechanism, and obtaining conditional attention features of the multiple sample data; Determining predicted labels of the plurality of sample data under respective task types according to respective conditional attention features of the plurality of sample data; The behavior prediction model is trained with the goal of converging the gap between the real labels and the predicted labels of the multiple sample data under the respective task types to a specified range.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to: perform the steps in the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement the behavior prediction method described in any one of claims 1 to 8.

Images