Behavior prediction method and device, storage medium, and electronic device

Abstract

The present disclosure provides a behavior prediction method, a behavior prediction device, a storage medium and an electronic device, which relate to the field of computer technology. The behavior prediction method includes: obtaining the user state of a target user in an execution environment; predicting the user state in a manner in combination with a plurality of behavior types used to represent a plurality of feedback information to obtain a behavior corresponding to the user state; and executing a corresponding target behavior on the target user according to the behavior. The technical solution of the present disclosure can improve the accuracy of behavior prediction.

Description

Background Art

[0002] In order to achieve better behavioral effects, it is necessary to determine appropriate behavioral strategies and behaviors for different users.

[0003] In related technologies, user segmentation is generally achieved through machine learning models such as clustering, classification, and recommendation, or deep learning models, to assist in behavioral decision-making, or behavioral decision-making is achieved based on multiple effect goals of users.

[0004] In the above method, the applicable scenarios have certain limitations, and the dimensions considered are relatively one-sided, which makes the accuracy of the determined behavior less accurate.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a behavior prediction method and device, a computer-readable storage medium and an electronic device, thereby overcoming the problems of low flexibility and poor accuracy in related technologies at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a behavior pattern prediction method is provided, including: obtaining a user state of a target user in an execution environment; predicting the user state in a pattern by combining a plurality of behavior types used to represent a plurality of feedback information to obtain a behavior pattern corresponding to the user state; and executing a corresponding target behavior on the target user according to the behavior pattern.

[0009] In an exemplary embodiment of the present disclosure, the user state is predicted in a manner by combining multiple behavior types for representing multiple feedback information to obtain the behavior corresponding to the user state, including: predicting the user state including static features and dynamic features through an action network in a reinforcement learning decision model to obtain the behavior.

[0010] In an exemplary embodiment of the present disclosure, the multiple behavior types include positive feedback information, negative feedback information and no feedback information.

[0011] In an exemplary embodiment of the present disclosure, the method further includes: determining element information required by the reinforcement learning model; based on the element information, using historical behavior information of multiple users to train the reinforcement learning model to obtain the reinforcement learning decision model.

[0012] In an exemplary embodiment of the present disclosure, the element information required for determining the reinforcement learning model includes: obtaining all behavior nodes of the target user on the behavior link, and determining a target behavior type from multiple behavior types for each behavior node according to the business type; determining a reward and penalty function corresponding to each behavior node according to the target behavior type; and taking a reference behavior mode, a reference user state, feedback information, and a reward and penalty function as the element information.

[0013] In an exemplary embodiment of the present disclosure, the reinforcement learning model includes an action network and an evaluation network; based on the element information, the reinforcement learning model is trained using historical behavior information of multiple users to obtain the reinforcement learning decision model, including: adjusting the network parameters of the action network according to the predicted score of the reference behavior mode corresponding to the historical behavior information to obtain a trained action network; adjusting the network parameters of the evaluation network according to the reward-penalty function and other scores to obtain a trained evaluation network; and determining the trained action network and the trained evaluation network as the reinforcement learning decision model.

[0014] In an exemplary embodiment of the present disclosure, the method further includes: predicting the historical user states of multiple users through the action network to determine a reference behavior pattern; the historical user states include historical static features and historical dynamic features; and merging the historical user states and the reference behavior pattern through the evaluation network to score the reference behavior pattern to obtain the predicted score.

[0015] According to one aspect of the present disclosure, a behavior pattern prediction device is provided, comprising: a state acquisition module, used to obtain the user state of a target user in an execution environment; a pattern determination module, used to predict the pattern of the user state in combination with a plurality of behavior types used to represent a plurality of feedback information, and obtain a behavior pattern corresponding to the user state; and a behavior execution module, used to execute a corresponding target behavior on the target user according to the behavior pattern.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the behavior prediction method described in any one of the above items is implemented.

[0017] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned behavior prediction methods by executing the executable instructions.

[0018] The behavior prediction method, behavior prediction device, computer-readable storage medium, and electronic device provided in the embodiments of the present disclosure can combine multiple behavior types representing multiple feedback information to predict the user state of the target user in the execution environment, thereby obtaining the behavior corresponding to the user state. Since multiple behavior types are used to represent multiple feedback information, the limitations caused by predicting the behavior based on only one feedback information are avoided, and the user state can be comprehensively predicted from multiple aspects and dimensions, which improves the accuracy and completeness of the behavior, and also improves the reliability and stability.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 The system architecture diagram for implementing the behavior prediction method according to an embodiment of the present disclosure is schematically shown.

[0022] Figure 2 A schematic diagram schematically illustrates a behavior prediction method in an embodiment of the present disclosure.

[0023] Figure 3 The specific process diagram of model training in the embodiment of the present disclosure is schematically shown.

[0024] Figure 4 A system diagram of a reinforcement learning decision model in an embodiment of the present disclosure is schematically shown.

[0025] Figure 5 A schematic diagram of a process for determining a reinforcement learning decision model in an embodiment of the present disclosure is schematically shown.

[0026] Figure 6 A schematic diagram schematically illustrates the overall process of determining a behavior mode in an embodiment of the present disclosure.

[0027] Figure 7 A block diagram of a behavior prediction device in an embodiment of the present disclosure is schematically shown.

[0028] Figure 8 A block diagram of an electronic device in an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] In the related art, clustering models, classification models, recommendation models or deep reinforcement learning methods can be used to determine marketing strategies. Clustering models are usually used when there is no specific user category. User characteristics are used to measure user similarity, and users with high similarity are clustered into a category of users. The characteristics of each category of users formed by clustering are studied, and marketing strategies are artificially designed based on their characteristics. Classification models are usually used when there are specific user categories. Users know some of their user categories and hope to classify users outside of them. They mainly use known types of user training models to predict the types of unknown types of users, and design marketing strategies based on the possibility that users are users of a certain category. Recommendation models are usually aimed at scenarios with a relatively rich number of users and products. User-based product recommendations and product-based product recommendations can be made based on the similarity between users or products. The purpose is to increase user browsing, clicking and purchasing behaviors under limited product exposure. Or the recommendation marketing strategy is studied using deep reinforcement learning methods, and the marketing strategy is updated through continuous interaction between users and the marketing environment to maximize the preset marketing effect.

[0032] Among them, clustering models and classification models solve the problem of reaching users, that is, what kind of users to reach, but do not solve the problem of how to reach users. The recommendation model solves the problem of recommending product categories when the product dimensions are rich, but is not suitable for products with less rich product dimensions. The applicable application scenarios are relatively fixed and belong to user-active marketing. It is not suitable for scenarios that require active contact with users through push or SMS. There are two important problems in using reinforcement learning to update marketing strategies: 1. It assumes that users generate multiple execution results in response to marketing behaviors, and the execution results correspond to multiple effect targets on the marketing effect chain. The effect target is a positive target, that is, the execution result on the marketing chain is positive, which belongs to positive feedback. This assumption does not take into account the situation where the marketing strategy interferes with the user, that is, the situation where negative feedback is generated, such as users canceling their accounts, unfollowing, redeeming products, etc. 2. The training method of the reinforcement learning model is not explained: Generally, the training of reinforcement learning requires a large amount of data for interaction with the environment, so the offline model can be trained using historical data, and then the marketing strategy is improved according to the trained model based on the user's status and the trained model is further updated as needed.

[0033] In order to solve the above technical problems, a behavior prediction method is provided in an embodiment of the present disclosure. Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present disclosure can be applied.

[0034] like Figure 1 As shown, the system architecture 100 may include a first end 101, a network 102, and a second end 103. The first end 101 may be a client, such as a terminal that can use an application such as a smart phone, a computer, or a tablet computer. The network 102 is used to provide a medium for a communication link between the first end 101 and the second end 103. The network 102 may include various connection types, such as a wired communication link, a wireless communication link, and the like. In the embodiment of the present disclosure, the network 102 between the first end 101 and the second end 103 may be a wired communication link, such as a communication link provided by a serial port connection line, or a wireless communication link, and a communication link provided by a wireless network. The second end 103 may be a server or a client with a data processing function, such as a portable computer, a desktop computer, a smart phone, or other terminal equipment with a data processing function, which is used to process the user status obtained by the first end.

[0035] In the embodiment of the present disclosure, the first end can respond to the user operation and determine the user state of the target user in the execution environment. The second end inputs the user state into the reinforcement learning decision model to perform mode prediction based on the user state of the target user in the execution environment obtained by the first end, combined with multiple behavior types used to represent multiple feedback information, so as to obtain the behavior mode corresponding to the user state, and can use the behavior mode to perform the corresponding target behavior on the target user.

[0036] It should be understood that Figure 1 The numbers of first ends, networks, and second ends in FIG. 1 are merely illustrative.

[0037] It should be noted that the behavior pattern prediction method provided in the embodiment of the present disclosure may be completely executed by the second end, and accordingly, the behavior pattern prediction device may be disposed in the second end 103 .

[0038] Based on the above system architecture, a behavior prediction method is provided in the embodiment of the present disclosure. Figure 2 As shown in , the behavior prediction method includes steps S210 to S230, which are described in detail as follows:

[0039] In step S210, the user status of the target user in the execution environment is obtained.

[0040] In the embodiments of the present disclosure, the execution environment refers to the environment in which the target user is located, which is specifically associated with the business type. For example, when the business type is a marketing recommendation business, its execution environment is a marketing recommendation environment. The target user may be any user who uses a preset type application in the execution environment. The preset type application may, for example, be a variety of applications that can implement business processing functions, such as transaction applications or community applications, which may specifically include but are not limited to shopping applications, video applications, or reading applications, etc. The user state refers to the user state of the target user at the current moment, which is used to represent the characteristics of each target user. For different target users, the corresponding user states may be the same or different, which is not limited here.

[0041] For the user status of each user, it may include static features and dynamic features. Static features refer to fixed features, and dynamic features refer to features determined based on each behavior. Static features can be determined based on the target user's account information, and may include but are not limited to gender, age, home address, etc. Dynamic features mainly include but are not limited to user information that will change due to the execution of the behavior, and may be dynamically determined based on the target user's behavior log. For example, a dynamic feature may be the duration of a preset operation of the target user on a page associated with a certain behavior, and the preset operation may be a click operation, a browsing operation, etc. It should be noted that the behavior can be used to represent a business method, such as a recommendation method, a marketing strategy, or a marketing method, etc.

[0042] refer to Figure 2 As shown in , in step S220, the user state is predicted in a manner by combining multiple behavior types used to represent multiple feedback information to obtain a behavior manner corresponding to the user state.

[0043] In the embodiments of the present disclosure, in order to avoid the problem of determining the behavior strategy only based on positive feedback information in the related art, the user status can be predicted in a combined manner using multiple behavior types representing multiple feedback information. Specifically, the multiple behavior types representing multiple feedback information may include positive feedback information, negative feedback information, and no feedback information. Among them, positive feedback information refers to multiple effect targets on a behavior chain. When the target user sequentially performs the operations on the behavior chain, it can be considered that multiple effect targets have been obtained, such as when the user purchases an item. Negative feedback information refers to the negative impact of the behavior on the user. When the target user does not promote the key behavior of the behavior chain, it can be considered to be negative feedback information, such as when the user abandons an item. No feedback information means that the user has not performed any operation on an item.

[0044] Before determining a variety of feedback information, all behavior nodes associated with the target user on the behavior link can be determined first. The content contained in the behavior node may vary according to different behavior links, and can be specifically determined according to the business type. For example, when the business type is a transaction business, the behavior nodes on the behavior link corresponding to the transaction business may include but are not limited to browsing, clicking, registering, and trading. The key behavior may be the behavior node with the largest association with the business type, such as a transaction behavior. Further, the corresponding behavior type can be determined for each behavior node based on the business characteristics. The business characteristics here can be used to measure whether each behavior node on the behavior link is completed. The behavior type is used to represent feedback information, and multiple behavior nodes can have multiple behavior types, and the size of each feedback information is different. For example, on a user's behavior link, browsing is positive feedback information, clicking is positive feedback information, registering is no feedback information, and trading is negative feedback information.

[0045] When combining multiple behavior types to predict the user state to obtain the behavior corresponding to the user state, a reinforcement learning decision model can be used to perform. In order to ensure the accuracy of the prediction results, a reinforcement learning model can be trained to use the trained reinforcement learning model as a reinforcement learning decision model for application.

[0046] Figure 3 The flowchart for model training is shown schematically in Figure 3 As shown in , it mainly includes the following steps:

[0047] In step S310 , element information required by the reinforcement learning model is determined.

[0048] In the disclosed embodiment, the element information required by the reinforcement learning model refers to the information necessary for constructing the reinforcement learning model. The element information may include, but is not limited to, reference behavior, reference user status, feedback information and its corresponding reward and penalty function. Among them, the reference behavior can be determined according to the specific business type, for example, the amount of coupons issued, the size of discounts, and the gift items, etc. The reward and penalty function is used to describe the user's feedback information, and its predicted score can be determined according to the reward and penalty function. In the disclosed embodiment, all feedback information can be used as variables to construct a reward and penalty function. For example, the weight of each behavior node can be determined according to its importance, and then the reward and penalty function can be constructed according to the weight. Among them, it can be considered that the behavior nodes on the behavior link increase in importance in the order of arrangement, and the behavior node arranged in the last place has the highest importance. In addition, the reward and penalty function can also be set arbitrarily according to other methods, which is not limited here.

[0049] In the disclosed embodiment, negative feedback information corresponds one-to-one to positive feedback information, such as cancellation corresponds to registration, and redemption corresponds to purchase. When the reward-penalty function of the positive feedback information is f(x), the penalty-reward function of the negative feedback information can be set to -f(x).

[0050] In step S320, based on the element information, the reinforcement learning model is trained using historical behavior information of multiple users to obtain a reinforcement learning decision model.

[0051] In the disclosed embodiment, after obtaining the element information, the reinforcement learning model can be trained based on the reinforcement learning model constructed by the element information and according to the historical behavior information of multiple users, so as to obtain a trained reinforcement learning model as a reinforcement learning decision model.

[0052] refer to Figure 4 As shown in , the reinforcement learning model includes two parts: the execution environment and the agent, which is used to describe and solve the problem of how the agent can maximize the reward or achieve a specific goal by learning strategies during the interaction with the execution environment. The agent continuously learns and optimizes its behavior through interaction and feedback with the execution environment. The agent can be an entity that makes decisions. Specifically, the agent obtains the user state corresponding to the execution environment, processes the user state based on the action network, selects an action (behavior mode) according to the user state, and the execution environment changes its state after accepting the action. It also generates a reward or punishment feedback (reward and punishment function) to the agent based on the behavior mode and user state, and then obtains the difference Q between the predicted score and the actual score. The behavior nodes on the behavior chain include clicks, registration / logout, transactions, etc., among which transactions are the ultimate goal of the behavior chain. The reward or punishment feedback is determined according to the behavior type of these behavior nodes. In addition, the agent judges whether the previous behavior mode is correct and whether the strategy needs to be adjusted based on the feedback obtained. Through continuous adjustment, the accumulated feedback obtained is maximized. That is, the reward-penalty function of the reinforcement learning model can be determined according to the behavior node, and the reward-penalty function can be returned to the agent so that the agent updates its behavior according to the reward-penalty function.

[0053] exist Figure 4 In the example, the state vector constructed based on the reference user state of the execution environment is input to the agent, and the agent determines the reference behavior to be taken for the reference user state based on some mapping methods. The mapping method is used to map the reference user state to the next reference behavior. The mapping method can be implemented by deep learning algorithms, etc. Moreover, the historical static features and historical dynamic features at each moment can generate a reference behavior based on the mapping method, and then generate a reward and penalty function for each reference behavior.

[0054] The historical behavior information may be historical marketing information, and specifically may include historical user status. The historical user status may be of the same type as the user status, and may include historical static features and historical dynamic features. The historical static features may be determined based on the account information of multiple users. The historical dynamic features may be dynamically determined based on the historical behavior logs of multiple users, including but not limited to user information that has changed due to changes in the execution behavior. The specific method of the historical user status may be the same as the above-mentioned user status, and will not be repeated here.

[0055] Figure 5 The schematic diagram of the process of determining the reinforcement learning decision model is shown in Figure 5 As shown in , it mainly includes the following steps:

[0056] In step S510, the network parameters of the action network are adjusted according to the predicted scores of the reference action modes corresponding to the historical action information to obtain a trained action network.

[0057] In this step, since the reinforcement learning model consists of two parts, the action network and the evaluation network, when training the model, in order to improve the accuracy, both the action network and the evaluation network need to be trained at the same time.

[0058] The prediction score can be obtained through the following steps: Step 1, predict the historical user status of multiple users through the action network to determine the reference behavior mode; the historical user status includes historical static features and historical dynamic features; Step 2, merge the historical user status and the reference behavior mode through the evaluation network to score the reference behavior mode and obtain the prediction score. That is to say, first, after obtaining the historical user status including historical static features and historical dynamic features, the action network will predict the historical user status and determine a reference behavior mode. Furthermore, the evaluation network can merge the reference behavior mode and the historical user status, and score the reference behavior mode made by the action network according to the scoring strategy determined by the network parameters to obtain the prediction score of the reference behavior mode. Through these two networks, the prediction score for each reference behavior mode can be accurately determined.

[0059] For example, the action network obtains the historical user status of multiple users and makes a reference behavior, such as issuing a 3% discount coupon to the reference user in a marketing scenario. The evaluation network gives a predicted score to the performance of the action network based on the historical user status and reference behavior of multiple users, so as to judge whether the conditions are met based on the predicted score.

[0060] After obtaining the predicted score, the action network adjusts the network parameters of the action network according to the predicted score of the evaluation network until the value of the loss function in the action network is minimized and the value of the loss function tends to be stable, and the network parameters when the value of the loss function is stable are determined as the network parameters of the action network to obtain a trained action network. By training the action network, the performance of the network can be made more stable, so as to strive to make a reference behavior mode that is more suitable for the historical user status next time, and improve the matching and accuracy of the reference behavior mode.

[0061] In step S520, the network parameters of the evaluation network are adjusted according to the reward-penalty function and other scores to obtain a trained evaluation network.

[0062] In this step, the evaluation network adjusts the network parameters of the evaluation network itself according to the reward-penalty function given by the system and other scores used to represent the score labels until the value of the loss function in the evaluation network is minimized and the value of the loss function tends to be stable, and the network parameters when the value of the loss function is stable are determined as the network parameters of the evaluation network to obtain a trained evaluation network. By training the evaluation network, the performance of the network can be made more stable, and the accuracy and reliability of the score can be improved.

[0063] In step S530, the trained action network and the trained evaluation network are determined as the reinforcement learning decision model.

[0064] In this step, after the action network and the evaluation network are trained, a reinforcement learning decision model can be formed based on the trained action network and the trained evaluation network, so as to perform prediction processing through the reinforcement learning decision model.

[0065] In the disclosed embodiment, the reinforcement learning model includes an action network and an evaluation network. At the beginning, the action network randomly determines the behavior mode, and the evaluation network randomly scores. However, due to the existence of the reward and penalty function, the scoring of the evaluation network is getting more and more accurate, and the performance of the action network is getting better and better. Therefore, by training the reinforcement learning model through historical user states, a model with more stable performance can be obtained, thereby improving reliability. In addition, since the reward and penalty function in the disclosed embodiment takes into account a variety of behavior types corresponding to a variety of feedback information, that is, including positive feedback information, negative feedback information, and no feedback information, etc., it is possible to comprehensively and comprehensively train the reinforcement learning decision model from multiple dimensions to make the reinforcement learning decision model more accurate and stable, thereby improving reliability.

[0066] After obtaining the trained reinforcement learning decision model, since the action network is used to determine the behavior mode and the evaluation network is used for scoring, in the application stage, only the action network in the reinforcement learning decision model needs to be used to predict the user state containing static features and dynamic features to obtain the behavior mode. That is, the user state is input into the action network for feature extraction, and prediction is made based on its static features and dynamic features to obtain the behavior mode that matches the user state. For example, if the user state is that the browsing time of the page associated with the coupon is 1 minute, the behavior mode can be determined as issuing coupons. If the behavior mode is displayed on the target user side, it can be considered to have reached the user. If the user clicks on the page associated with the coupon and then conducts a transaction, it can be considered that all operations on the behavior link are completed and the target behavior is achieved.

[0067] In step S230, a corresponding target behavior is performed on the target user according to the behavior mode.

[0068] In the embodiment of the present disclosure, after determining the behavior mode corresponding to the user status of the target user, the behavior mode can be configured to the target user so as to perform the corresponding target behavior on the target user according to the behavior mode. The target behavior here can be a marketing behavior or a recommendation behavior, etc., which can be determined according to the business type.

[0069] It should be noted that, for each target user, the steps for determining the behavior mode are: obtaining the user state of the target user in the execution environment, predicting the user state in combination with multiple behavior types used to represent multiple feedback information, and obtaining the behavior mode corresponding to the user state, so it will not be repeated here.

[0070] Figure 6 The overall flow chart of determining the behavior mode in the embodiment of the present disclosure is schematically shown, with reference to Figure 6 As shown in , it mainly includes steps S610 to S690. Among them, steps S610 to S660 are the model training process, and steps S670 to S690 are the model application process:

[0071] In step S610, the model environment and behavior are defined.

[0072] In step S620, the user status of the target user is extracted.

[0073] In step S630, feedback information in the user behavior link is defined.

[0074] In step S640, a reward-penalty function including multiple feedback information is determined, where the multiple feedback information includes positive feedback information, negative feedback information, and no feedback information.

[0075] In step S650, historical behavior information is obtained.

[0076] In step S660, the reinforcement learning decision model is trained.

[0077] In step S670, the user status of the target user is extracted.

[0078] In step S680, a reinforcement learning decision model is obtained.

[0079] In step S690, a behavior mode is obtained, and the behavior mode may be a marketing strategy.

[0080] Figure 6 The technical solution in , when determining the behavior of the target user in a certain business scenario, takes into account the various behavior types corresponding to each behavior node on the behavior chain, including the user's positive feedback information, negative feedback information, and no feedback information, etc., so it can comprehensively train the reinforcement learning decision model from multiple dimensions. And when applying the model, the user state can be processed based on the reinforcement learning decision model determined from multiple dimensions to obtain its corresponding behavior, which solves the problem of how to reach the user and avoids the limitations caused by prediction based on only one feedback information. Therefore, the behavior obtained is more accurate, more comprehensive, more in line with the actual scenario, and can improve business conversion efficiency.

[0081] In the embodiment of the present disclosure, a behavior prediction device is also provided, referring to Figure 7 As shown in , the behavior prediction device 700 mainly includes the following modules:

[0082] A status acquisition module 701 is used to acquire the user status of a target user in the execution environment;

[0083] A mode determination module 702, configured to predict the user state in a mode by combining multiple behavior types used to represent multiple feedback information, and obtain a behavior mode corresponding to the user state;

[0084] The behavior execution module 703 is used to execute the corresponding target behavior on the target user according to the behavior mode.

[0085] In an exemplary embodiment of the present disclosure, the user state is predicted in a manner by combining multiple behavior types for representing multiple feedback information to obtain the behavior corresponding to the user state, including: predicting the user state including static features and dynamic features through an action network in a reinforcement learning decision model to obtain the behavior.

[0086] In an exemplary embodiment of the present disclosure, the multiple behavior types include positive feedback information, negative feedback information and no feedback information.

[0087] In an exemplary embodiment of the present disclosure, the method further includes: determining element information required by the reinforcement learning model; based on the element information, using historical behavior information of multiple users to train the reinforcement learning model to obtain the reinforcement learning decision model.

[0088] In an exemplary embodiment of the present disclosure, the element information required for determining the reinforcement learning model includes: obtaining all behavior nodes of the target user on the behavior link, and determining a target behavior type from multiple behavior types for each behavior node according to the business type; determining a reward and penalty function corresponding to each behavior node according to the target behavior type; and taking a reference behavior mode, a reference user state, feedback information, and a reward and penalty function as the element information.

[0089] In an exemplary embodiment of the present disclosure, the reinforcement learning model includes an action network and an evaluation network; based on the element information, the reinforcement learning model is trained using historical behavior information of multiple users to obtain the reinforcement learning decision model, including: adjusting the network parameters of the action network according to the predicted score of the reference behavior mode corresponding to the historical behavior information to obtain a trained action network; adjusting the network parameters of the evaluation network according to the reward-penalty function and other scores to obtain a trained evaluation network; and determining the trained action network and the trained evaluation network as the reinforcement learning decision model.

[0090] In an exemplary embodiment of the present disclosure, the method further includes: predicting the historical user states of multiple users through the action network to determine a reference behavior pattern; the historical user states include historical static features and historical dynamic features; and merging the historical user states and the reference behavior pattern through the evaluation network to score the reference behavior pattern to obtain the predicted score.

[0091] In addition, the specific details of each part of the above-mentioned behavior prediction device have been described in detail in the implementation method of the behavior prediction method part. The undisclosed details can be found in the implementation method content of the method part, and thus will not be repeated here.

[0092] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0093] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0094] In an embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0095] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0096] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0098] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 Follow the steps shown in .

[0099] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0100] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0101] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, a graphics acceleration interface, a processing unit, or a local bus using any of a variety of bus architectures.

[0102] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0103] In an embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0104] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0105] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0106] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0108] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0109] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0110] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A behavior prediction method, characterized in that: include: Get the user status of the target user in the execution environment; The user state is predicted by a reinforcement learning decision model in combination with multiple behavior types used to represent multiple feedback information to obtain a behavior mode corresponding to the user state. The multiple feedback information includes positive feedback information, negative feedback information and no feedback information. The positive feedback information refers to multiple effect targets on the behavior chain, the negative feedback information refers to the negative impact of the behavior on the user, and the no feedback information refers to the user not performing any operation on a certain item. executing a corresponding target behavior on the target user according to the behavior mode; Wherein, obtaining the reinforcement learning decision model includes: Acquire all behavior nodes of the target user on the behavior link, and determine a target behavior type from multiple behavior types for each behavior node according to the business type; Determine a reward-penalty function corresponding to each behavior node according to the target behavior type; The reference behavior, historical user status, feedback information, and reward-penalty function are used as element information; Based on the element information, the reinforcement learning decision model is trained using historical behavior information of multiple users to obtain the reinforcement learning decision model, wherein the historical behavior information is historical marketing information, including historical user status; for each user's user status, it includes static features and dynamic features, the static features are determined according to the target user's account information, including gender, age, and home address; the dynamic features are dynamically determined according to the target user's behavior log, including the duration of the target user's preset operation on a page associated with a certain behavior mode, the preset operation being a click operation and a browse operation; Wherein, obtaining the reinforcement learning decision model includes: adjusting the network parameters of the action network according to the predicted score of the reference behavior mode corresponding to the historical behavior information to obtain a trained action network; adjusting the network parameters of the evaluation network according to the reward and penalty function and other scores to obtain a trained evaluation network; determining the trained action network and the trained evaluation network as the reinforcement learning decision model, wherein the reinforcement learning decision model includes the action network and the evaluation network; wherein the reward and penalty function is constructed with feedback information as a variable; Determining the predicted score includes: predicting the historical user states of multiple users through the action network to determine a reference behavior pattern; merging the historical user states and the reference behavior pattern through the evaluation network to score the reference behavior pattern to obtain the predicted score.

2. The behavior prediction method according to claim 1, characterized in that: The method of predicting the user state by combining multiple behavior types for representing multiple feedback information through a reinforcement learning decision model to obtain a behavior mode corresponding to the user state includes: The user state including static features and dynamic features is predicted and processed through the action network in the reinforcement learning decision model to obtain the behavior mode.

3. A behavior prediction device, characterized in that: include: A status acquisition module, used to acquire the user status of the target user in the execution environment; A mode determination module is used to predict the user state through a reinforcement learning decision model in combination with multiple behavior types used to represent multiple feedback information, and obtain a behavior mode corresponding to the user state. The multiple feedback information includes positive feedback information, negative feedback information, and no feedback information. The positive feedback information refers to multiple effect targets on the behavior chain, the negative feedback information refers to the negative impact of the behavior on the user, and the no feedback information refers to the user not performing any operation on a certain item; A behavior execution module, used for executing a corresponding target behavior on the target user according to the behavior mode; Wherein, obtaining the reinforcement learning decision model includes: Acquire all behavior nodes of the target user on the behavior link, and determine a target behavior type from multiple behavior types for each behavior node according to the business type; Determine a reward-penalty function corresponding to each behavior node according to the target behavior type; The reference behavior, historical user status, feedback information, and reward-penalty function are used as element information; Based on the element information, the reinforcement learning decision model is trained using historical behavior information of multiple users to obtain the reinforcement learning decision model, wherein the historical behavior information is historical marketing information, including historical user status, and for each user's user status, the historical behavior information includes static features and dynamic features, wherein the static features are determined according to the target user's account information, including gender, age, and home address; the dynamic features are dynamically determined according to the target user's behavior log, including the duration of the target user's preset operation on a page associated with a certain behavior mode, wherein the preset operation is a click operation and a browse operation; Wherein, obtaining the reinforcement learning decision model includes: adjusting the network parameters of the action network according to the predicted score of the reference behavior mode corresponding to the historical behavior information to obtain a trained action network; adjusting the network parameters of the evaluation network according to the reward and penalty function and other scores to obtain a trained evaluation network; determining the trained action network and the trained evaluation network as the reinforcement learning decision model, wherein the reinforcement learning decision model includes the action network and the evaluation network; wherein the reward and penalty function is constructed with feedback information as a variable; Determining the predicted score includes: predicting the historical user states of multiple users through the action network to determine a reference behavior pattern; merging the historical user states and the reference behavior pattern through the evaluation network to score the reference behavior pattern to obtain the predicted score.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the behavior prediction method described in any one of claims 1 or 2 is implemented.

5. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the behavior prediction method described in any one of claims 1 or 2 by executing the executable instructions.

Images