Method, device, storage medium and electronic device for object management

Abstract

The embodiments of this specification disclose a method, an apparatus, a storage medium and an electronic device for object management, which construct a recommendation model based on object characteristics of multiple objects and a set of alternative management and control actions corresponding to multiple object groups, and obtain a recommended management and control action corresponding to at least one object predicted by the recommendation model; obtain feedback information about the recommended management and control action from the at least one object, so that the recommendation model obtains unbiased estimated values ​​of preset indicators for the at least one object under each management and control action by performing reinforcement learning based on the feedback information; and optimizes the management and control actions for the multiple objects based on business objectives and the unbiased estimated values, and obtains target management and control actions corresponding to the multiple objects under the business objectives.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium and electronic device for object management. Background Art

[0002] In daily life, there are a large number of various types of objects (for example, merchants). Among such a large group of objects, there must be risky objects, which will seriously affect the ecological environment of the object operation platform. We need to guide and educate these objects around their risk performance, or impose penalties and controls of varying degrees of strictness. If the control method is too strict, it will affect the normal business activities of the object, and the object needs to lift the control through appeal, which will have a poor experience; but if the control is not in place, it will lead to further expansion of the object's risk and cause financial losses to the platform. Therefore, how to find a better control solution is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for object management.

[0004] The embodiment of this specification provides a method for object control, and designs an object management decision system based on online reinforcement learning and real-time operations optimization method. The feedback of the object on different control actions is obtained in real time by online reinforcement learning, so as to achieve unbiased estimation of preset indicators for different objects under different control actions; at the same time, after obtaining reliable indicator estimation, the best control plan for optimizing business objectives is sought through real-time operations optimization and mathematical multi-objective constraints, which can solve the problem of control exposure bias. Through online real-time feedback accumulation, the unbiased estimation of preset indicators is continuously approached in a data-driven manner, and unbiased control action feedback is directly learned. Furthermore, a dynamic tuning mechanism is also set up. Through real-time optimization of business objectives, real-time automatic adjustment can be made according to the calculated optimal control plan, which can adapt to different business needs and reduce manpower and resource costs, and achieve the best overall control efficiency and effect. The method includes:

[0005] A recommendation model is constructed according to the object features of multiple objects and the candidate control action sets corresponding to multiple object groups, and a recommended control action corresponding to at least one object predicted by the recommendation model is obtained, wherein each object group includes one or more objects, and the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the candidate control action set corresponding to the object group to which the object belongs;

[0006] Obtaining feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information;

[0007] According to the business objectives and the unbiased estimated values, the management and control actions are optimized for the multiple objects to obtain the target management and control actions corresponding to the multiple objects under the business objectives.

[0008] Furthermore, the method further comprises:

[0009] The multiple objects are grouped according to their object features to obtain multiple object groups, wherein each object group includes one or more objects.

[0010] Further, the method comprises:

[0011] For each object group in the multiple object groups, a candidate control action set corresponding to the object group is obtained, wherein the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group.

[0012] Furthermore, the method further comprises:

[0013] Adjust the learning rate of the recommendation model for the feedback information according to the feedback duration corresponding to the feedback information.

[0014] Furthermore, obtaining an unbiased estimated value of a preset indicator of the at least one object under each control action includes:

[0015] In the reinforcement learning process of the recommendation model, a current real-time value of a preset indicator of the at least one object under each control action is obtained by performing real-time statistics on multiple feedback information of the at least one object on each control action;

[0016] The current real-time value is used as an unbiased estimated value of the preset indicator for the at least one object under each control action.

[0017] Furthermore, the using the current real-time value as an unbiased estimated value of the preset indicator for the at least one object under each control action includes:

[0018] If the current real-time value satisfies a preset unbiased condition, the current real-time value is used as an unbiased estimated value of the preset indicator for the at least one object under each control action.

[0019] Furthermore, the preset indicators include risk probability indicators and complaint probability indicators.

[0020] Furthermore, performing optimal solution of control actions for the multiple objects according to the business objectives and the unbiased estimated values ​​to obtain target control actions corresponding to the multiple objects under the business objectives includes:

[0021] Determine an objective function and one or more constraints corresponding to the objective function according to the business objective, wherein the input parameters of the objective function and the one or more constraints include the control action corresponding to each object, and the input parameters of the objective function and at least one constraint also include an unbiased estimated value of the object under the control action with respect to the preset indicator;

[0022] According to the objective function and the one or more constraints, the control actions are optimized for the multiple objects to obtain the target control actions corresponding to the multiple objects under the business objectives.

[0023] Furthermore, the control action corresponding to each object is one or more recommended control actions corresponding to the object that were predicted by the recommendation model.

[0024] Furthermore, the method further comprises:

[0025] If an unbiased estimated value of the preset indicator for the object under at least one recommended control action corresponding to the object is not currently obtained, the initial value of the preset indicator for the object under the at least one recommended control action is used to replace the corresponding unbiased estimated value as an input parameter of the objective function and the at least one constraint condition.

[0026] Furthermore, the control action corresponding to each object is a recommended control action for which a corresponding unbiased estimated value has been obtained among one or more recommended control actions corresponding to the object that were predicted by the recommendation model.

[0027] The embodiment of this specification also provides a device for object management and control, including:

[0028] A control action recommendation module is used to build a recommendation model based on the object features of multiple objects and the candidate control action sets corresponding to multiple object groups, and obtain a recommended control action corresponding to at least one object predicted by the recommendation model, wherein each object group includes one or more objects, and the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the candidate control action set corresponding to the object group to which the object belongs;

[0029] A reinforcement learning module, configured to obtain feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information;

[0030] The optimization solution module is used to perform optimization solutions for the management and control actions for the multiple objects according to the business objectives and the unbiased estimated values, and obtain the target management and control actions corresponding to the multiple objects under the business objectives.

[0031] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0032] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0033] In the embodiments of the present specification, a method for object control is proposed, and an object management decision system based on online reinforcement learning and real-time operations optimization methods is designed. The feedback of the object on different control actions is obtained in real time through online reinforcement learning, so as to achieve unbiased estimation of preset indicators for different objects under different control actions; at the same time, after obtaining reliable indicator estimates, through real-time operations optimization, the best control plan for optimizing business objectives is sought by using mathematical multi-objective constraints, which can solve the problem of control exposure bias, and through the accumulation of real-time feedback online, the unbiased estimate of the preset indicators is continuously approached in a data-driven manner, and the unbiased control action feedback is directly learned. Furthermore, a dynamic tuning mechanism is also set up, which can perform real-time automatic adjustments according to the calculated optimal control plan through real-time optimization of business objectives, adapt to different business needs and reduce manpower and resource costs, and achieve the optimization of overall control efficiency and effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for object management provided in an embodiment of this specification;

[0035] Figure 2 A flowchart of an example method for object management provided in an embodiment of this specification;

[0036] Figure 3 A schematic diagram of the structure of a device for object management and control provided in an embodiment of this specification;

[0037] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0039] See also Figure 1 , is a flowchart of a method for object control provided in an embodiment of this specification. In an embodiment of this specification, the method for object control is applied to a device for object control (hereinafter referred to as "object control device") or an electronic device equipped with an object control device. Figure 1 The process shown is described in detail, and the method for object management may specifically include the following steps:

[0040] S102, constructing a recommendation model based on object characteristics of multiple objects and a set of alternative control actions corresponding to multiple object groups, and obtaining a recommended control action corresponding to at least one object predicted by the recommendation model, wherein each object group includes one or more objects, and the set of alternative control actions corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the set of alternative control actions corresponding to the object group to which the object belongs.

[0041] In some embodiments, the object includes but is not limited to any object that needs to be controlled when there is a risk, such as a merchant, a company, an institution, etc. This specification does not limit this. It should be noted that the above objects are only examples, not limitations. Those skilled in the art should understand that any type of object can be included in the protection scope of this specification. This specification takes the merchant as an example to introduce the object. Those skilled in the art should understand that this specification can also be applied to any other type of object, and will not be repeated here.

[0042] In some embodiments, object characteristics (e.g., merchant characteristics) include, but are not limited to, the object's size, operating conditions, historical complaints, historical management and control conditions, qualifications, historical transaction records, historical litigation case participation, historical risk attributes, related field conditions, etc. This example embodiment does not specifically limit this.

[0043] In some embodiments, multiple objects are divided into multiple object groups (for example, merchant groups) according to their respective object characteristics in advance, each object group includes one or more objects, and each object belongs to one of the multiple object groups. In some embodiments, for each object group, a set of candidate control actions including multiple control actions is assigned to the object group in advance, and the same control actions may exist between the sets of candidate control actions corresponding to different object groups, which is not specifically limited in this example embodiment. In some embodiments, the control action includes but is not limited to at least one of the control type (for example, passing (not controlling), limiting collection, limiting collection amount, delayed arrival, etc.), control duration (for example, one day, one week, etc.), and control amount (for example, 100 yuan, 1000 yuan, etc.). For example, the control action is passing (not controlling), limiting collection for one day, limiting collection amount of 1000 yuan, delayed arrival of collection for one week, etc. It should be noted that the above control actions are only examples, not limitations. Those skilled in the art should understand that any form and any content of control actions can be included in the protection scope of this specification.

[0044] In some embodiments, a recommendation model is constructed based on the object features of each object in a plurality of objects and the set of alternative control actions corresponding to each object group in a plurality of object groups. For example, a recommendation model is constructed using the LinUCB recommendation algorithm in combination with the merchant features of each merchant and the set of alternative control actions of each merchant group. The LinUCB algorithm is a recommendation algorithm that enables the algorithm to converge faster by adding feature information of the recommended object, process a dynamic recommendation candidate pool, and independently update parameters. It should be noted that the above recommendation algorithm is only an example, not a limitation. Those skilled in the art should understand that any structure of the recommendation model and any model construction method can be included in the protection scope of this specification. In some embodiments, the input of the recommendation model includes the object features of an object and the set of alternative control actions corresponding to the object group to which the object belongs. The output of the recommendation model includes the control action recommended for the object (i.e., the recommended control action), or the output also includes the predicted value of the preset indicator of the object under the recommended control action, wherein the recommended control action is one of the multiple control actions included in the set of alternative control actions.

[0045] In some embodiments, when there is a risk to an object, that is, when a control action needs to be taken on the object, by inputting the object characteristics of the object and the set of alternative control actions corresponding to the object group to which the object belongs into the recommendation model, the control action recommended for the object (i.e., the recommended control action) output by the recommendation model can be obtained. The recommended control action is one of the multiple control actions included in the set of alternative control actions.

[0046] S104, obtaining feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information.

[0047] In some embodiments, after obtaining a control action recommended for a certain object, the recommended control action is implemented on the object, and feedback information from the object regarding the recommended control action can be obtained. The feedback information includes, but is not limited to, whether the object files a complaint about the control action within a preset time range from the start of the implementation of the recommended control action on the object, whether a risk related to the object occurs within the preset time range, etc. This example embodiment does not specifically limit this.

[0048] In some embodiments, the recommendation model is made to perform online reinforcement learning based on the feedback information obtained about the control actions historically recommended to each object and the preset reward function, that is, by inputting each feedback information into the preset reward function to obtain the corresponding reward score, the recommendation model is made to perform online reinforcement learning based on each feedback information and its corresponding reward score. For example, if the feedback information is that the object has a risk, the reward score corresponding to the feedback information is 1, and if the feedback information is that the object has not a risk, the reward score corresponding to the feedback information is 0. In some embodiments, the recommendation model will continuously learn how to make the best decision in a given situation, that is, the recommendation model will continuously learn how to recommend the best control action for each merchant. In the process of reinforcement learning, the recommendation model will adjust the behavior strategy according to the preset reward function to maximize the cumulative reward.

[0049] In some embodiments, during the reinforcement learning process, when there is a risk to an object, that is, when a control action needs to be taken on the object, the recommendation model will input the recommendation model based on the current learning level according to the input object characteristics of the object and the set of alternative control actions corresponding to the object group to which the object belongs, and the control action recommended for the object output by the recommendation model can be obtained, and the control action is implemented on the object, and reinforcement learning is performed based on the feedback information of the object regarding the control action, and when the object is at risk again, the recommendation model will output the latest recommended control action for the object based on the current latest learning level again, so as to continue iterating. In the continuous reinforcement learning process of the recommendation model, for the same object, due to the difference in the current learning level of the recommendation model at different times, the recommended control actions output by the recommendation model for the object at different times may also be different, so that the feedback information of the object regarding each control action in the set of alternative control actions corresponding to the object group to which the object belongs can be obtained in the continuous reinforcement learning process.

[0050] In some embodiments, during the continuous learning process of the recommendation model, as the feedback information of each object on each control action implemented on it is continuously updated, the current real-time value of each object under each control action on the preset indicator can be obtained. Based on the data-driven feedback information, the current real-time value will continue to approach the unbiased estimate of the true value of the preset indicator, and finally obtain the unbiased estimated value of each object under each control action on the preset indicator. In some embodiments, the preset indicators include but are not limited to PRO (probability of risk), PCO (probability of complain), etc., and this example embodiment does not specifically limit this.

[0051] S106, performing optimal management and control actions for the multiple objects according to the business objectives and the unbiased estimated values, and obtaining target management and control actions corresponding to the multiple objects under the business objectives.

[0052] In some embodiments, after obtaining the unbiased estimated values ​​of the preset indicators for each object under each control action through online reinforcement learning, based on the current business goal, the balanced risk control and object experience are modeled as a knapsack optimization problem for real-time solution, that is, the real-time optimization solution of the control actions for the multiple objects is performed to obtain the target control actions corresponding to each object under the business goal, so as to achieve the business goal by implementing the corresponding target control actions on the object when there is a risk for a certain object (that is, when a control action needs to be implemented on a certain object). In some embodiments, the business goal refers to the control goal formulated by the current business for the overall situation of the preset indicators for the multiple objects. The business goal includes but is not limited to the overall experience goal, the overall risk goal, etc. of the multiple objects. For example, the experience goal is "the merchant experience does not decline", the risk goal is "minimize the risk rate", and the business goal is "minimize the risk rate on the basis of ensuring that the merchant experience does not decline". This example embodiment does not specifically limit the specific content of the business goal. In some embodiments, the target control action corresponding to each object refers to the control plan applicable to the object determined by optimizing the control actions for the multiple objects; the combination of the target control actions corresponding to each object can achieve the optimal overall control efficiency and effect while achieving the business goals, that is, implementing the corresponding target control action for each object is best for the entire business goal.

[0053] The embodiment of this specification designs an object management decision system based on online reinforcement learning and real-time operations optimization methods. The feedback of the object on different management and control actions is obtained in real time through online reinforcement learning, so as to achieve unbiased estimation of preset indicators for different objects under different management and control actions; at the same time, after obtaining reliable indicator estimates, through real-time operations optimization, the best management and control plan for optimizing business objectives is sought by using mathematical multi-objective constraints, which can solve the problem of management and control exposure bias. Through the accumulation of real-time feedback online, the unbiased estimate of the preset indicators is continuously approached in a data-driven manner, and the unbiased management and control action feedback is directly learned. Furthermore, a dynamic tuning mechanism is also set up. Through the real-time optimization solution of the business objectives, real-time automatic adjustment can be made according to the calculated optimal management and control plan, which can adapt to different business needs and reduce manpower and resource costs, and achieve the optimization of overall management and control efficiency and effect.

[0054] In some embodiments, the method further includes: grouping the multiple objects according to the object characteristics of the multiple objects to obtain multiple object groups, wherein each object group includes one or more objects. In some embodiments, according to a preset grouping rule, multiple objects are pre-divided into multiple object groups (e.g., merchant groups) according to their respective object characteristics, and each object group includes one or more objects. For example, each object group corresponds to a different risk level. For an object, the risk level corresponding to the object is first determined according to the object characteristics of the object, and then the object is divided into an object group corresponding to the risk level. For another example, each object group corresponds to a different historical control number interval. For an object, the historical control number of the object is first determined according to the historical control situation of the object, and then the object is divided into an object group corresponding to the historical control number interval into which the historical control number falls. Here, it should be noted that the above grouping method is only an example, not a limitation. Those skilled in the art should understand that any grouping method can be included in the protection scope of this specification.

[0055] In some embodiments, the method further includes: for each object group among the multiple object groups, obtaining a set of alternative management and control actions corresponding to the object group, wherein the set of alternative management and control actions corresponding to each object group includes multiple management and control actions pre-assigned to the object group. In some embodiments, for each object group, a set of alternative control actions corresponding to the object group including multiple control actions is obtained, for example, a set of alternative control actions pre-specified by the system or system operator for the object group is obtained, for example, a certain object group has a more serious risk problem, so its corresponding set of alternative control actions often includes stricter control actions and longer control time, and a certain object group has a lower risk, so its corresponding set of alternative control actions includes a shorter control time, and the control often does not exceed 7 days, and can even be directly let go (no control), for example, first determine the group attribute characteristics of the object group according to the grouping rules, and from the preset multiple alternative control actions, a set of one or more alternative control actions that match the group attribute characteristics is used as the set of alternative control actions corresponding to the object group, for example, the group attribute characteristics are input into a trained control action recommendation model, and the set of multiple control actions output by the model is used as the set of alternative control actions corresponding to the object group. The embodiments of this specification can ensure that the overall object feedback situation can be reasonably controlled before starting online reinforcement learning through grouping and alternative control action sets.

[0056] In some embodiments, the method further includes: adjusting the learning rate of the recommendation model for the feedback information according to the feedback duration corresponding to the feedback information. In some embodiments, in order to solve the problem that different objects have different feedback times and delayed feedback under different control actions, the learning rate of the recommendation model can be adjusted according to the different feedback times received. The update of the recommended control action for objects with long delayed feedback time will also be slower, helping the recommendation model to make real-time decisions more reasonably. The specific formula is as follows:

[0057] a = a0*exp(-r*d)

[0058] Wherein, a0 is the predefined basic learning rate, r is the decay weight of delayed feedback, d is the time (in days) since the control, and a is the final learning rate of the recommendation model for the feedback information. The embodiments of this specification can achieve unbiased estimation of preset indicators by adjusting the learning rate according to the delayed feedback problem, and can solve the exposure bias problem that is naturally present in object control decisions.

[0059] In some embodiments, the obtaining of the unbiased estimated value of the preset index of the at least one object under each control action includes: in the reinforcement learning process of the recommendation model, by real-time statistics of multiple feedback information of the at least one object on each control action, obtaining the current real-time value of the preset index of the at least one object under each control action; using the current real-time value as the unbiased estimated value of the preset index of the at least one object under each control action. In some embodiments, in the reinforcement learning process of the recommendation model, real-time statistics are performed on multiple feedback information of each object on each control action, and statistical information of the object on each control action is obtained respectively. For example, control action action1 is recommended 3 times for merchant A, that is, control action action1 is implemented 3 times for merchant A, and according to multiple feedback information of merchant A on the 3 control actions action1, real-time statistics are performed to obtain corresponding statistical information "In these 3 control actions action1, the user has once filed a complaint against control action action1, and merchant A still encountered risks after implementing control action action1 twice". In some embodiments, the current real-time value of the preset indicator of the object under a certain control action can be determined based on the statistical information of each object about the control action. Continuing with the above example, according to the statistical information "In these three control actions action1, the user once filed a complaint against control action action1, and merchant A still encountered risks after control action action1 was implemented on merchant A twice", it can be determined that the current real-time value of merchant A's risk probability under control action action1 is 66%, and the current real-time value of the complaint probability is 33%. In some embodiments, each time the current real-time value of the preset indicator of an object under a certain control action is obtained, the current real-time value is used as the unbiased estimate value of the preset indicator of the object under the control action. For example, after the current real-time value of the preset indicator of the object under a certain control action is updated according to the feedback information of the object about the control action, the updated current real-time value will be used as the unbiased estimate value of the preset indicator of the object under the control action.In some embodiments, for an object, before obtaining feedback information about a certain control action of the object (for example, the control action has not yet been implemented on the object), an initial value of the preset indicator of the object under the control action will be set in advance, and after obtaining the corresponding current real-time value based on the feedback information, the current real-time value will be used to replace the previous initial value, wherein the initial value may be set by the system or a system operator, or may be determined based on the object characteristics of the object and the action characteristics of the control action, for example, the object characteristics of the object and the action characteristics of the control action are input into a trained model to obtain the initial value of the preset indicator of the object under the control action output by the model.

[0060] In some embodiments, the current real-time value is used as the unbiased estimated value of the preset index of the at least one object under each control action, including: if the current real-time value satisfies the preset unbiased condition, the current real-time value is used as the unbiased estimated value of the preset index of the at least one object under each control action. In some embodiments, only when the current real-time value obtained satisfies the preset unbiased condition, the current real-time value will be used as the unbiased estimated value of the preset index of the object under the control action, otherwise, the current real-time value will be ignored until the latest current real-time value is obtained again later, wherein the preset unbiased condition includes but is not limited to the similarity between the current real-time value and the last recently obtained real-time value satisfies the predetermined similarity condition (for example, the absolute value of the difference between the two is less than or equal to the predetermined threshold), the stability between the multiple recently obtained real-time values ​​including the current real-time value satisfies the predetermined stability condition (for example, the variance corresponding to the multiple real-time values ​​is less than or equal to the predetermined threshold), the number of feedbacks corresponding to the multiple feedback information used to obtain the current real-time value is greater than or equal to the predetermined number threshold, etc. This exemplary embodiment does not specifically limit the unbiased condition.

[0061] In some embodiments, the preset indicators include a risk probability indicator and a complaint probability indicator. In some embodiments, the preset indicators include a PRO (probability of risk) indicator and a PCO (probability of complain) indicator, PRO refers to the probability of the occurrence of a risk on the object after the control action is implemented on the object, and PCO refers to the probability that the object will file a complaint against the control action.

[0062] In some embodiments, the optimization solution of control actions is performed for the multiple objects according to the business objectives and the unbiased estimated values ​​to obtain target control actions corresponding to the multiple objects under the business objectives, including: determining an objective function and one or more constraints corresponding to the objective function according to the business objectives, wherein the input parameters of the objective function and the one or more constraints include the control action corresponding to each object, and the input parameters of the objective function and at least one constraint also include an unbiased estimated value of the object under the control action regarding the preset indicator; according to the objective function and the one or more constraints, the optimization solution of control actions is performed for the multiple objects to obtain the target control actions corresponding to the multiple objects under the business objectives. In some embodiments, after obtaining the unbiased estimated value of the preset indicator for each object under each control action through online reinforcement learning, the objective function and one or more constraints corresponding to the objective function can be determined according to the current real-time business goal, the input parameters of the objective function include the control action to be implemented corresponding to each object (that is, the control action to be implemented on each object) and the unbiased estimated value of the preset indicator for the object under the control action, the input parameters of the one or more constraints include the control action corresponding to each object, and at least one of the one or more constraints also includes the unbiased estimated value of the preset indicator for the object under the control action, according to the objective function and the one or more constraints, the control action is optimized for the multiple objects, that is, the objective function and the one or more constraints are optimized to obtain the target control action corresponding to each object under the business goal, and when there is a risk in one of the objects (that is, when a control action needs to be implemented on a certain object), the corresponding target control action is implemented on the object, wherein, while satisfying the one or more constraints, the solution of the above-mentioned constrained optimization problem is mainly to transform the dual problem through the Lagrangian function, and then solve it through a professional solver. For example, if the business goal is "to minimize the risk rate while ensuring that the merchant experience does not decline", the objective function and constraints determined based on this business goal are as follows:

[0063]

[0064] The first line is the objective function, the second and third lines are two constraints, i represents the merchant, j represents the control action, and a ij represents the implementation of control action j on merchant i, risk ij represents the unbiased estimate of the risk probability index of merchant i under control action j, exp ijRepresents the unbiased estimate of the complaint probability index of merchant i under control action j. The objective function represents minimizing the overall risk rate. The two constraints are "only one control action can be implemented for each merchant at a time" and "the overall complaint rate cannot exceed the initial value before the model iteration, that is, to ensure that the merchant experience does not decline". That is, the objective function is optimized under the condition that these two constraints are met to obtain the target control action corresponding to each merchant under the business goal, and the business goal is achieved by implementing the corresponding target control action on the merchant. In some embodiments, the control action to be implemented for each object included in the input parameters of the objective function and its corresponding constraints (that is, the control action to be implemented on each object) is one of the control actions in the set of alternative control actions corresponding to the object group to which the object belongs, that is, for a certain object, the target control action corresponding to the object under the business goal can only be one of the control actions in the set of alternative control actions corresponding to the object group to which the object belongs, and cannot be other control actions outside the set of alternative control actions.

[0065] In some embodiments, the control action corresponding to each object is one or more recommended control actions corresponding to the object predicted by the recommendation model. In some embodiments, the control action to be implemented for each object (i.e., the control action to be implemented for each object) included in the input parameters of the objective function and its corresponding constraint conditions is one of the one or more recommended control actions that the recommendation model has output for the object (i.e., the control actions that have been implemented on the object in history), that is, for a certain object, the target control action corresponding to the object under the business goal can only be one of the one or more recommended control actions that the model has recommended for the object in history (i.e., the control actions that have been implemented on the object in history), and cannot be other control actions other than the one or more recommended control actions. In some embodiments, if the recommendation model has not recommended a control action for the object before, that is, the control action has not been implemented on the object before, then the object does not participate in the optimization solution of the objective function, that is, the target control action corresponding to the object under the business goal will not be obtained.

[0066] In some embodiments, the method further includes: if the unbiased estimated value of the preset indicator of the object under at least one recommended control action corresponding to the object is not currently obtained, the initial value of the preset indicator of the object under the at least one recommended control action is used to replace the corresponding unbiased estimated value as the input parameter of the objective function and the at least one constraint. In some embodiments, for each object, if the unbiased estimated value of the preset indicator of the object under a recommended control action is not currently obtained, the initial value of the preset indicator of the object under the control action is used to replace the unbiased estimated value as the input parameter of the objective function and its corresponding constraint, that is, the initial value is used to replace the corresponding unbiased estimated value to participate in the optimization solution of the objective function, wherein the initial value has been described in detail in the previous text and will not be repeated here.

[0067] In some embodiments, the control action corresponding to each object is a recommended control action that has obtained a corresponding unbiased estimated value among one or more recommended control actions corresponding to the object predicted by the recommendation model. In some embodiments, the control action to be implemented for each object (i.e., the control action to be implemented for each object) included in the input parameters of the objective function and its corresponding constraint conditions is one of the one or more recommended control actions that the recommendation model has outputted for the object (i.e., the control actions that have been implemented on the object in history) that has obtained a corresponding unbiased estimated value, that is, for a certain object, the target control action corresponding to the object under the business goal can only be one of the one or more recommended control actions that the model has recommended for the object in history (i.e., the control actions that have been implemented on the object in history) that has obtained a corresponding unbiased estimated value, and cannot be a control action other than the one or more recommended control actions, nor can it be a control action in the one or more recommended control actions that has not yet obtained a corresponding unbiased estimated value. In some embodiments, if any of the one or more recommended control actions corresponding to the object has not obtained a corresponding unbiased estimated value, that is, if there is no control action in the one or more recommended control actions that has obtained a corresponding unbiased estimated value, then the object does not participate in the optimization solution of the objective function, that is, the target control action corresponding to the object under the said business objective will not be obtained.

[0068] Figure 2 A flowchart of an example method for object management provided in an embodiment of this specification.

[0069] like Figure 2As shown, multiple merchants are divided into multiple merchant groups according to their respective merchant characteristics, each merchant group includes one or more merchants, each merchant belongs to one of the multiple merchant groups, each merchant group corresponds to a set of candidate control actions, each candidate control action set includes multiple control actions pre-assigned to the merchant group, the merchant characteristics of each merchant and the set of candidate control actions corresponding to the merchant group to which the merchant belongs are input into the recommendation model, the recommendation model will output the control action recommended for the merchant (i.e., the recommended control action), after implementing the recommended control action on the merchant, the merchant's feedback information on the recommended control action is obtained, the recommendation model continuously performs reinforcement learning based on the feedback information, and finally obtains the unbiased estimated value of the merchant under the preset indicator under the control action, and then according to the current business goal and the unbiased estimated value, the control action is optimized for multiple merchants to obtain the optimal control action (i.e., the target control action) corresponding to each merchant under the business goal.

[0070] Figure 3 The present invention provides a schematic diagram of a structure of a device for object control provided in an embodiment of the present specification. The device for object control (hereinafter referred to as "object control device 1") can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the object control device 1 includes a control action recommendation module 11, a reinforcement learning module 12 and an optimization solution module 13.

[0071] The control action recommendation module 11 is used to build a recommendation model according to the object characteristics of multiple objects and the candidate control action sets corresponding to multiple object groups, and obtain the recommended control action corresponding to at least one object predicted by the recommendation model, wherein each object group includes one or more objects, and the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the candidate control action set corresponding to the object group to which the object belongs;

[0072] A reinforcement learning module 12, configured to obtain feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information;

[0073] The optimization solution module 13 is used to perform optimization solutions for the management and control actions for the multiple objects according to the business objectives and the unbiased estimated values, and obtain target management and control actions corresponding to the multiple objects under the business objectives.

[0074] In some embodiments, the object management and control device 1 is further used to group the multiple objects according to their object features to obtain multiple object groups, wherein each object group includes one or more objects.

[0075] In some embodiments, the object management device 1 is also used to: for each object group among the multiple object groups, obtain a set of alternative management and control actions corresponding to the object group, wherein the set of alternative management and control actions corresponding to each object group includes multiple management and control actions pre-assigned to the object group.

[0076] In some embodiments, the object management device 1 is further used to: adjust the learning rate of the recommendation model for the feedback information according to the feedback duration corresponding to the feedback information.

[0077] In some embodiments, obtaining an unbiased estimated value of a preset indicator for the at least one object under each control action includes: during the reinforcement learning process of the recommendation model, obtaining a current real-time value of the preset indicator for the at least one object under each control action by performing real-time statistics on multiple feedback information of the at least one object regarding each control action; and using the current real-time value as an unbiased estimated value of the preset indicator for the at least one object under each control action.

[0078] In some embodiments, the using the current real-time value as an unbiased estimated value of the preset indicator for the at least one object under each control action includes: if the current real-time value satisfies a preset unbiased condition, using the current real-time value as an unbiased estimated value of the preset indicator for the at least one object under each control action.

[0079] In some embodiments, the preset indicators include a risk probability indicator and a complaint probability indicator.

[0080] In some embodiments, the optimization solution of control actions is performed for the multiple objects according to the business objectives and the unbiased estimated values ​​to obtain target control actions corresponding to the multiple objects under the business objectives, including: determining an objective function and one or more constraints corresponding to the objective function according to the business objectives, wherein the input parameters of the objective function and the one or more constraints include the control action corresponding to each object, and the input parameters of the objective function and at least one constraint also include an unbiased estimated value of the object under the control action regarding the preset indicator; according to the objective function and the one or more constraints, the optimization solution of control actions is performed for the multiple objects to obtain the target control actions corresponding to the multiple objects under the business objectives.

[0081] In some embodiments, the control action corresponding to each object is one or more recommended control actions corresponding to the object that were predicted by the recommendation model.

[0082] In some embodiments, the object control device 1 is also used to: if the unbiased estimated value of the preset indicator of the object under at least one recommended control action corresponding to the object is not currently obtained, the initial value of the preset indicator of the object under the at least one recommended control action is used to replace the corresponding unbiased estimated value as the input parameter of the objective function and the at least one constraint condition. In some embodiments, the control action corresponding to each object is a recommended control action that has obtained a corresponding unbiased estimated value among one or more recommended control actions corresponding to the object predicted by the recommendation model.

[0083] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0084] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0085] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0086] The embodiments of this specification also provide Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of the embodiment of this specification.

[0087] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0088] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0095] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for object management and control, comprising: A recommendation model is constructed according to the object features of multiple objects and the candidate control action sets corresponding to multiple object groups, and a recommended control action corresponding to at least one object predicted by the recommendation model is obtained, wherein each object group includes one or more objects, and the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the candidate control action set corresponding to the object group to which the object belongs; Obtaining feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information; According to the business objectives and the unbiased estimated values, the management and control actions are optimized for the multiple objects to obtain the target management and control actions corresponding to the multiple objects under the business objectives.

2. The method according to claim 1, further comprising: The multiple objects are grouped according to their object features to obtain multiple object groups, wherein each object group includes one or more objects.

3. The method according to claim 2, further comprising: For each object group in the multiple object groups, a candidate control action set corresponding to the object group is obtained, wherein the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group.

4. The method according to claim 1, further comprising: Adjust the learning rate of the recommendation model for the feedback information according to the feedback duration corresponding to the feedback information.

5. According to the method of claim 1, obtaining an unbiased estimated value of a preset indicator of the at least one object under each control action comprises: In the reinforcement learning process of the recommendation model, a current real-time value of a preset indicator of the at least one object under each control action is obtained by performing real-time statistics on multiple feedback information of the at least one object on each control action; The current real-time value is used as an unbiased estimated value of the preset indicator for the at least one object under each control action.

6. According to the method of claim 5, the step of using the current real-time value as an unbiased estimated value of the at least one object with respect to the preset indicator under each control action comprises: If the current real-time value satisfies a preset unbiased condition, the current real-time value is used as an unbiased estimated value of the preset indicator for the at least one object under each control action.

7. According to the method according to any one of claims 1 to 6, the preset indicators include risk probability indicators and complaint probability indicators.

8. The method according to claim 1, wherein the optimizing solution of the control actions for the multiple objects is performed according to the business objectives and the unbiased estimated values ​​to obtain the target control actions corresponding to the multiple objects under the business objectives, comprising: Determine an objective function and one or more constraints corresponding to the objective function according to the business objective, wherein the input parameters of the objective function and the one or more constraints include the control action corresponding to each object, and the input parameters of the objective function and at least one constraint also include an unbiased estimated value of the object under the control action with respect to the preset indicator; According to the objective function and the one or more constraints, the control actions are optimized for the multiple objects to obtain the target control actions corresponding to the multiple objects under the business objectives.

9. According to the method of claim 8, the control action corresponding to each object is one or more recommended control actions corresponding to the object that were predicted by the recommendation model.

10. The method according to claim 9, further comprising: If an unbiased estimated value of the preset indicator for the object under at least one recommended control action corresponding to the object is not currently obtained, the initial value of the preset indicator for the object under the at least one recommended control action is used to replace the corresponding unbiased estimated value as an input parameter of the objective function and the at least one constraint condition.

11. According to the method described in claim 9, the control action corresponding to each object is a recommended control action that has obtained a corresponding unbiased estimated value among one or more recommended control actions corresponding to the object that were predicted by the recommendation model.

12. A device for object management and control, comprising: A control action recommendation module is used to build a recommendation model based on the object features of multiple objects and the candidate control action sets corresponding to multiple object groups, and obtain a recommended control action corresponding to at least one object predicted by the recommendation model, wherein each object group includes one or more objects, and the candidate control action set corresponding to each object group includes multiple control actions pre-assigned to the object group, and the recommended control action corresponding to an object is one of the control actions in the candidate control action set corresponding to the object group to which the object belongs; A reinforcement learning module, configured to obtain feedback information of the at least one object regarding the recommended control action, so that the recommendation model obtains an unbiased estimated value of a preset indicator of the at least one object under each control action by performing reinforcement learning based on the feedback information; The optimization solution module is used to perform optimization solutions for the management and control actions for the multiple objects according to the business objectives and the unbiased estimated values, and obtain the target management and control actions corresponding to the multiple objects under the business objectives.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 11.

15. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images