Model increment training method and device and object grade analysis system

Abstract

The embodiment of the invention discloses a model increment training method and device and an object level analysis system. A specific embodiment of the method comprises the following steps: in response to determining that a prediction result of an online object grade prediction model is attenuated, obtaining object information for executing a target operation on an application within a preset time period; performing incremental training on the object grade estimation model by using the object information to obtain a new object grade estimation model; and checking the new object grade estimation model, and in response to the passing of the checking, carrying out online processing on the new object grade estimation model to update the object grade estimation model. The implementation mode is related to a big data risk control technology, and the incremental training of the model can be carried out by utilizing the new data after the user data is deviated, so that new data distribution knowledge is learned, and the prediction effect of the model, namely the distinguishing capability of the user risk, is ensured. This helps to reduce or avoid the number of overdue users.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of big data risk control technology, and specifically to a model incremental training method, device, and object level analysis system. Background Art

[0002] In the risk control scenario, credit models are usually used to rate users’ credit. Credit rating is usually based on a set of relevant indicators, which indicate the ability and willingness of individuals or enterprises to repay their debts.

[0003] However, the inventors found that with the macroeconomic regulation of the country and the passage of time, the effectiveness of these credit models in rating users tends to decline. This will lead to the inability to accurately rate users, thus increasing the number of overdue users.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a model incremental training method, a model incremental training device, an object level analysis system, an electronic device, a computer-readable medium and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a model incremental training method, including: in response to determining that the prediction result of an online object level estimation model has decayed, obtaining object information of a target operation performed on an application within a preset time period; using the object information, incrementally training the object level estimation model to obtain a new object level estimation model; testing the new object level estimation model, and in response to passing the test, processing the new object level estimation model online to update the object level estimation model.

[0008] In some embodiments, obtaining object information that performs a target operation on an application within a preset time period includes: obtaining object information that performs a target operation on an application within a recent first time period; and using the object information to incrementally train an object level prediction model to obtain a new object level prediction model, including: using the object information within the first time period to perform real-time incremental training on the object level prediction model to obtain an incremental update model.

[0009] In some embodiments, obtaining object information on a target operation performed on an application within a preset time period includes: obtaining object information on all objects from an application database, wherein the application database stores the latest data on all objects that have performed the target operation on the application; and using the object information to incrementally train an object level estimation model to obtain a new object level estimation model, including: using the object information in the application data to retrain the object level estimation model to obtain a fully updated model.

[0010] In some embodiments, a new object level estimation model is tested, including: using verification object data to evaluate the prediction effect of the new object level estimation model; in response to determining that an evaluation indicator of the model reaches an indicator threshold, determining that the effect evaluation of the new object level estimation model passes, wherein the evaluation indicator includes at least one of the following: area under the curve, Lorenz curve, recall rate.

[0011] In some embodiments, testing the new object level estimation model also includes: packaging and storing the new object level estimation model that has passed the effect evaluation into a model database; in response to determining that the storage is complete, performing a consistency test on the new object level estimation model in the model database using the test object data; in response to determining that the prediction result of the new object level estimation model in the model database is consistent with the prediction result of the new object level estimation model, determining that the new object level estimation model in the model database has passed the test.

[0012] In some embodiments, bringing a new object level estimation model online includes: configuring parameters of the new object level estimation model in a model database, and bringing the configured new object level estimation model online, wherein the parameters include version parameters of the model.

[0013] In some embodiments, the method further includes at least one of the following: detecting that a current prediction result of an online object level estimation model deviates from a previous prediction result of the object level estimation model by more than a deviation threshold, and determining that the prediction result of the object level estimation model has decayed; or detecting that an operation behavior of an object on an application is inconsistent with a prediction result of the online object level estimation model for the object, and determining that the prediction result of the object level estimation model has decayed; or in response to determining that a set update cycle duration has been reached, determining that the prediction result of the online object level estimation model has decayed.

[0014] In a second aspect, some embodiments of the present disclosure provide a model incremental training device, including: an acquisition unit, configured to acquire object information of a target operation performed on an application within a preset time period in response to determining that a prediction result of an online object level estimation model has decayed; an incremental training unit, configured to perform incremental training on the object level estimation model using the object information to obtain a new object level estimation model; a model verification unit, configured to verify the new object level estimation model, and in response to passing the verification, to process the new object level estimation model online to update the object level estimation model.

[0015] In some embodiments, the acquisition unit is further configured to acquire object information of a target operation performed on an application within a recent first time period; and the incremental training unit is further configured to perform real-time incremental training on the object level prediction model using the object information within the first time period to obtain an incremental update model.

[0016] In some embodiments, the acquisition unit is further configured to acquire object information of all objects from an application database, wherein the application database stores the latest data of all objects that have performed target operations on the application; and the incremental training unit is further configured to use the object information in the application data to retrain the object level estimation model to obtain a fully updated model.

[0017] In some embodiments, the model verification unit is further configured to use verification object data to evaluate the prediction effect of the new object level estimation model; in response to determining that the evaluation index of the model reaches the index threshold, it is determined that the effect evaluation of the new object level estimation model passes, wherein the evaluation index includes at least one of the following: area under the curve, Lorenz curve, recall rate.

[0018] In some embodiments, the model verification unit is further configured to package and store the new object level estimation model that has passed the effect evaluation into the model database; in response to determining that the storage is completed, use the test object data to perform a consistency check on the new object level estimation model in the model database; in response to determining that the prediction result of the new object level estimation model in the model database is consistent with the prediction result of the new object level estimation model, it is determined that the new object level estimation model in the model database has passed the verification.

[0019] In some embodiments, the model verification unit is further configured to perform parameter configuration on the new object level estimation model in the model database, and to bring the configured new object level estimation model online, wherein the parameters include version parameters of the model.

[0020] In some embodiments, the device also includes a decay detection unit, which is configured to do at least one of the following: detecting that a current prediction result of an online object level estimation model deviates from a previous prediction result of the object level estimation model by more than a deviation threshold, and determining that the prediction result of the object level estimation model decays; or detecting that an operation behavior of an object on an application is inconsistent with a prediction result of the online object level estimation model for the object, and determining that the prediction result of the object level estimation model decays; or in response to determining that a set update cycle duration has been reached, determining that the prediction result of the online object level estimation model decays.

[0021] In a third aspect, some embodiments of the present disclosure provide an object level analysis system, including: an application server, configured to, in response to receiving an object analysis request, determine a target object level estimation model to call according to the object analysis request, perform level prediction on the target object indicated by the object analysis request through the target object level estimation model, provide corresponding application services to the target object according to the prediction result, and perform attenuation detection on the prediction result of the target object level estimation model, and send a training task to a model training server in response to detecting model attenuation; a model training server, configured to, in response to receiving a training task sent by the application server, perform incremental training on the model indicated by the training task using the model incremental training method described in any implementation method of the first aspect above to update the model.

[0022] In a fourth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the model incremental training method described in any implementation method in the above-mentioned first aspect.

[0023] In a fifth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the model incremental training method described in any implementation method in the above-mentioned first aspect is implemented.

[0024] In a sixth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the model incremental training method described in any implementation method in the first aspect above.

[0025] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the incremental model training method of some embodiments of the present disclosure can ensure the accuracy of the model prediction results. Specifically, as time goes by, the effectiveness of existing credit models in rating users will generally decline. The decline in model rating effect may be the result of the offset of user data, or it may be the decline in the predictive ability of the model itself, which is unable to learn new knowledge, resulting in the inability to accurately rate users after the arrival of new user data, and ultimately resulting in a decline in business results. A related solution is to improve the model effect by accessing different types of data sources for model training in the early stage of model training. In other words, the related technology improves the effect of the model by accessing multiple data sources and using enhanced credit data for model training. However, as the distribution of customer groups shifts, the model's prediction effect will still decay because the model cannot be automatically updated and upgraded. As a result, the credit rating of some users is not accurate enough.

[0026] Based on this, the incremental model training method of some embodiments of the present disclosure, as the customer data distribution shifts or the model shifts, causing the model prediction effect to shift, the system can promptly warn of model effect attenuation. And in the case of determining that the model effect is attenuated, incremental learning of the model can be performed by acquiring new data. This allows the model to learn new knowledge and accurately score customer ratings for the new data distribution. And before the incrementally trained model goes online, the model can be automatically tested. The new model that passes the test is used as an updated model to replace the attenuated model, and online real-time scoring is performed to meet business needs. In this way, the accuracy of the model prediction results can be always guaranteed, thereby achieving accurate rating of user credit and reducing the number of overdue users caused by prediction attenuation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0028] Figure 1 is a flowchart of some embodiments of the model incremental training method disclosed in the present invention;

[0029] Figure 2 It is a schematic diagram of the structure of some embodiments of the model incremental training device disclosed in the present invention;

[0030] Figure 3 is a schematic diagram of the architecture of some embodiments of the object level analysis system disclosed in the present invention;

[0031] Figure 4A yes Figure 3 A functional structural diagram of some embodiments of the application server shown in FIG.

[0032] Figure 4B yes Figure 3 A functional structural diagram of some embodiments of the model training server shown in;

[0033] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0035] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0038] With regard to the collection, storage, and use of user personal information (such as user attribute information and user operations on the application) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0039] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0040] Figure 1 A process 100 of some embodiments of the model incremental training method according to the present disclosure is shown. The training method comprises the following steps:

[0041] Step 101 : in response to determining that a prediction result of an online object level estimation model decays, obtaining object information of a target operation performed on an application within a preset time period.

[0042] In some embodiments, the execution body of the model incremental training method (for example Figure 3 In the example of the model training server shown in , when it is determined that the prediction result of the online object level estimation model has attenuated, the object information of the target operation performed on the application within a preset time period can be obtained through a wired connection or a wireless connection. Among them, the object level estimation model can be used to estimate the model of the relevant level of the object (user). The model can adopt a commonly used machine learning network model structure, such as support vector machine, Bayesian, etc. in the commonly used classification models. Support vector machine is a type of generalized linear classifier that performs binary classification of data in a supervised learning manner. This classification algorithm has been proven to have good results in many problems and is considered to be one of the most adaptable algorithms. The Bayesian method usually uses the knowledge of probability statistics to classify sample data sets.

[0043] The relevant levels and level divisions here can be set according to actual conditions. For example, the user credit rating in the credit scenario can be divided into good credit, medium credit and poor credit. The above target operation is also not limited, for example, it can be an operation related to the object's relevant level. In a credit application, when a user performs a credit transaction or a new user is granted credit, it is necessary to request the user's relevant authorization information to perform a real-time rating on the applicant. In other words, when a user performs a target operation, it is usually necessary to determine the user's level information.

[0044] In some embodiments, the execution entity can determine whether the prediction result of the model has decayed in a variety of ways. As an example, if it is detected that the current prediction result of the online object level estimation model deviates from the previous prediction result of the object level estimation model by more than a deviation threshold, it can be determined that the prediction result of the object level estimation model has decayed. For example, the model previously predicted that the proportion of users with good credit was approximately 80% or more (or a specific number), but recently predicted that the proportion (or number) of users with good credit has decreased. And the decrease difference is greater than the deviation threshold, then it can be determined that the model prediction result has decayed. This method can achieve real-time detection of model decay.

[0045] Optionally, it is also possible to determine whether the prediction results of the model have decayed through the actual operation behavior of the object on the application. As an example, if it is detected that the operation behavior of the object on the application is inconsistent with the prediction result of the online object-level prediction model for the object, it can be determined that the prediction result of the object-level prediction model has decayed. For example, the model predicts that a certain user is a user with good credit, but later the user is overdue, that is, he did not repay the loan on time. At this time, it can be considered that the user's operation behavior is inconsistent with the prediction result. Then it can be determined that the model prediction result has decayed. This method usually requires a business cycle (such as from borrowing to repayment cycle), that is, it takes a certain amount of time to achieve detection.

[0046] In some optional implementations, an update cycle may also be set. In this case, if it is determined that the set update cycle duration has been reached, the execution entity may determine that the prediction result of the online object level estimation model has decayed, thereby updating the model. The update cycle duration here may be set according to demand, for example, by statistically analyzing the duration of the model decay.

[0047] In addition, the above-mentioned preset time period is also not limited. For example, it can be a recent period of time (first time period), such as one day, three days or one week. For another example, it can also be the entire application duration, such as from the launch of the application to the present.

[0048] Get information about objects that perform target operations on the application within a preset time period

[0049] Step 102: perform incremental training on the object level estimation model using the object information to obtain a new object level estimation model.

[0050] In some embodiments, the execution subject may use the object information obtained in step 101 to perform incremental training on the above object level estimation model to obtain a new object level estimation model. Figure 3In the near real-time subsystem shown in , the execution subject can perform incremental training on the model. For example, if the execution subject obtains object information within a recent first time period, the object level estimation model can be incrementally trained in real time using the object information within the first time period, thereby obtaining an incremental update model.

[0051] It is understandable that through real-time online data (i.e., real-time computing platform), data from a short period of time is used to update the model under fixed network and hyperparameters. In this way, the effect of the model can be improved by performing quasi-real-time incremental training on the decaying model. The incrementally trained model not only has the customer group distribution at the beginning of training, but also can learn the distribution knowledge of new data. When new customer group data arrives, the user's level can be accurately estimated (such as credit rating).

[0052] In some application scenarios, such as Figure 3 In the offline subsystem shown in , if the execution subject obtains the object information of all objects from the application database, such as MQ (message queue), LOG (log file), HIVE (data warehouse), etc., the application database may store the latest data of all objects that have performed the target operation on the application. In this case, the execution subject can use the object information in the application data to retrain the object level estimation model, thereby obtaining a fully updated model.

[0053] It is understandable that the object level estimation model (such as the credit model) is retrained through offline large batches of data. The model is trained using the latest customer data so that the model can fully learn the data distribution of the new customer group and can accurately estimate the user level.

[0054] Step 103 , testing the new object level estimation model, and in response to passing the test, performing online processing on the new object level estimation model to update the object level estimation model.

[0055] In some embodiments, the execution subject may test the new object level prediction model. If the test is confirmed to be passed, the new object level prediction model may be put online to update the object level prediction model. It should be noted that the test content and test method are also not limited.

[0056] As an example, the above-mentioned test may include an evaluation of the prediction effect of the model. Specifically, the execution subject may use the verification object data to evaluate the prediction effect of the new object level estimation model. If it is determined that the evaluation index of the model reaches the index threshold, it can be determined that the effect evaluation of the new object level estimation model has passed. Among them, the evaluation index may include at least one of the following: AUC (Area Under Curve, area under the ROC curve), Lorenz curve (KS, Kolmogorov-Smirnov), recall rate, accuracy rate, etc. Among them, ROC (Receiver Operating Characteristic) is generally referred to as the receiver operating characteristic. KS is often used to evaluate the risk differentiation ability of the model, and the indicator measures the difference between the cumulative distributions of good and bad samples. The greater the cumulative difference between good and bad samples, the larger the KS index, and the stronger the risk differentiation ability of the characterization model.

[0057] In some embodiments, the above-mentioned verification may also include a consistency check of the model results. Specifically, the execution entity may package and store the new object level estimation model that has passed the effect evaluation into a model database, i.e., a model warehouse. Among them, the model database can be used for unified storage and management of models. Then, after determining that the storage is completed, the new object level estimation model in the model database may be subjected to a consistency check using the test object data. Here, if it is determined that the prediction result of the new object level estimation model in the model database is consistent with the prediction result of the new object level estimation model obtained by the previous training, it can be determined that the new object level estimation model in the model database has passed the test.

[0058] That is to say, after the new object level estimation model is trained, it can be tested using the test object data. In addition, after the model is stored in the model database, the model in the model database can also be tested using the test object data. If the two test results are the same, the consistency check of the model passes. At this point, it means that there is no error in the model during the packaging, uploading and storage process. That is, the model in the model database is the same as the trained model. It can be understood that the above-mentioned verification object data and test object data usually contain object information and corresponding level labels.

[0059] Furthermore, the execution entity can delete the online object level estimation model and put the new object level estimation model that has passed the inspection online to replace the previous model. This can reduce the storage space occupied. For example, the execution entity can delete the model from the storage location and store the new model in the storage location. For another example, the execution entity can store the new model in a different location, thereby changing the interface or address for calling the model online.

[0060] In some optional implementations, the execution entity may configure parameters for the new object level estimation model in the model database. The parameters may include, but are not limited to, model version parameters. They may also include feature parameter requirements for model input data, etc. Then, the configured new object level estimation model may be put online. In this way, the model version may be used to manage and distinguish the model.

[0061] It should be noted that in order to improve the prediction effect of the model, the relevant technology focuses on accessing more different credit information during the model training process, including strong credit data and consumer information, to accurately predict and control the user's credit rating. However, when the model effect decays after a period of time, this method still cannot solve the problem of the deterioration of the model rating effect. That is, it is impossible to learn new data distribution knowledge after the user data is offset, resulting in a decrease in the model's prediction ability, and even inability to meet business needs.

[0062] From the above description, it can be seen that the incremental model training method in some embodiments of the present disclosure, as the customer data distribution shifts or the model shifts, causing the model prediction effect to shift, the system can promptly warn of model effect attenuation. And in the case of determining that the model effect is attenuated, incremental learning of the model can be performed by acquiring new data. As a result, the model can learn new knowledge and accurately score customer ratings for the new data distribution. And before the incrementally trained model goes online, the model can be automatically tested. The new model that passes the test is used as an updated model to replace the attenuated model, and online real-time scoring is performed to meet business needs. In this way, the accuracy of the model prediction results can always be guaranteed, thereby achieving accurate rating of user credit and reducing the number of overdue users caused by prediction attenuation.

[0063] Continue to refer Figure 2 , as a response to the above Figure 1 In order to realize the training method shown in the figure, the present disclosure provides some embodiments of a model incremental training device. These training device embodiments are similar to Figure 1 The model incremental training device can be specifically applied to various electronic devices.

[0064] like Figure 2As shown, the model incremental training device 200 of some embodiments may include: an acquisition unit 201, configured to acquire object information of a target operation performed on an application within a preset time period in response to determining that a prediction result of an online object level estimation model has decayed; an incremental training unit 202, configured to perform incremental training on the object level estimation model using the object information to obtain a new object level estimation model; a model verification unit 203, configured to verify the new object level estimation model, and in response to passing the verification, to process the new object level estimation model online to update the object level estimation model.

[0065] In some embodiments, the acquisition unit 201 can be further configured to acquire object information that performs a target operation on an application within a recent first time period; and the incremental training unit 202 can be further configured to use the object information within the first time period to perform real-time incremental training on the object level prediction model to obtain an incremental update model.

[0066] In some embodiments, the acquisition unit 201 can be further configured to acquire object information of all objects from an application database, wherein the application database stores the latest data of all objects that have performed target operations on the application; and the incremental training unit 202 can be further configured to use the object information in the application data to retrain the object level estimation model to obtain a fully updated model.

[0067] In some embodiments, the model verification unit 203 can be further configured to use the verification object data to evaluate the prediction effect of the new object level estimation model; in response to determining that the evaluation index of the model reaches the index threshold, it is determined that the effect evaluation of the new object level estimation model passes, wherein the evaluation index includes at least one of the following: area under the curve, Lorenz curve, recall rate.

[0068] In some embodiments, the model verification unit 203 may be further configured to package and store the new object level estimation model that has passed the effect evaluation into the model database; in response to determining that the storage is completed, using the test object data to perform a consistency check on the new object level estimation model in the model database; in response to determining that the prediction result of the new object level estimation model in the model database is consistent with the prediction result of the new object level estimation model, it is determined that the new object level estimation model in the model database has passed the verification.

[0069] In some embodiments, the model verification unit 203 may be further configured to perform parameter configuration on the new object level estimation model in the model database, and to bring the configured new object level estimation model online, wherein the parameters include version parameters of the model.

[0070] In some embodiments, the training device 200 may further include a decay detection unit (not shown in the figure), which is configured to do at least one of the following: detecting that the current prediction result of the online object level estimation model deviates from the previous prediction result of the object level estimation model by more than a deviation threshold, and determining that the prediction result of the object level estimation model decays; or detecting that the operation behavior of the object in the application is inconsistent with the prediction result of the online object level estimation model for the object, and determining that the prediction result of the object level estimation model decays; or in response to determining that the set update cycle duration has been reached, determining that the prediction result of the online object level estimation model decays.

[0071] It can be understood that the units recorded in the model incremental training device 200 are similar to the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model incremental training device 200 and the units contained therein, and will not be repeated here.

[0072] Further references Figure 3 , which shows a schematic diagram of the system architecture of some embodiments of the object level analysis system disclosed in the present invention. The object level analysis system may include an application server and a model training server that interact with each other.

[0073] Here, when the application server receives the object analysis request, it can first determine the target object level estimation model to be called according to the object analysis request. Here, the object analysis request can be used to indicate the object information to be analyzed, as well as the model used for the analysis. For example, the request may include a user identifier and a version number of the model. Alternatively, the execution subject can determine the model to be used based on the object information indicated by the request. For example, if the object information mainly involves the user's credit operation data, it can be determined to use a credit (credit rating) model. For another example, if the object data mainly involves the user's order placement and product browsing data, it can be determined to use a user consumption level model.

[0074] Next, the target object level estimation model can be used to predict the level of the target object indicated by the object analysis request. After that, the corresponding application service can be provided to the target object according to the prediction result. For example, according to the credit rating result of the user, it can be determined whether to provide credit services and credit lines to the user. And the prediction result of the target object level estimation model can be attenuated. And when the model attenuation is detected, the training task can be sent to the model training server.

[0075] In addition, after receiving the training task sent by the application server, the model training server can use Figure 1The model incremental training method described in any implementation manner in the embodiments performs incremental training on the model indicated by the training task to update the model.

[0076] from Figure 3 As can be seen from the figure, the application server is the online reasoning subsystem, which is mainly responsible for online real-time calculations. For example, the online real-time calculation of the credit model in the risk control scenario. When a user processes a credit transaction or grants credit to a new user, it is necessary to perform real-time calculation of the model by requesting the user's relevant authorization information to perform real-time rating on the applicant. The functional modules of the application server can be as follows: Figure 4A As shown, it mainly includes gateway, feature and OS (Operating System, computer management control program).

[0077] Among them, the main function of the gateway is to limit the flow of upstream services to avoid the stability of the overall function of the inference service. By configuring and managing the DAG configuration files of different model inference services, the subsequent different feature processing nodes and the OS model online inference service are called and managed. In addition, A / B testing in the business process is carried out through the calling and proportion of different versions of models. Model support and request calling strategies (single and batch calls) are carried out through the configuration files of different models. The subsequent feature processing module and online computing module are required to support batch computing functions. Access permission management identifies and controls permissions for upstream callers. The DAG graph configuration file structure details the subsequent design, etc. Among them, DAG graph is an information technology term. The DAG data structure tracks the calculation and assignment of values ​​and variables in the basic block; the values ​​used from other places in the block are represented as leaf nodes; the operations on the values ​​are represented as internal nodes; the assignment of new values ​​is represented by appending the name of the target variable or temporary variable to the node representing the assignment. The purpose of A / B testing is generally to obtain representative experimental conclusions through scientific experimental design, sampling sample representativeness, traffic segmentation and small traffic testing, and to ensure that the conclusions are credible when extended to all traffic.

[0078] Feature processing nodes mainly perform feature processing for different model services. They perform calculations and management for different feature processing and processing functions. Online computing OS is mainly responsible for online reasoning of different models. OSS (Object Storage Service) object storage service is needed as a model warehouse to store model assets. For models with different performance requirements (concurrency, latency, etc.), python (programming language) or java version reasoning services can be provided to meet the needs of various scenarios. Models after incremental / full training are uniformly stored and managed through the model warehouse. The model launch pipeline process facilitates component management and labor efficiency improvement, and reduces the error rate of model service launch. Among them, pipeline process generally refers to the componentization of services in the system. When a service call is required, it is similar to the pipeline to call different module services in sequence. Here, the application server can use systems such as Kubernetes (often referred to as K8s, an open source system) for mixed deployment service management.

[0079] The model training server, i.e. the near real-time subsystem and the offline subsystem, solves the problem of learning new customer group distribution knowledge after the customer group distribution shifts through near real-time user features or offline full users. It memorizes the distribution of previously trained customer group data, thereby effectively improving the prediction effect of the model. The functional modules of this part can be as follows: Figure 4B As shown. The model can be trained through the model training platform, in which information such as data source and result storage location is configured. Training tasks can be started manually or by the system itself. After the training is completed, the available model is pushed to the model warehouse through the model evaluation and verification process, and the model can be updated and deployed online in the online reasoning system. The Embedding (vector) part can also be updated manually or regularly to update the full amount of embedded vectors in a timely manner. The data service proxy can obtain the data through the DataAPI (Application Program Interface).

[0080] It should be noted that the big data platform or MQ and other middleware mentioned in the object level analysis system of the present disclosure can be replaced by different components according to different scenarios. The key technical point of the technical solution disclosed in the present disclosure is to design a system solution for near-line / offline incremental training of models for the risk control credit scenario model due to the model effect attenuation problem caused by the offset of customer data distribution. After the credit model is attenuated, it can be monitored in the first time and the incremental training of the model can be automatically performed, as well as the full training service. And the new version of the model that meets the conditions will be automatically updated and iterated to meet business needs.

[0081] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0082] like Figure 5 As shown, the electronic device 500 may include a processing device 501 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the terminal device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0083] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a speaker, a vibrator, etc.; storage devices 508 including, for example, a disk, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0084] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0085] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0086] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0087] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to determining that the prediction result of the online object level estimation model has decayed, obtains the object information of the target operation performed on the application within a preset time period; uses the object information to incrementally train the object level estimation model to obtain a new object level estimation model; verifies the new object level estimation model, and in response to passing the verification, processes the new object level estimation model online to update the object level estimation model.

[0088] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0090] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, an incremental training unit, and a model verification unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as "a unit for acquiring object information for performing a target operation on an application within a preset time period".

[0091] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0092] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned model incremental training methods.

[0093] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A model incremental training method, comprising: In response to determining that a prediction result of the online object level estimation model decays, obtaining object information for performing a target operation on the application within a preset time period; Using the object information, incrementally training the object level estimation model to obtain a new object level estimation model; The new object level prediction model is tested, and in response to passing the test, the new object level prediction model is processed online to update the object level prediction model.

2. The model incremental training method according to claim 1, wherein: The obtaining of information about an object that performs a target operation on an application within a preset time period includes: Obtaining information about objects that have performed a target operation on the application within a recent first time period; and The method of incrementally training the object level estimation model by using the object information to obtain a new object level estimation model includes: The object level estimation model is incrementally trained in real time using the object information in the first time period to obtain an incremental update model.

3. The model incremental training method according to claim 1, wherein: The obtaining of information about an object that performs a target operation on an application within a preset time period includes: Acquire object information of all objects from an application database, wherein the application database stores the latest data of all objects on which a target operation has been performed on the application; and The method of incrementally training the object level estimation model by using the object information to obtain a new object level estimation model includes: The object level estimation model is retrained using the object information in the application data to obtain a fully updated model.

4. The model incremental training method according to claim 1, wherein: The testing of the new object level estimation model includes: Using the validation object data, evaluating the prediction effect of the new object level estimation model; In response to determining that an evaluation index of the model reaches an index threshold, it is determined that the effect evaluation of the new object level prediction model passes, wherein the evaluation index includes at least one of the following: area under the curve, Lorenz curve, and recall rate.

5. The model incremental training method according to claim 4, wherein: The testing of the new object level prediction model further includes: Packaging and storing the new object level estimation model that has passed the effect evaluation into a model database; In response to determining that the storing is complete, performing a consistency check on the new object level estimation model in the model database using the test object data; In response to determining that the prediction result of the new object level estimation model in the model database is consistent with the prediction result of the new object level estimation model, it is determined that the new object level estimation model in the model database has passed verification.

6. The model incremental training method according to claim 5, wherein: The online processing of the new object level estimation model includes: Parameters of the new object level estimation model in the model database are configured, and the configured new object level estimation model is put online, wherein the parameters include version parameters of the model.

7. The incremental model training method according to any one of claims 1 to 6, wherein: The method further comprises at least one of the following: Detecting that a current prediction result of the online object level estimation model deviates from a previous prediction result of the object level estimation model by more than a deviation threshold, and determining that a prediction result of the object level estimation model is attenuated; or Detecting that the operation behavior of the object on the application is inconsistent with the prediction result of the online object level prediction model for the object, and determining that the prediction result of the object level prediction model is attenuated; or In response to determining that the set update cycle duration has been reached, it is determined that the prediction result of the online object level estimation model decays.

8. A model incremental training device, comprising: An acquisition unit is configured to acquire object information for performing a target operation on an application within a preset time period in response to determining that a prediction result of an online object level estimation model decays; an incremental training unit, configured to perform incremental training on the object level estimation model using the object information to obtain a new object level estimation model; The model verification unit is configured to verify the new object level prediction model, and in response to passing the verification, perform online processing on the new object level prediction model to update the object level prediction model.

9. An object level analysis system, comprising: The application server is configured to, in response to receiving an object analysis request, determine a target object level estimation model to be called according to the object analysis request, perform level prediction on the target object indicated by the object analysis request through the target object level estimation model, provide corresponding application services to the target object according to the prediction result, and perform attenuation detection on the prediction result of the target object level estimation model, and send a training task to a model training server in response to detecting model attenuation; The model training server is configured to, in response to receiving a training task sent by the application server, perform incremental training on the model indicated by the training task using the model incremental training method described in one of claims 1-7 to update the model.

10. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the model incremental training method as described in any one of claims 1-7.

11. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the model incremental training method as described in any one of claims 1-7 is implemented.

12. A computer program product, comprising a computer program, which, when executed by a processor, implements the model incremental training method as described in any one of claims 1 to 7.

Images