Order quality evaluation model training method, evaluation method, device and medium

Abstract

The present invention provides an order quality evaluation model training method, evaluation method, device and medium. The method includes: obtaining an initial sample set according to mobile Internet number card order data, the initial sample set including the user's original feature vector, conversion rate and first recharge number; obtaining a mixed sample set, a sample set of users from other networks and a sample set of users from this network according to the basic feature vector in the initial sample set; the basic feature vector is a feature vector shared by user samples from this network and user samples from other networks; constructing and training a random forest model according to the mixed sample set, the sample set of users from other networks and the sample set of users from this network, obtaining a trained random forest model, the random forest model is used to output the conversion rate and the first recharge number according to the user's feature vector, and the conversion rate and the first recharge number are used to evaluate the order quality. The method of the present invention can effectively, scientifically and accurately evaluate Internet number card orders.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an order quality evaluation model training method, evaluation method, device and medium. Background Art

[0002] Operators can obtain activity indicator scores, business indicator scores or user value predictions through Internet channels and make reasonable statistics and quality divisions of customer data to facilitate better service resource allocation, secondary marketing and other businesses.

[0003] Existing relevant technical solutions include channel level assessment and user value acquisition. In terms of order prediction, the current method used by operators is to predict whether an order is malicious; the judgment conditions include false contact numbers, repeated orders within 30 days, and delivery address blacklists; if a malicious order is found, the subsequent order process will be stopped.

[0004] However, the current prediction and evaluation methods of technical solutions have relatively simple functions and lack certain effectiveness and scientificity. It is difficult to accurately allocate service resources and conduct secondary marketing for high-quality users based on the evaluation results. Summary of the invention

[0005] The present invention provides an order quality evaluation model training method, evaluation method, device and medium, which are used to solve the problems in the prior art that the functions are relatively single, lack certain effectiveness and scientificity, and it is difficult to accurately tilt service resources according to evaluation results and perform secondary marketing for high-quality users.

[0006] In a first aspect, the present invention provides an order quality evaluation model training method, comprising:

[0007] According to the mobile Internet number card order data, an initial sample set is obtained, wherein the initial sample set includes the user's original feature vector, conversion rate and first recharge amount, wherein the conversion rate is used to indicate the probability of a user placing an order to access the network, and the first recharge amount is the first recharge amount of a user who has accessed the network;

[0008] According to the basic feature vector in the initial sample set, a mixed sample set, a sample set of users from different networks and a sample set of users from the local network are obtained; the basic feature vector is a feature vector shared by the user samples from the local network and the user samples from different networks;

[0009] A random forest model is constructed and trained based on the mixed sample set, the cross-network user sample set and the local network user sample set to obtain a trained random forest model. The random forest model is used to output a conversion rate and a first-time recharge number based on the user's feature vector. The conversion rate and the first-time recharge number are used to evaluate order quality.

[0010] Furthermore, the obtaining of the mixed sample set, the sample set of users from different networks and the sample set of users from the local network according to the basic feature vectors in the initial sample set includes:

[0011] Selecting basic feature vectors and conversion rates in the initial sample set to form the mixed sample set;

[0012] Filter out samples that only include data of users from different networks in the initial sample set, and select the basic feature vector and the first charge number to form the sample set of users from different networks;

[0013] The initial sample set only includes samples of the local network user data, and the basic feature vector, the local network feature vector and the first charge number are selected to form the local network user sample set.

[0014] Furthermore, the basic feature vector includes: operating system, medium information, information streaming media, user information, number card information and order information;

[0015] The network feature vector includes: user network access time, user consumption data, pre-stored data and arrears data.

[0016] Furthermore, the random forest model includes: a hybrid random forest model, a different network random forest model, and a local network random forest model;

[0017] The construction and training processes of the hybrid random forest model, the cross-network random forest model, and the local network random forest model are the same;

[0018] Among them, the training data of the mixed random forest model is the mixed sample set, and the conversion rate is used as the output for training; the training data of the cross-network random forest model is the cross-network user sample set, and the first recharge number is used as the output for training; the sample data of the local network random forest model is the local network user sample set, and the first recharge number is used as the output for training.

[0019] Furthermore, the construction and training process of the target random forest model includes:

[0020] S1-1, randomly taking out N samples with replacement from the mixed sample set, repeating n times, to obtain n training sample sets;

[0021] S1-2, based on the n training sample sets, respectively training n regression decision tree models;

[0022] S1-3, select a regression decision tree model and a constant m, where the constant m is an integer less than the total number of feature vector types in the mixed sample set;

[0023] S1-4, randomly selecting m types of feature vectors from the total feature vectors of the mixed sample set as feature subsets;

[0024] S1-5, traverse all feature vectors in the feature subset, perform binary segmentation on each feature vector, and calculate the mean of the target data corresponding to the two groups of data after each feature vector is split and the mean square error of the target data;

[0025] S1-6, comparing the mean square errors of all target data, selecting two arrays with the smallest mean square errors of target data, and obtaining two arrays for the first binary split; the data in the arrays include the feature vector type, the number of samples, the mean of the target data, and the mean square error of the target data;

[0026] S1-7, return to step S1-5, and repeat the traversal until the mean square error of the target data corresponding to the leaf node of the regression decision tree split is 0;

[0027] S1-8, combine all the split regression decision trees into an initial target random forest model, calculate the out-of-bag error rate of the initial target random forest model, adjust the size of the feature vector species m according to the out-of-bag error rate, return to step S1-3 and repeat the training until the out-of-bag error rate is less than or equal to the set threshold or the target random forest model in training has overfitting phenomenon, and obtain the trained target random forest model;

[0028] Among them, the target random forest model is one of a mixed random forest model, a different network random forest model, and a local random forest model. The target data of the mixed random forest model is the conversion rate, and the target data of the different network random forest model and the local random forest model are the first charge number.

[0029] In a second aspect, the present invention provides an order quality evaluation method, the method comprising:

[0030] Extract user feature vectors based on test orders;

[0031] Inputting the user feature vector into a random forest model to obtain a conversion rate and a first-time recharge number output by the random forest model, wherein the random forest model is obtained by any of the methods described above;

[0032] The quality of the test order is evaluated based on the conversion rate and the first charge amount.

[0033] Furthermore, the step of inputting the user feature vector into a random forest model to obtain a conversion rate and a first-time recharge number output by the random forest model includes:

[0034] Inputting the user feature vector into a hybrid random forest model to obtain a first conversion rate output by the hybrid random forest model;

[0035] Determine whether the first conversion rate is less than the conversion threshold. If so, determine that the first recharge amount is equal to 0, and output the first conversion rate and the first first recharge amount. If not, determine whether the user is a user of this network according to the number card of the test order.

[0036] If the user is a user of this network, the user feature vector is input into the random forest model of this network to obtain the second filler;

[0037] If the user is not a user of this network, the user feature vector is input into the random forest model of the other network to obtain the third filler.

[0038] In a third aspect, the present invention provides an order quality evaluation model training device, comprising:

[0039] An order data initialization module is used to obtain an initial sample set according to the mobile Internet number card order data, wherein the initial sample set includes the user's original feature vector, conversion rate and first recharge amount, wherein the conversion rate is used to indicate the probability of the user placing an order to access the network, and the first recharge amount is the first recharge amount of the user who has accessed the network;

[0040] A sample set acquisition module, used to obtain a mixed sample set, a sample set of users from different networks and a sample set of users from the local network according to a basic feature vector in the initial sample set; the basic feature vector is a feature vector shared by the user samples of the local network and the user samples of different networks;

[0041] The random forest model establishment and training module is used to construct and train a random forest model based on the mixed sample set, the cross-network user sample set and the local network user sample set to obtain a trained random forest model. The random forest model is used to output a conversion rate and a first-time recharge number based on the user's feature vector. The conversion rate and the first-time recharge number are used to evaluate order quality.

[0042] In a fourth aspect, the present invention provides an order quality evaluation device, comprising:

[0043] Test order feature vector extraction module, used to extract user feature vector;

[0044] An order quality evaluation data acquisition module, used to input the user feature vector into a random forest model to obtain the conversion rate and first recharge number output by the random forest model;

[0045] An order quality evaluation module is used to evaluate the quality of the test order according to the conversion rate and the first recharge amount.

[0046] In a fifth aspect, an order quality evaluation device includes: at least one processor and a memory;

[0047] The memory stores computer-executable instructions;

[0048] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the order quality evaluation model training method or the order quality evaluation method.

[0049] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an order quality evaluation model training method and an evaluation method.

[0050] The order quality evaluation model training method provided by the present invention obtains an initial sample set based on mobile Internet number card order data, the initial sample set including the user's original feature vector, conversion rate and first recharge number, the conversion rate is used to indicate the probability of the user placing an order to access the network, and the first recharge number is the first recharge amount of the user who has accessed the network; based on the basic feature vector in the initial sample set, a mixed sample set, a sample set of users from other networks and a sample set of users from this network are obtained; the basic feature vector is a feature vector shared by the user samples of this network and the user samples of other networks; based on the mixed sample set, the sample set of users from other networks and the sample set of users from this network, a random forest model is constructed and trained to obtain a trained random forest model, the random forest model is used to output the conversion rate and the first recharge number according to the user's feature vector, and the conversion rate and the first recharge number are used to evaluate the order quality; the acquisition of Internet number card orders and the quality prediction and evaluation of the corresponding order users are realized, and the Internet number card order evaluation can be effectively, scientifically, and accurately performed, and then the service resources can be accurately tilted according to the evaluation results and secondary marketing can be performed for high-quality users. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0052] Figure 1 A schematic diagram of a system for evaluating order quality provided by an embodiment of the present invention;

[0053] Figure 2 A process of training a model for evaluating order quality provided by an embodiment of the present invention Figure 1 ;

[0054] Figure 3 A process of training a model for evaluating order quality provided by an embodiment of the present invention Figure 2 ;

[0055] Figure 4 A process of an order quality evaluation method provided by an embodiment of the present invention Figure 1 ;

[0056] Figure 5A process of an order quality evaluation method provided by an embodiment of the present invention Figure 2 ;

[0057] Figure 6 A diagram of an order quality evaluation model training device provided by an embodiment of the present invention;

[0058] Figure 7 A diagram of an order quality evaluation device provided by an embodiment of the present invention;

[0059] Figure 8 A hardware schematic diagram of an order quality evaluation device provided by an embodiment of the present invention;

[0060] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0062] Figure 1 A schematic diagram of an order quality evaluation system provided by an embodiment of the present invention. Figure 1 As shown, the system provided in this embodiment includes a terminal and a server. The terminal can be a mobile phone, a tablet, a computer, or other device that can interact with a user. For example, the terminal can be Figure 1 The computer 101 and / or mobile phone 102 shown. The computer 101 can communicate with the server 103 via a wired or wireless network. The mobile phone 102 can communicate with the server 103 via a wireless network.

[0063] The server 103 can provide various services to users, such as user order processing services. The server 103 obtains the access information of the terminal and generates training samples based on the access information. The server 103 can also send the obtained access information to other servers, and the other servers generate training samples based on the access information. This embodiment does not specifically limit the server that generates the training samples.

[0064] After obtaining the training sample, the server can train the training sample based on the model to obtain a trained model. The model in this embodiment is a random forest model.

[0065] Figure 2 A process of training a model for evaluating order quality provided by an embodiment of the present invention Figure 1 The execution subject of the method provided in this embodiment can be the above Figure 1 The server or other servers in the embodiment are not particularly limited here. Figure 2 As shown, the method includes:

[0066] S201. Based on the mobile Internet number card order data, an initial sample set is obtained, wherein the initial sample set includes the user's original feature vector, conversion rate and first recharge amount. The conversion rate is used to indicate the probability of the user placing an order to join the network, and the first recharge amount is the first recharge amount of the user who has joined the network.

[0067] This embodiment can be applied to a user order system, etc. For example, when a user browses and clicks to enter the number card order information interface, corresponding order data will be generated, and the model training can be performed by generating a corresponding initial sample set based on the order data.

[0068] Mobile Internet SIM card order data includes SIM card order data obtained by operators through mobile Internet. The order data may include some edge data, including delivery mobile phone number, delivery address, invalid user name and invalid identity document information. After removing the edge data, the initial sample set is obtained; the initial sample set includes the user's original feature vector, conversion rate and first recharge number; specifically, the conversion rate is used to indicate the probability of a user placing an order to access the network, and the first recharge number is the first recharge amount for a user who has already accessed the network.

[0069] Specifically, the conversion rate is used to indicate the probability of a user placing an order to access the network. That is, when the user only clicks to enter the number card order information interface but does not place an order to access the network, the first recharge number will not be generated, and the first recharge number will be 0.

[0070] Placing an order and entering the network means that the customer confirms the order and fills in the relevant information. Not placing an order and entering the network means that the customer did not place an order or canceled the order after reading the order.

[0071] S202. Obtain a mixed sample set, a sample set of users from different networks, and a sample set of users from the local network based on the basic feature vector in the initial sample set; the basic feature vector is a feature vector common to the user samples from the local network and the user samples from different networks.

[0072] According to the basic feature vector in the initial sample set, a mixed sample set, a sample set of users from different networks and a sample set of users from the local network are obtained; specifically, the basic feature vector is a feature vector common to the user samples from the local network and the user samples from different networks.

[0073] Specifically, the mixed sample set includes all samples in the initial sample set, that is, it includes both samples of users from different networks and samples of users from the local network. The mixed sample set, the sample set of users from different networks, and the sample set of users from the local network all include basic feature vectors, which are feature information included in all samples selected from the initial sample set obtained after removing marginal data, and are used as input data when establishing and training the model.

[0074] In a possible implementation, the mixed sample set, the sample set of users from different networks, and the sample set of users from the local network may be obtained in the following manner:

[0075] Selecting basic feature vectors and conversion rates in the initial sample set to form the mixed sample set;

[0076] Filter out samples that only include data of users from different networks in the initial sample set, and select the basic feature vector and the first charge number to form the sample set of users from different networks;

[0077] The initial sample set only includes samples of the local network user data, and the basic feature vector, the local network feature vector and the first charge number are selected to form the local network user sample set.

[0078] Specifically, the basic feature vector includes: operating system, media information, information streaming media, user information, number card information and order information; the local network feature vector includes user access time, user consumption data, pre-stored data and arrears data.

[0079] The operating system includes iOS and Android; the media information includes H5 pages, applications (APP) and public accounts; the information flow includes news consultation, short videos, shopping and social networking; the user information includes user age and user gender; the number card information includes the type of beautiful number, number card ownership and blacklist address, and the order information includes 30-day repeated orders and order time.

[0080] Specifically, the feature vector in the mixed sample set also includes the number of first recharges; the feature vectors in the different network user sample set and the local network user sample set also include the conversion rate.

[0081] The random forest model includes feature vectors and labels, and the label includes the mean output of conversion rate or the mean output of first recharge number. For a mixed sample set, the feature vector includes the first recharge number when the label is the conversion rate. For a sample set of users from different networks and a sample set of users from the same network, the feature vector includes the conversion rate when the label is the first recharge number.

[0082] S203. Construct and train a random forest model based on the mixed sample set, the cross-network user sample set, and the local network user sample set to obtain a trained random forest model. The random forest model is used to output a conversion rate and a first-time recharge number based on a user's feature vector. The conversion rate and the first-time recharge number are used to evaluate order quality.

[0083] A random forest model is constructed and trained based on the mixed sample set, the cross-network user sample set and the local network user sample set to obtain a trained random forest model; specifically, the random forest model is used to output a conversion rate and a first-charge number based on the user's feature vector, and the conversion rate and the first-charge number are used to evaluate order quality.

[0084] In a possible implementation, the random forest model includes: a hybrid random forest model, a different network random forest model, and a local network random forest model;

[0085] The construction and training process of the hybrid random forest model, the cross-network random forest model, and the local random forest model are the same;

[0086] Among them, the training data of the mixed random forest model is the mixed sample set, and the conversion rate is used as the output for training; the training data of the cross-network random forest model is the cross-network user sample set, and the first recharge number is used as the output for training; the sample data of the local network random forest model is the local network user sample set, and the first recharge number is used as the output for training.

[0087] An order quality evaluation model training method provided by an embodiment of the present invention obtains an initial sample set based on mobile Internet number card order data, the initial sample set including the user's original feature vector, conversion rate and first recharge number, the conversion rate is used to indicate the probability of the user placing an order to access the network, and the first recharge number is the first recharge amount of the user who has accessed the network; based on the basic feature vector in the initial sample set, a mixed sample set, a sample set of users from other networks and a sample set of users from this network are obtained; the basic feature vector is a feature vector shared by the user samples of this network and the user samples of other networks; based on the mixed sample set, the sample set of users from other networks and the sample set of users from this network, a random forest model is constructed and trained to obtain a trained random forest model, the random forest model is used to output the conversion rate and the first recharge number according to the user's feature vector, and the conversion rate and the first recharge number are used to evaluate the order quality; the acquisition of Internet number card orders and the quality prediction and evaluation of the corresponding order users are realized, and the Internet number card order evaluation can be effectively, scientifically, and accurately performed, and then service resources can be accurately tilted according to the evaluation results and secondary marketing can be performed for high-quality users.

[0088] Figure 3 A process of training a model for evaluating order quality provided by an embodiment of the present invention Figure 2 ,like Figure 3As shown in the figure, the construction and training process of the target random forest model in this method includes:

[0089] S301, randomly taking out N samples with replacement from the mixed sample set, repeating n times, to obtain n training sample sets;

[0090] Specifically, the method of extracting samples from the mixed sample set includes Bootstraping method;

[0091] S302, training n regression decision tree models respectively based on the n training sample sets;

[0092] S303, selecting a regression decision tree model and a constant m, wherein the constant m is an integer less than the total number of feature vector types in the mixed sample set;

[0093] S304, randomly selecting m types of feature vectors from the total feature vectors of the mixed sample set as feature subsets;

[0094] S305, traversing all feature vectors in the feature subset, performing binary segmentation on each feature vector, and calculating the mean of the target data corresponding to the two groups of data after each feature vector is split and the mean square error of the target data;

[0095] S306, comparing the mean square errors of all target data, selecting two arrays with the smallest mean square errors of target data, and obtaining two arrays for the first binary split; the data in the arrays include the feature vector type, the number of samples, the mean of the target data, and the mean square error of the target data;

[0096] Specifically, the method for calculating the mean square error of the target data includes the formula:

[0097]

[0098] Among them, MSE is the mean square error, k is the number of data in one of the two arrays of binary split, and y i is the i-th target data value of the array, is the mean of all target data in this array;

[0099] S307, return to step S305, and repeat the traversal until the mean square error of the target data corresponding to the leaf node of the regression decision tree split is 0;

[0100] S308, combining all the split regression decision trees into an initial target random forest model, calculating the out-of-bag error rate of the initial target random forest model, adjusting the size of the feature vector species m according to the out-of-bag error rate, returning to step S303 to repeat the training until the out-of-bag error rate is less than or equal to the set threshold or the target random forest model in training has overfitting, and obtaining a trained target random forest model;

[0101] Among them, the target random forest model is one of a mixed random forest model, a different network random forest model, and a local random forest model. The target data of the mixed random forest model is the conversion rate, and the target data of the different network random forest model and the local random forest model are the first charge number.

[0102] Specifically, the mean of the target data is constantly being calculated and updated throughout the training process.

[0103] The construction and training process of the target random forest model provided by the embodiment of the present invention, in the process of selecting samples, randomly selecting samples with replacement can make the selected samples cover multiple samples in the original mixed sample set at multiple levels, enhance the sample richness, ensure the possibility that the trained model can be applicable to a variety of number card orders, and enhance the accuracy of subsequent training; in the process of training the regression decision tree model after selecting samples, in order to ensure the effectiveness of data splitting and not increase the burden on the computer, a constant as small as possible is selected to select the feature vector; and the selected feature vector is split by comparing the mean square error, and the selected constant is corrected by calculating the out-of-bag error rate, so as to achieve accurate model training step by step. And in the training process, the users of this network include the user information of the users of this network obtained through the corresponding business system, and more feature vectors can be added for training than users of other networks, which increases the feasibility of the model.

[0104] Figure 4 A process of an order quality evaluation method provided by an embodiment of the present invention Figure 1 ,like Figure 4 As shown, the method includes:

[0105] S401, extracting a user feature vector according to a test order;

[0106] S402, inputting the user feature vector into a random forest model to obtain a conversion rate and a first-time recharge number output by the random forest model, wherein the random forest model is the target random forest model trained in S308;

[0107] S403: Evaluate the quality of the test order according to the conversion rate and the first recharge amount.

[0108] In this embodiment, the operator evaluates the quality of the order based on the output conversion rate and first-charge number.

[0109] Specifically, it is determined whether the conversion rate of the order to be tested is less than the conversion threshold. If so, the order to be tested is determined to be a C-level quality order, and the order quality assessment is completed; otherwise, it is determined whether the first charge number of the order to be tested is less than the first charge threshold;

[0110] If the first charge number of the order to be tested is less than the first charge threshold, the order to be tested is determined to be a Class B quality order, and the order quality evaluation is completed; otherwise, the order to be tested is determined to be a Class A quality order, and the order quality evaluation is completed.

[0111] The operator can provide subsequent services based on the order quality evaluation results. This embodiment proposes a method for the operator to evaluate the quality of an order, and this embodiment will not repeat other quality evaluation methods here.

[0112] The order quality evaluation method provided by the embodiment of the present invention inputs the test order into a trained model for prediction, and evaluates the quality of the user's number card order by selecting conversion rate and first recharge number as output in a feature output manner. This method can accurately and scientifically describe the quality of the order. For example, the higher the predicted conversion rate of the user's number card order, the more likely the user is to choose to place an order. The larger the predicted first recharge number of the user's number card order, the higher the first recharge amount selected by the user after placing the order, which means that the user is a high-quality user, and the higher the success rate of tilting service resources and secondary marketing to him.

[0113] In the above Figure 4 Based on the embodiment, the process of obtaining the conversion rate and first charge number output by the random forest model is described in detail. Figure 5 A process of an order quality evaluation method provided by an embodiment of the present invention Figure 2 ,like Figure 5 As shown, the method includes:

[0114] S501, inputting the user feature vector into a hybrid random forest model to obtain a first conversion rate output by the hybrid random forest model;

[0115] S502, determining whether the first conversion rate is less than a conversion threshold; if so, determining that the first recharge amount is equal to 0, and outputting the first conversion rate and the first first recharge amount; if not, determining whether the user is a user of this network according to the number card of the test order;

[0116] Specifically, the conversion threshold may be any probability value of 0-100%, and a value less than the conversion threshold may be set according to actual conditions to indicate that the user has not placed an order to join the network;

[0117] If yes, the first charge number is 0, then the first conversion rate and the first first charge number are output;

[0118] If no, it means the user has placed an order to join the network, and we can further predict the estimated first recharge amount of the order;

[0119] S503, if the user is a user of this network, input the user feature vector into the random forest model of this network to obtain a second filler;

[0120] Specifically, after the calculation is completed, the first conversion rate and the second first charge number are output;

[0121] S504, if the user is not a user of the local network, input the user feature vector into a random forest model of a different network to obtain a third filler;

[0122] Specifically, after the calculation is completed, the first conversion rate and the third first charge number are output.

[0123] The embodiment of the present invention provides a process for obtaining the conversion rate and first-time recharge number output by the random forest model. The test order is input into the training model, and the order data is predicted through the trained model and the corresponding model judgment structure. When the conversion rate is met, the first-time recharge number is predicted through the random forest model of this heterogeneous network. In this process, sufficient feature vectors are used for prediction and calculation, and the corresponding results are output to facilitate subsequent user quality evaluation.

[0124] Figure 6 A diagram of an order quality evaluation model training device provided by an embodiment of the present invention, such as Figure 6 As shown, the device includes: an order data initialization module 601, a sample set acquisition module 602 and a random forest model establishment and training module 603.

[0125] The order data initialization module 601 is used to obtain an initial sample set based on the mobile Internet card order data, and the initial sample set includes the user's original feature vector, conversion rate and first recharge amount. The conversion rate is used to indicate the probability of the user placing an order to access the network, and the first recharge amount is the first recharge amount of the user who has accessed the network. The sample set acquisition module 602 is used to obtain a mixed sample set, a sample set of users from other networks and a sample set of users from this network based on the basic feature vector in the initial sample set; the basic feature vector is a feature vector shared by the user samples of this network and the user samples of other networks.

[0126] In a possible implementation, the sample set acquisition module 602 is further used to select basic feature vectors and conversion rates in the initial sample set to form the mixed sample set;

[0127] Filter out samples that only include data of users from different networks in the initial sample set, and select the basic feature vector and the first charge number to form the sample set of users from different networks;

[0128] Filter out samples that only include local network user data in the initial sample set, and select the basic feature vector, the local network feature vector and the first recharge number to form the local network user sample set;

[0129] The basic feature vector includes: operating system, medium information, information streaming media, user information, number card information and order information;

[0130] The network feature vector includes: user network access time, user consumption data, pre-stored data and arrears data.

[0131] The random forest model establishment and training module 603 is used to construct and train a random forest model based on the mixed sample set, the cross-network user sample set and the local network user sample set to obtain a trained random forest model. The random forest model is used to output the conversion rate and the number of first recharges based on the user's feature vector. The conversion rate and the number of first recharges are used to evaluate the order quality.

[0132] In a possible implementation, the random forest model building and training module 603 is further used to:

[0133] Construct hybrid random forest models, cross-network random forest models, and local network random forest models;

[0134] The construction and training processes of the hybrid random forest model, the cross-network random forest model, and the local network random forest model are the same;

[0135] Among them, the training data of the mixed random forest model is the mixed sample set, and the conversion rate is used as the output for training; the training data of the cross-network random forest model is the cross-network user sample set, and the first recharge number is used as the output for training; the sample data of the local network random forest model is the local network user sample set, and the first recharge number is used as the output for training.

[0136] In a possible implementation, the random forest model building and training module 603 is further used to:

[0137] S1-1, randomly taking out N samples with replacement from the mixed sample set, repeating n times, to obtain n training sample sets;

[0138] S1-2, based on the n training sample sets, respectively training n regression decision tree models;

[0139] S1-3, select a regression decision tree model and a constant m, where the constant m is an integer less than the total number of feature vector types in the mixed sample set;

[0140] S1-4, randomly selecting m types of feature vectors from the total feature vectors of the mixed sample set as feature subsets;

[0141] S1-5, traverse all feature vectors in the feature subset, perform binary segmentation on each feature vector, and calculate the mean of the target data corresponding to the two groups of data after each feature vector is split and the mean square error of the target data;

[0142] S1-6, comparing the mean square errors of all target data, selecting two arrays with the smallest mean square errors of target data, and obtaining two arrays for the first binary split; the data in the arrays include the feature vector type, the number of samples, the mean of the target data, and the mean square error of the target data;

[0143] S1-7, return to step S1-5, and repeat the traversal until the mean square error of the target data corresponding to the leaf node of the regression decision tree split is 0;

[0144] S1-8, combine all the split regression decision trees into an initial target random forest model, calculate the out-of-bag error rate of the initial target random forest model, adjust the size of the feature vector species m according to the out-of-bag error rate, return to step S1-3 and repeat the training until the out-of-bag error rate is less than or equal to the set threshold or the target random forest model in training has overfitting phenomenon, and obtain the trained target random forest model;

[0145] Among them, the target random forest model is one of a mixed random forest model, a different network random forest model, and a local random forest model. The target data of the mixed random forest model is the conversion rate, and the target data of the different network random forest model and the local random forest model are the first charge number.

[0146] Figure 7 A diagram of an order quality evaluation device provided by an embodiment of the present invention, such as Figure 7 As shown, the device includes: a test order feature vector extraction module 701, an order quality evaluation data acquisition module 702 and an order quality evaluation module 703.

[0147] The test order feature vector extraction module 701 is used to extract user feature vectors.

[0148] The order quality evaluation data acquisition module 702 is used to input the user feature vector into the random forest model to obtain the conversion rate and first recharge number output by the random forest model.

[0149] In a possible implementation, the order quality evaluation data acquisition module 702 is further used to input the user feature vector into a hybrid random forest model to obtain a first conversion rate output by the hybrid random forest model;

[0150] Determine whether the first conversion rate is less than the conversion threshold. If so, determine that the first recharge amount is equal to 0, and output the first conversion rate and the first first recharge amount. If not, determine whether the user is a user of this network according to the number card of the test order.

[0151] If the user is a user of this network, the user feature vector is input into the random forest model of this network to obtain the second filler;

[0152] If the user is not a user of this network, the user feature vector is input into the random forest model of the other network to obtain the third filler.

[0153] The order quality evaluation module 703 is used to evaluate the quality of the test order according to the conversion rate and the first recharge amount.

[0154] The present application also provides an order quality evaluation device, comprising: at least one processor and a memory;

[0155] The memory stores computer-executable instructions;

[0156] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the order quality evaluation model training method or the order quality evaluation method.

[0157] Figure 8 The hardware diagram of the order quality evaluation device provided by the embodiment of the present invention is shown in FIG. Figure 8 As shown, the model training device 80 based on virtual samples provided in this embodiment includes: at least one processor 801 and a memory 802. The device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0158] In the specific implementation process, at least one processor 801 executes the computer execution instructions stored in the memory 802, so that at least one processor 801 executes the above-mentioned model training method based on virtual samples.

[0159] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0160] In the above Figure 8In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0161] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0162] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0163] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the order quality evaluation model training method and order quality evaluation method as described above are implemented.

[0164] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0165] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0166] The division of the units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0169] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0170] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0171] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for training an order quality evaluation model, characterized in that: include: According to the mobile Internet number card order data, an initial sample set is obtained, wherein the initial sample set includes the user's original feature vector, conversion rate and first recharge amount, wherein the conversion rate is used to indicate the probability of a user placing an order to access the network, and the first recharge amount is the first recharge amount of a user who has accessed the network; According to the basic feature vector in the initial sample set, a mixed sample set, a sample set of users from different networks and a sample set of users from the local network are obtained; the basic feature vector is a feature vector shared by the user samples from the local network and the user samples from different networks; A random forest model is constructed and trained according to the mixed sample set, the sample set of users from different networks, and the sample set of users from the local network to obtain a trained random forest model, wherein the random forest model is used to output a conversion rate and a first recharge number according to a feature vector of the user, and the conversion rate and the first recharge number are used to evaluate order quality; The step of obtaining a mixed sample set, a sample set of users from different networks, and a sample set of users from the local network according to the basic feature vectors in the initial sample set includes: Selecting basic feature vectors and conversion rates in the initial sample set to form the mixed sample set; Filter out samples that only include data of users from different networks in the initial sample set, and select the basic feature vector and the first charge number to form the sample set of users from different networks; Filter out samples that only include local network user data in the initial sample set, and select the basic feature vector, the local network feature vector and the first recharge number to form the local network user sample set; The random forest model includes: a hybrid random forest model, a different network random forest model, and a local network random forest model; The construction and training process of the random forest model includes: S1-1, randomly taking out N samples with replacement from the mixed sample set, repeating n times, to obtain n training sample sets; S1-2, based on the n training sample sets, respectively training n regression decision tree models; S1-3, select a regression decision tree model and a constant m, where the constant m is an integer less than the total number of feature vector types in the mixed sample set; S1-4, randomly selecting m types of feature vectors from the total feature vectors of the mixed sample set as feature subsets; S1-5, traverse all feature vectors in the feature subset, perform binary segmentation on each feature vector, and calculate the mean of the target data corresponding to the two groups of data after each feature vector is split and the mean square error of the target data; S1-6, comparing the mean square errors of all target data, selecting two arrays with the smallest mean square errors of target data, and obtaining two arrays for the first binary split; the data in the arrays include the feature vector type, the number of samples, the mean of the target data, and the mean square error of the target data; S1-7, return to step S1-5, and repeat the traversal until the mean square error of the target data corresponding to the leaf node of the regression decision tree split is 0; S1-8, combine all the split regression decision trees into an initial random forest model, calculate the out-of-bag error rate of the initial random forest model, adjust the size of the feature vector species m according to the out-of-bag error rate, return to step S1-3 and repeat the training until the out-of-bag error rate is less than or equal to the set threshold or the random forest model in training is overfitted, and obtain a trained random forest model; Among them, the random forest model is one of a mixed random forest model, a different network random forest model, and a local random forest model. The target data of the mixed random forest model is the conversion rate, and the target data of the different network random forest model and the local random forest model are the first charge number.

2. The method according to claim 1, characterized in that The basic feature vector includes: operating system, medium information, information streaming media, user information, number card information and order information; The network feature vector includes: user network access time, user consumption data, pre-stored data and arrears data.

3. The method according to claim 1, characterized in that The construction and training processes of the hybrid random forest model, the cross-network random forest model, and the local network random forest model are the same; Among them, the training data of the mixed random forest model is the mixed sample set, and the conversion rate is used as the output for training; the training data of the cross-network random forest model is the cross-network user sample set, and the first recharge number is used as the output for training; the sample data of the local network random forest model is the local network user sample set, and the first recharge number is used as the output for training.

4. A method for evaluating order quality, characterized in that: The method comprises: Extract user feature vectors based on test orders; Inputting the user feature vector into a random forest model to obtain a conversion rate and a first-charge number output by the random forest model, wherein the random forest model is obtained by the method according to any one of claims 1 to 3; The quality of the test order is evaluated based on the conversion rate and the first charge amount.

5. The method according to claim 4, characterized in that The step of inputting the user feature vector into a random forest model to obtain a conversion rate and a first-time recharge number output by the random forest model includes: Inputting the user feature vector into a hybrid random forest model to obtain a first conversion rate output by the hybrid random forest model; Determine whether the first conversion rate is less than the conversion threshold. If so, determine that the first recharge amount is equal to 0, and output the first conversion rate and the first first recharge amount. If not, determine whether the user is a user of this network according to the number card of the test order. If the user is a user of this network, the user feature vector is input into the random forest model of this network to obtain the second filler; If the user is not a user of this network, the user feature vector is input into the random forest model of the other network to obtain the third filler.

6. An order quality evaluation model training device, characterized in that: include: An order data initialization module is used to obtain an initial sample set according to the mobile Internet number card order data, wherein the initial sample set includes the user's original feature vector, conversion rate and first recharge amount, wherein the conversion rate is used to indicate the probability of the user placing an order to access the network, and the first recharge amount is the first recharge amount of the user who has accessed the network; A sample set acquisition module, used to obtain a mixed sample set, a sample set of users from different networks, and a sample set of users from the local network according to a basic feature vector in the initial sample set; the basic feature vector is a feature vector shared by the user samples of the local network and the user samples of different networks; A random forest model building and training module, used to build and train a random forest model based on the mixed sample set, the cross-network user sample set and the local network user sample set to obtain a trained random forest model, wherein the random forest model is used to output a conversion rate and a first recharge number based on a user's feature vector, and the conversion rate and the first recharge number are used to evaluate order quality; The sample set acquisition module is further used to select the basic feature vector and conversion rate in the initial sample set to form the mixed sample set; filter out the samples in the initial sample set that only include the data of users from other networks, and select the basic feature vector and the number of first recharges to form the sample set of users from other networks; filter out the samples in the initial sample set that only include the data of users from this network, and select the basic feature vector, the feature vector of this network and the number of first recharges to form the sample set of users from this network; The random forest model establishment and training module is also used to construct a hybrid random forest model, a different network random forest model, and a local network random forest model; The random forest model establishment and training module is also used to: S1-1, randomly taking out N samples with replacement from the mixed sample set, repeating n times, to obtain n training sample sets; S1-2, based on the n training sample sets, respectively training n regression decision tree models; S1-3, select a regression decision tree model and a constant m, where the constant m is an integer less than the total number of feature vector types in the mixed sample set; S1-4, randomly selecting m types of feature vectors from the total feature vectors of the mixed sample set as feature subsets; S1-5, traverse all feature vectors in the feature subset, perform binary segmentation on each feature vector, and calculate the mean of the target data corresponding to the two groups of data after each feature vector is split and the mean square error of the target data; S1-6, comparing the mean square errors of all target data, selecting two arrays with the smallest mean square errors of target data, and obtaining two arrays for the first binary split; the data in the arrays include the feature vector type, the number of samples, the mean of the target data, and the mean square error of the target data; S1-7, return to step S1-5, and repeat the traversal until the mean square error of the target data corresponding to the leaf node of the regression decision tree split is 0; S1-8, combine all the split regression decision trees into an initial random forest model, calculate the out-of-bag error rate of the initial random forest model, adjust the size of the feature vector species m according to the out-of-bag error rate, return to step S1-3 and repeat the training until the out-of-bag error rate is less than or equal to the set threshold or the random forest model in training is overfitted, and obtain a trained random forest model; Among them, the random forest model is one of a mixed random forest model, a different network random forest model, and a local random forest model. The target data of the mixed random forest model is the conversion rate, and the target data of the different network random forest model and the local random forest model are the first charge number.

7. An order quality evaluation device, characterized in that: include: Test order feature vector extraction module, used to extract user feature vector; An order quality evaluation data acquisition module, used for inputting the user feature vector into a random forest model to obtain a conversion rate and a first-time recharge number output by the random forest model, wherein the random forest model is obtained by the method according to any one of claims 1 to 3; An order quality evaluation module is used to evaluate the quality of the test order according to the conversion rate and the first recharge amount.

8. An order quality evaluation device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the order quality evaluation model training method as described in any one of claims 1-3 or the order quality evaluation method as described in any one of claims 4-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the order quality evaluation model training method and the evaluation method as described in any one of claims 1 to 3 or claims 4 to 5 are implemented.

Images