Operator package recommendation method and device

Abstract

The invention discloses an operator package recommendation method and device. The method comprises the following steps: acquiring a first operator package ordering behavior sequence of a first user in a first time period; analyzing the first operator package ordering behavior sequence by using a pre-trained operator package recommendation model to obtain a target operator package recommended to the first user at the next moment of the first time period, the operator package recommendation model is obtained by training a plurality of target training sample data subsets of which the contribution degree is higher than a preset threshold value and the sample quantity is not higher than a preset quantity threshold value, and the contribution degree is determined by utilizing a Sharpley value method. The technical problem that the accuracy of the recommendation result output by the model is poor due to the fact that the performance stability of the operator package recommendation model trained by the training sample data set selected through a random sampling method is poor is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, in particular to an operator package recommendation method and device. BACKGROUND

[0002] In the field of communication operators, package recommendation model is an important application based on big data and machine learning technology, aiming to predict and recommend the most suitable package options for users according to their historical consumption behavior, preferences, and market activities and other factors. However, with the development of enterprises, the number of users and the amount of business handling are growing, and the amount of data stored in the underlying layer is growing exponentially. When using machine learning algorithms to analyze and mine target user groups, the following difficulties and pain points will inevitably be encountered:

[0003] (1) The amount of data stored by current large enterprises has grown from the early GB level to TB, PB level, and the number of data records has also expanded from the early ten thousand records to one hundred million, ten billion. The increase in data volume also increases the demand for model computing resources during mining, which puts a lot of pressure on the supply of enterprise computing resources.

[0004] (2) When encountering some business scenarios with a large proportion of positive and negative sample data (the proportion has exceeded 1:100), the model training data set selected by random sampling methods such as oversampling and undersampling is prone to fluctuations in model training performance, which weakens the generalization ability of the model when facing new data, ultimately affecting the accuracy of the prediction results of the model output. In the operator package recommendation scenario, if the model recommended package does not meet the actual needs of the user, not only will it reduce user experience and satisfaction, but also may lead to increased operating costs and decreased market competitiveness.

[0005] Currently, there is no effective solution to the above problems. SUMMARY

[0006] The embodiments of the present application provide an operator package recommendation method and device to at least solve the technical problem that the performance stability of the operator package recommendation model trained by the training sample data set selected by the random sampling method is poor, resulting in poor accuracy of the recommendation results output by the model.

[0007] According to one aspect of an embodiment of the present application, a method for recommending an operator package is provided, including: obtaining a first operator package subscription behavior sequence of a first user within a first time period; analyzing the first operator package subscription behavior sequence using a pre-trained operator package recommendation model to obtain a target operator package recommended to the first user at a next moment in the first time period, wherein the operator package recommendation model is trained using multiple target training sample data subsets whose contribution is higher than a preset threshold value and whose sample quantity is no higher than a preset quantity threshold, and the contribution is determined using a Shapley value method.

[0008] Optionally, the training process of the operator package recommendation model includes: obtaining an initial training sample data set, wherein the initial training sample data set includes multiple groups of training sample data, and each group of training sample data includes: a second operator package subscription behavior sequence of the second user in a third time period and the operator package subscription of the second user at the next moment of the third time period; dividing the initial training sample data set according to a preset division rule to obtain multiple training sample data subsets, and randomly combining the multiple training sample data subsets to obtain multiple first data set alliances, wherein the division rule includes one of the following: dividing the sample data set based on features, dividing the sample data set based on model prediction results, and dividing the sample data set based on sample usage scenarios; determining a preset machine learning model; using multiple first The dataset alliance conducts multiple iterative training on the machine learning model, evaluates the model performance indicators of the machine learning model after each training, and uses the Shapley value method to analyze the model performance indicators corresponding to each first dataset alliance to obtain the contribution of each training sample data subset; for each training sample data subset, determine whether the contribution of the current training sample data subset is higher than the preset threshold value, and determine whether the number of samples in the current training sample data subset is higher than the preset quantity threshold; when the contribution of the current training sample data subset is higher than the preset threshold value and the number of samples is not higher than the quantity threshold, use the current training sample data subset as the target training sample data subset; use each target training sample data subset to iteratively train the machine learning model to obtain an operator package recommendation model.

[0009] Optionally, the machine learning model is iteratively trained multiple times using multiple first data set alliances, and the model performance indicators of the machine learning model after each training are evaluated, including: for each first data set alliance, the machine learning model is iteratively trained using the first data set alliance, and the trained machine learning model is model tested using a preset test sample data set to obtain the model performance indicators of the trained machine learning model, wherein the model performance indicators include at least one of the following: recall rate, accuracy rate, area under the ROC curve, and area under the PR curve.

[0010] Optionally, the contribution degree at least comprises a marginal contribution degree, wherein the contribution degree of each training sample data subset is determined according to the model performance indicators corresponding to each first data set alliance, including: for each training sample data subset, determining a plurality of first target data set alliances at least comprising the current training sample data subset; traversing each first target data set alliance, determining the number of members in the current first target data set alliance, and determining the model performance indicator after the current first target data set alliance removes the current training sample data subset and trains the machine learning model; calculating a first factorial value of a first difference value obtained by subtracting one from the number of members, calculating a second factorial value of a second difference value obtained by subtracting the number of members from the set number of the plurality of training sample data subsets, and calculating a product value of the first factorial value and the second factorial value divided by a fourth factorial value of the set number of the plurality of training sample data subsets to obtain a quotient value; multiplying the quotient value by the difference between the model performance indicator corresponding to the current first target data set alliance and the model performance indicator corresponding to the current first target data set alliance after removing the current training sample data subset to obtain the contribution degree of the current training sample data subset in the current first target data set alliance; and summing the contribution degrees of the current training sample data subset in each first target data set alliance to obtain the marginal contribution degree of the current training sample data subset.

[0011] Optionally, the contribution degree further comprises a comprehensive contribution degree, wherein the contribution degree of each training sample data subset is determined according to the model performance indicators corresponding to each first data set alliance, and further comprising: for each training sample data subset, multiplying the marginal contribution degree of the current training sample data subset by the model performance indicator corresponding to the first target data set alliance containing only the current training sample data subset to obtain the comprehensive contribution degree of the current training sample data subset.

[0012] Optionally, the method further comprises: in the case that the contribution degree of the current training sample data subset is higher than a preset threshold value and the sample number is higher than a number threshold value, dividing the current training sample data subset according to the division rule, and randomly combining the plurality of training sample data subsets obtained by the division to obtain a plurality of second data set alliances.

[0013] Optionally, the method further comprises: in the case that the contribution degree of each training sample data subset is not higher than a preset threshold value, re-dividing the initial training sample data set according to the division rule, and randomly combining the plurality of new training sample data subsets obtained by the re-division to obtain a plurality of new first data set alliances.

[0014] According to another aspect of the embodiments of the present application, an operator package recommendation device is also provided, which comprises: an acquisition module configured to acquire a first operator package subscription behavior sequence of a first user in a first time period; and a recommendation module configured to analyze the first operator package subscription behavior sequence by using a pre-trained operator package recommendation model to obtain a target operator package to be recommended to the first user at a next time in the first time period, wherein the operator package recommendation model is trained by using a plurality of target training sample data subsets whose contribution degrees are higher than a preset threshold value and whose sample quantities are not higher than a preset quantity threshold value, and the contribution degrees are determined by using a Shapley value method.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, which comprises a computer program, wherein the computer program is executed by a processor to implement the operator package recommendation method described above.

[0016] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the operator package recommendation method described above by using the computer program.

[0017] In the embodiments of the present application, the first operator package subscription behavior sequence of the first user in the first time period is analyzed by using the pre-trained operator package recommendation model to obtain the target operator package to be recommended to the first user at the next time in the first time period, wherein the operator package recommendation model is trained by using the plurality of target training sample data subsets whose contribution degrees are higher than the preset threshold value and whose sample quantities are not higher than the preset quantity threshold value, and the contribution degrees are determined by using the Shapley value method. The technical effects of optimizing the performance of the operator package recommendation model and improving the recommendation accuracy are achieved, the purposes of reducing the consumption of computing resources, improving the generalization ability of the model, and ensuring the stability and effectiveness of the recommendation strategy are achieved, and the technical problems that the performance stability of the operator package recommendation model trained by using the training sample data set selected by using the random sampling method is poor, and the accuracy of the recommendation result output by the model is poor are solved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, illustrate the exemplary embodiments of the present application and their description serves to explain the present application, and do not limit the present application in any way. In the drawings:

[0019] Figure 1 is a flow diagram of an optional operator package recommendation method according to the embodiments of the present application;

[0020] Figure 2is a schematic diagram of an optional training process of an operator package recommendation model according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional division of a training sample data set according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an optional structure of an operator package recommendation device according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an optional structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, first, some nouns or terms appearing in the description of the embodiments of the present application are translated and explained as follows:

[0027] Shapley Value: is a concept in game theory, used to fairly distribute the contributions of members in cooperative games, which is based on the marginal contribution of members in all possible alliances, and calculates the contribution degree of each member by weighted summation.

[0028] Embodiment 1

[0029] According to the embodiment of the present application, a method for recommending an operator package is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0030] Figure 1 is a flowchart of a method for recommending an operator package according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0031] Step S102, obtaining a first operator package subscription behavior sequence of a first user in a first time period.

[0032] The first operator package subscription behavior sequence refers to all records of the first user's subscription interaction with an operator in a specific time range (i.e. the first time period). It includes but is not limited to the timestamp, operation type and detailed information related to the operation of the first user's query, comparison, subscription, cancellation, upgrade or downgrade of different operator packages, etc.

[0033] Step S104, using a pre-trained operator package recommendation model to analyze the first operator package subscription behavior sequence to obtain a target operator package recommended to the first user at the next moment in the first time period.

[0034] The operator package recommendation model refers to a machine learning model trained by a specific optimization method, which is used to predict and recommend the operator package that best meets the user's needs. The training process of the model depends on a plurality of target training sample data subsets with a contribution degree higher than a preset threshold and a sample number not higher than a preset number threshold. The contribution degree of the target training sample data subset is determined by the Shapley value method. Such target training sample data subsets have a significant positive impact on the final model performance, while also controlling the sample size required for model training, avoiding resource waste and excessive computation.

[0035] The steps of the method for recommending an operator package are described below in conjunction with a specific implementation process.

[0036] In the technical solution provided in step S102, from the enterprise database or the big data warehouse, the data table or data set containing the personal information and the package subscription record of the first user is located through the SQL query or the data extraction tool, the start date and the end date of the search are set according to the first time period, the time filtering condition is used to filter all the package subscription behavior data of the first user in the first time period from the located data table or data set, and the initial operator package subscription behavior sequence is obtained. Further, the initial operator package subscription behavior sequence is preprocessed, such as data cleaning (for removing invalid or incomplete records), missing value processing, and abnormal value processing; the preprocessed initial operator package subscription behavior sequence is arranged in chronological order, and the first operator package subscription behavior sequence of the first user in the first time period is constructed. Thus, high-quality data input is provided for subsequent personalized recommendation using the pre-trained operator package recommendation model.

[0037] Further, the first operator package subscription behavior sequence is input into the pre-trained operator package recommendation model, the prediction ability of the model is used to analyze and predict the possible package demand of the first user at the next moment in the first time period, so as to obtain the target operator package recommended to the first user at the next moment in the first time period output by the model. Compared with the traditional package recommendation scheme based on only static user information or historical package sales data, this recommendation scheme can capture the changing needs of users over time, especially in the digital age where user behavior patterns evolve rapidly, ensuring that the business system or the operator prepares in advance for package recommendation and customer retention, improving the timeliness of prediction, providing a forward-looking basis for operator decision-making, and helping to adjust the recommendation strategy in time to push personalized package recommendations to the first user, optimize user experience, and promote business growth.

[0038] The training process of the operator package recommendation model is shown in Figure 2 , which includes:

[0039] Step S1, obtaining an initial training sample data set. The initial training sample data set can be historical package subscription data extracted from the operator database. Therefore, the initial training sample data set includes multiple groups of training sample data, and each group of training sample data includes: a second operator package subscription behavior sequence of a second user in a third time period and a subscription operator package of the second user at the next moment in the third time period.

[0040] Step S2, dividing the initial training sample data set according to a preset division rule to obtain multiple training sample data subsets, and randomly combining the multiple training sample data subsets to obtain multiple first data set alliances.

[0041] Among them, the above-mentioned division rules can be to divide the sample data set based on features (such as division based on user type, operator product type), to divide the sample data set based on model prediction results (such as separating samples with accurate predictions from samples with inaccurate predictions), to divide the sample data set based on sample usage scenarios (such as separating samples related to broadband plus operator packages and 5G operator packages), etc., or it can be random division (that is, dividing the initial training sample data set into several non-overlapping training sample data subsets through random sampling).

[0042] For example, Figure 3 is a schematic diagram of an optional division of a training sample data set according to an embodiment of the present application, such as Figure 3 The training sample dataset is a dataset of 1 million records with 200 field features. Figure 3 The three extraction mechanisms shown are: horizontal analysis (i.e., the data records remain unchanged, and the field features are extracted and split), vertical analysis (i.e., the field features remain unchanged, and the data records are extracted and split), and slice analysis (i.e., both field features and data records are extracted and split), so as to divide the training sample data set into three data subsets A, B, and C.

[0043] Step S3: Determine a preset machine learning model. The machine learning model can be any model suitable for processing sequence data and classification tasks, including but not limited to: deep learning models (such as recurrent neural networks (RNNs) and long short-term memory (LSTMs), ensemble learning models (such as random forests and gradient boosting trees), or other supervised learning classifiers (such as logistic regression and support vector machines (SVMs)).

[0044] In step S4, the machine learning model is iteratively trained multiple times using multiple first data set alliances, and the model performance indicators of the machine learning model after each training are evaluated. The model performance indicators corresponding to each first data set alliance are analyzed using the Shapley value method to obtain the contribution of each training sample data subset.

[0045] Optionally, in the technical solution provided in the above step S4, for each dataset alliance, the machine learning model is iteratively trained using the dataset alliance, and the trained machine learning model is tested using a preset test sample dataset to obtain the model performance indicators of the trained machine learning model.

[0046] That is, each dataset union is traversed, and each training sample data in the first dataset union being currently traversed is input into the machine learning model, and the machine learning model adjusts its internal parameters by learning the training sample data in the dataset union to achieve the prediction target. After completing the iterative training, the trained machine learning model is tested using a preset test sample data set, and the model performance index of the model is evaluated by calculating the difference between the prediction result and the actual result of the trained machine learning model on the test sample data set. Commonly used model performance indexes include:

[0047] (1) Recall: also known as sensitivity or true positive rate (TPR), indicating the proportion of positive samples correctly identified by the model, and the calculation formula is: Recall = true positive / (true positive + false negative).

[0048] (2) Precision: indicating the proportion of positive samples actually positive in the samples predicted as positive by the model. The calculation formula is: Precision = true positive / (true positive + false negative).

[0049] (3) Area under the ROC curve: the ROC curve represents the relationship between the true positive rate and the false positive rate, and the area under the curve (AUC) is an index for measuring the performance of the classifier. The value of the area under the curve is between 0.5 and 1, and the closer to 1 indicates that the model can effectively separate positive and negative samples; 0.5 indicates that the model has no distinguishing ability.

[0050] (4) Area under the PR curve: the PR curve represents the relationship between the accuracy and the recall, and the area under the curve (AUC-PR) reflects the performance of the classifier at different thresholds. AUC-PR pays more attention to the case where the number of positive samples is less than the number of negative samples, and the higher the value indicates that the model has a stronger recognition ability for positive samples, especially on unbalanced data sets. AUC-PR is more meaningful than AUC-ROC.

[0051] (5) F1 Score: is the harmonic mean of precision and recall. Therefore, the value of F1 Score is between 0 and 1, and 1 indicates that the precision and recall of the model are both high.

[0052] Further, in the technical solution provided in step S4, if the contribution degree is marginal contribution degree, then for each training sample data subset, the contribution degree of the training sample data subset can be determined by the following steps:

[0053] First step: determine a plurality of first target data set alliances including at least a current training sample data subset;

[0054] Second step: traverse each first target data set alliance and perform the following steps: first, determine the number of members in the current first target data set alliance and determine the model performance index after training the current first target data set alliance after removing the current training sample data subset; then calculate the first factorial value of the first difference obtained by subtracting one from the number of members, calculate the second factorial value of the second difference obtained by subtracting the number of members from the set number of the plurality of training sample data subsets, and calculate the product value of the first factorial value and the second factorial value divided by the fourth factorial value of the set number of the plurality of training sample data subsets, multiply the quotient value by the difference between the model performance index corresponding to the current first target data set alliance and the model performance index corresponding to the current first target data set alliance after removing the current training sample data subset, to obtain the contribution degree of the current training sample data subset in the current first target data set alliance.

[0055] Third step: sum the contribution degrees of the current training sample data subset in each first target data set alliance to obtain the marginal contribution degree of the current training sample data subset.

[0056] For example, if there are three training sample data subsets, denoted as A, B, and C, and the A, B, and C training sample data subsets are sequentially sent into and merged into a machine learning model for training and the model performance index after training is evaluated, the following results are obtained: the model performance index F1 after the A training sample data subset is sent into the model alone is 0.62; the model performance index F1 after the B training sample data subset is sent into the model alone is 0.58; the model performance index F1 after the C training sample data subset is sent into the model alone is 0.47; the model performance index F1 after the A training sample data subset and the B training sample data subset are merged into the model is 0.73; the model performance index F1 after the A training sample data subset and the C training sample data subset are merged into the model is 0.69; the model performance index F1 after the B training sample data subset and the C training sample data subset are merged into the model is 0.79; and the model performance index F1 after the A training sample data subset, the B training sample data subset, and the C training sample data subset are merged into the model is 0.84. Based on this, the marginal contribution degrees of the A training sample data subset, the B training sample data subset, and the C training sample data subset can be calculated according to Tables 1-3 as follows.

[0057] Table 1

[0058]

[0059] Therefore, the marginal contribution value of the A training sample data subset is:

[0060] Table 2

[0061]

[0062] Therefore, the marginal contribution value of the B training sample data subset is:

[0063] Table 3

[0064]

[0065] Therefore, the marginal contribution value of the C training sample data subset is:

[0066] In addition, if the contribution degree is the comprehensive contribution degree, then for each training sample data subset, the contribution degree of the training sample data subset can be determined by the following method, including: multiplying the marginal contribution degree of the current training sample data subset by the model performance index corresponding to the first target data set alliance only containing the current training sample data subset, to obtain the comprehensive contribution degree of the current training sample data subset.

[0067] For example, the marginal contribution value 0.284 of the A training sample data subset calculated above is multiplied by the model performance index 0.62 corresponding to the target data set alliance only containing the A training sample data subset, to obtain the comprehensive contribution degree of the A training sample data subset as 0.17608; the marginal contribution value 0.314 of the B training sample data subset calculated above is multiplied by the model performance index 0.58 corresponding to the target data set alliance only containing the B training sample data subset, to obtain the comprehensive contribution degree of the B training sample data subset as 0.18212; the marginal contribution value 0.239 of the B training sample data subset calculated above is multiplied by the model performance index 0.47 corresponding to the target data set alliance only containing the C training sample data subset, to obtain the comprehensive contribution degree of the C training sample data subset as 0.11233.

[0068] In step S5, for each training sample data subset, it is judged whether the contribution degree of the current training sample data subset is higher than a preset threshold value, and whether the sample quantity of the current training sample data subset is higher than a preset quantity threshold value. The judgment of the contribution degree is to ensure the quality of the model training sample data, and to ensure that only those training sample data subsets that have a significant improvement on the model performance are selected for model training; and the judgment of the sample quantity is to control the complexity and calculation cost of the model training, and to ensure the efficient operation of the model.

[0069] Step S6: When the contribution of the current training sample data subset is higher than the preset threshold value and the number of samples is not higher than the number threshold value, the current training sample data subset is used as the target training sample data subset.

[0070] Step S7: Iteratively train the machine learning model using each target training sample data subset to obtain an operator package recommendation model.

[0071] In addition, when the contribution of the current training sample data subset is higher than the preset threshold value and the number of samples is higher than the quantity threshold, this indicates that the current training sample data subset itself may contain too much training sample data, resulting in excessive computational cost or overly complex model. Therefore, the embodiment of the present application proposes that the current training sample data subset can be divided according to the division rule, and the multiple training sample data subsets obtained by the division are randomly combined to obtain multiple second data set alliances; then, the machine learning model is iteratively trained multiple times using the multiple second data set alliances, and the model performance indicators of the machine learning model after each training are evaluated, and the model performance indicators corresponding to each second data set alliance are analyzed using the Shapley value method to obtain the contribution of each new training sample data subset; finally, a new training sample data subset with a contribution higher than the preset threshold value and a sample number not higher than the preset quantity threshold is selected from the multiple new training sample data subsets to train the machine learning model to obtain an operator package recommendation model.

[0072] If the contribution degrees of the multiple subsets of training sample data are all not higher than the preset threshold value, it indicates that the multiple subsets of training sample data currently divided fail to effectively improve the model performance or have relatively small contribution to the model, and are insufficient to serve as high-quality data input. In this case, the embodiments of the present application can re-divide the initial training sample data set according to the division rule, and randomly combine the multiple new subsets of training sample data obtained by re-division to obtain multiple new first data set alliances. In the current division process, different division rules or division strategies can be used to attempt different configurations of subsets of training sample data to find data set combinations with greater influence and contribution degree. Then, the multiple new first data set alliances are used to perform multiple iterations of training on the machine learning model, and the model performance indicators of the machine learning model after each training are evaluated. The model performance indicators corresponding to each new first data set alliance are analyzed by using the Shannon value method to obtain the contribution degrees of each new subset of training sample data. It is determined whether the contribution degrees of each new subset of training sample data are higher than the preset threshold value and whether the sample quantity is higher than the preset quantity threshold value. If not, the division operation and the recombination operation are performed in a loop until a new subset of training sample data with a contribution degree higher than the preset threshold value and a sample quantity not higher than the preset quantity threshold value is obtained, and the new subset of training sample data satisfying the above conditions is used to train the machine learning model to obtain the operator package recommendation model.

[0073] In the above model training process, the model performance indicators are used as a guide to objectively evaluate the performance of the model under different data set alliance training, and the contribution degrees of different data subsets are quantitatively evaluated by combining the Shannon value theory, so that the selection of the training sample data set is no longer random sampling, but a scientific contribution degree value is used to finely select data subsets that significantly improve the model performance for training, which avoids the blind use of a large amount of data, reduces the waste of computing resources, ensures the rationality and accuracy of the model training process, and enhances the generalization ability of the model.

[0074] It should be noted that machine learning is a key technology in the field of artificial intelligence, and it plays an increasingly important role in multiple industries. Therefore, the above model training process can be applied not only to the operator package recommendation model protected by the embodiments of the present application, but also to the training process of any machine learning model with a large training sample data set. For example:

[0075] (1) Credit risk assessment models, stock price prediction models, etc. in the field of financial services. Among them, the training sample data of the credit scoring card model includes: the personal credit information of the borrower in the historical time period (including the credit history of the borrower, such as past loan records, credit card repayment, overdue payment records, etc.), financial status (including the borrower's income level, debt ratio, cash flow, job stability and professional nature, etc.), credit score (such as credit score or rating provided by external credit evaluation agencies), loan details (including loan amount, term, interest rate, guarantee method, etc.), behavior data (such as the borrower's online bank usage frequency, telephone payment behavior, social media activity, etc.) as training sample data, and the borrower's credit risk level / score as the corresponding sample label. Therefore, the credit risk assessment model helps banks and financial institutions to quantify the future repayment ability and default risk of the applicant when processing user loan applications, and to decide whether to issue loans and the conditions of the loans, such as interest rate, repayment period, and limit. The stock price prediction model's training sample data includes: the stock price sequence of the stock in the historical time period (including opening price, closing price, highest price, lowest price, and trading volume, etc.) as training sample data, and the predicted stock price of the next timestamp of the stock in the historical time period as the corresponding sample label. Therefore, the stock price prediction model can predict the stock price trend to provide accurate investment advice to customers.

[0076] (2) Product recommendation model in retail industry. Among them, the training sample data of the product recommendation model includes: user behavior data (including user browsing, clicking, purchasing, adding to shopping cart, collecting, search records, etc.), user attribute information (such as age, gender, occupation, etc.), social network data (user activity on social media, friend relationship, like and share of goods, etc.) of the user in the historical time period as training sample data, and the goods purchased by the user in the next timestamp in the historical time period as the corresponding sample label. Therefore, the product recommendation model can recommend goods that the user may be interested in in real time, improve user experience and sales conversion rate.

[0077] (3) Network attack detection model, abnormal traffic identification model, etc. in the field of network security. Among them, the training sample data of the network attack detection model includes: network traffic data of the network in the historical time period as training sample data, and the corresponding type as label, including: normal network traffic data and abnormal network traffic data. Therefore, the network attack detection model can be deployed at the network boundary or data center to detect and warn abnormal network traffic data in the network in real time, providing timely response opportunities for administrators to prevent data from being leaked or damaged.

[0078] (4) Traffic flow prediction model in the field of smart city management. The training sample data of this traffic flow prediction model includes: historical traffic flow data of a certain area in a historical time period (including the number, type, speed, direction and timestamp of vehicles on the road section), time series data (including time of day, day of the week, season of the year, etc.), weather information (such as temperature, humidity, wind speed, visibility, rainfall, etc.), road infrastructure information (such as road type, road condition, number of intersections, road capacity and length, etc.) as training samples, and the traffic flow of a certain area at the next moment in the historical time period as the corresponding sample label. Therefore, this traffic flow prediction model can predict the vehicle flow of the road in real time, thereby scheduling signal light control and optimizing traffic flow.

[0079] The several model training scenarios listed above are only provided as examples and are not specifically limited in the embodiments of the present application.

[0080] Example 2

[0081] According to an embodiment of the present application, there is also provided an operator package recommendation device for implementing the operator package recommendation method in embodiment 1, such as Figure 4 As shown, the operator package recommendation device at least includes: an acquisition module 42 and a recommendation module 44, wherein:

[0082] An acquisition module 42 is configured to acquire a sequence of subscription behaviors of a first user to a first operator's package within a first time period;

[0083] The recommendation module 44 is configured to analyze the first operator package subscription behavior sequence using a pre-trained operator package recommendation model to obtain a target operator package recommended to the first user at the next moment in the first time period, wherein the operator package recommendation model is trained using a plurality of target training sample data subsets whose contribution is higher than a preset threshold value and whose number of samples is not higher than a preset number threshold value, and the contribution is determined using the Shapley value method.

[0084] The functions of each module of the operator package recommendation device are described below in conjunction with a specific implementation process.

[0085] Acquisition module 42 can locate a data table or dataset containing the first user's personal information and package subscription records from an enterprise database or big data warehouse using SQL queries or data extraction tools. It then sets a search start and end date based on a first time period and uses a time filter to filter all package subscription behavior data of the first user during the first time period from the located data table or dataset to obtain an initial carrier package subscription behavior sequence. Furthermore, preprocessing operations are performed on the initial carrier package subscription behavior sequence data, such as data cleaning, missing value processing, and outlier processing. The preprocessed initial carrier package subscription behavior sequence is then arranged in chronological order to construct a first carrier package subscription behavior sequence of the first user during the first time period. This provides high-quality data input for subsequent personalized recommendations using a pre-trained carrier package recommendation model.

[0086] Furthermore, recommendation module 44 invokes a pre-trained carrier package recommendation model, inputs the obtained first carrier package subscription behavior sequence into the carrier package recommendation model, and uses the model's predictive capabilities to analyze and predict the first user's likely package needs at the next moment in the first time period, thereby outputting a target carrier package recommended to the first user at the next moment in the first time period. The resulting target carrier package is then fed back to the business system or operations personnel to formulate a recommendation strategy or directly push personalized package recommendations to the first user, thereby optimizing user experience and promoting business growth.

[0087] As an optional implementation, the operator package recommendation device further includes a model training module, and the model training module is used to train the operator package recommendation model according to the following steps, including:

[0088] Step S1: Obtain an initial training sample data set, wherein the initial training sample data set includes multiple sets of training sample data, and each set of training sample data includes: a sequence of a second user's subscription behavior for a second operator package within a third time period, and the second user's subscription behavior for a second operator package at the next moment in the third time period.

[0089] Step S2: The initial training sample dataset is divided according to a preset division rule to obtain multiple training sample data subsets, and the multiple training sample data subsets are randomly combined to obtain multiple first dataset alliances. The division rule can be based on feature division, model prediction results division, sample dataset division based on sample usage scenarios, etc., or can be random division (i.e., dividing the initial training sample dataset into multiple non-overlapping training sample data subsets by random sampling).

[0090] Step S3: Determine the preset machine learning model.

[0091] Step S4, respectively, using multiple first data set alliances for multiple iterations of training machine learning model, and evaluate the model performance index of each trained machine learning model, using the shapley value method for each first data set alliance corresponding to the model performance index analysis, get each training sample data subset of the contribution degree.

[0092] Optionally, in the technical scheme provided in the above step S4, for each data set alliance, the model training module can use the data set alliance to iteratively train the machine learning model, and use the preset test sample data set to test the trained machine learning model, and obtain the model performance index of the trained machine learning model.

[0093] Further, if the contribution degree is marginal contribution degree, then for each training sample data subset, the model training module can determine the contribution degree of the training sample data subset by the following steps, including:

[0094] First step: determine at least a plurality of first target data set alliances including the current training sample data subset;

[0095] Second step: traverse each first target data set alliance, and execute the following steps: first, determine the number of members in the current first target data set alliance, and determine the model performance index of the current first target data set alliance after removing the current training sample data subset; Then calculate the first factorial value of the first difference obtained by subtracting one from the number of members, calculate the second factorial value of the second difference obtained by subtracting the number of members from the set number of the plurality of training sample data subsets, and calculate the product value of the first factorial value and the second factorial value divided by the fourth factorial value of the set number of the plurality of training sample data subsets. The quotient value, multiply the difference between the model performance index corresponding to the current first target data set alliance and the model performance index corresponding to the current first target data set alliance after removing the current training sample data subset, to obtain the contribution degree of the current training sample data subset in the current first target data set alliance.

[0096] Third step: sum the contribution degrees of the current training sample data subset in each first target data set alliance to obtain the marginal contribution degree of the current training sample data subset.

[0097] In addition, if the contribution degree is the comprehensive contribution degree, then for each training sample data subset, the model training module can determine the contribution degree of the training sample data subset by the following method, including: multiplying the marginal contribution degree of the current training sample data subset by the model performance index corresponding to the first target data set alliance containing only the current training sample data subset, to obtain the comprehensive contribution degree of the current training sample data subset.

[0098] Step S5, for each training sample data subset, it is judged whether the contribution degree of the current training sample data subset is higher than a preset threshold value, and whether the sample quantity of the current training sample data subset is higher than a preset quantity threshold value.

[0099] Step S6, in the case that the contribution degree of the current training sample data subset is higher than the preset threshold value and the sample quantity is not higher than the quantity threshold value, the current training sample data subset is taken as a target training sample data subset.

[0100] Step S7, the machine learning model is iteratively trained by using each target training sample data subset, and an operator package recommendation model is obtained.

[0101] In addition, in the case that the contribution degree of the current training sample data subset is higher than the preset threshold value and the sample quantity is higher than the quantity threshold value, it is indicated that the current training sample data subset itself may contain too many training sample data, resulting in too high calculation cost or too complex model. Therefore, the model training module can divide the current training sample data subset according to the division rule, and randomly combine the multiple training sample data subsets obtained by the division to obtain multiple second data set alliances; then, the machine learning model is iteratively trained multiple times by using the multiple second data set alliances respectively, and the model performance indicators of the machine learning model after each training are evaluated, the model performance indicators corresponding to each second data set alliance are analyzed by using the Shapley value method, and the contribution degrees of each new training sample data subset are obtained; finally, the machine learning model is trained by selecting a new training sample data subset with a contribution degree higher than the preset threshold value and a sample quantity not higher than the preset quantity threshold value from the multiple new training sample data subsets, and an operator package recommendation model is obtained.

[0102] If the contribution degrees of the multiple subsets of training sample data are all not higher than the preset threshold value, it indicates that the multiple subsets of training sample data currently divided fail to effectively improve the model performance or have relatively small contribution, and are insufficient to serve as high-quality data input. In this case, the model training module can re-divide the initial training sample data set according to the division rule, and randomly combine the multiple new subsets of training sample data obtained by re-division to obtain multiple new first data set alliances. In the current division process, different division rules or division strategies can be used to attempt different configurations of subsets of training sample data, so as to find a data set combination with greater influence and contribution degree. Then, the multiple new first data set alliances are used to perform multiple iterative training on the machine learning model, respectively, and the model performance indicators of the machine learning model after each training are evaluated. The model performance indicators corresponding to the multiple new first data set alliances are analyzed by using the Shannon value method, to obtain the contribution degrees of the multiple new subsets of training sample data. It is determined whether the contribution degrees of the multiple new subsets of training sample data are higher than the preset threshold value, and whether the sample quantities are higher than the preset quantity threshold value. If not, the division operation and the recombination operation are repeatedly performed until a new subset of training sample data with a contribution degree higher than the preset threshold value and a sample quantity not higher than the preset quantity threshold value is obtained, and the new subset of training sample data satisfying the above conditions is used to train the machine learning model to obtain the operator package recommendation model.

[0103] In the above model training process, the model training module is oriented to the model performance indicators, objectively evaluates the performance of the model under different data set alliance training, and quantitatively evaluates the contribution degrees of different data subsets by using the Shannon value theory, so that the selection of the training sample data set is no longer random sampling, but fine selection of data subsets with significant improvement of the model performance based on scientific contribution degree values. This avoids blind use of a large amount of data, reduces waste of computing resources, ensures rationality and accuracy of the model training process, and enhances the generalization ability of the model.

[0104] It should be noted that the modules in the operator package recommendation device in the embodiments of the present application correspond one by one to the implementation steps of the operator package recommendation method in Embodiment 1. Since Embodiment 1 has been described in detail, the details not embodied in this embodiment can be referred to Embodiment 1, and will not be described in detail here.

[0105] Embodiment 3

[0106] According to the embodiments of the present application, a computer program product is also provided, which includes a computer program. When the computer program is executed by a processor, the operator package recommendation method in Embodiment 1 is implemented.

[0107] According to an embodiment of the present application, a non-transitory storage medium is also provided, which includes a stored computer program, wherein a device in which the non-transitory storage medium is located executes the operator package recommendation method in embodiment 1 by running the computer program.

[0108] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein the computer program executes the operator package recommendation method in embodiment 1 when running.

[0109] According to an embodiment of the present application, an electronic device is also provided, which includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the operator package recommendation method in embodiment 1 by running the computer program.

[0110] Specifically, the computer program executes the following steps when running: obtaining a first operator package subscription behavior sequence of a first user in a first time period; and analyzing the first operator package subscription behavior sequence by using a pre-trained operator package recommendation model to obtain a target operator package to be recommended to the first user at a next time in the first time period, wherein the operator package recommendation model is trained by using a plurality of target training sample data subsets whose contribution degrees are higher than a preset threshold value and whose sample quantities are not higher than a preset quantity threshold value, and the contribution degree is determined by using a Shapley value method.

[0111] As an optional implementation, the electronic device can exist in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 5 A hardware structure block diagram of an electronic device for implementing an operator package recommendation method is shown. As shown in the figure, Figure 5 The electronic device 50 can include one or more processors 502 (the processor 502 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission device 506 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand, Figure 5 The structure shown in the figure is only schematic, and it does not limit the structure of the above-mentioned electronic device. For example, the electronic device 50 can include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 5 For example, the electronic device 50 can include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 5 For example, the electronic device 50 can include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure.

[0112] It should be noted that the one or more processors 502 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 50. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0113] The memory 504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the operator package recommendation method in the embodiment of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 504 may further include a memory remotely located relative to the processor 502, and these remote memories may be connected to the electronic device 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The transmission device 506 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the electronic device 50. In one embodiment, the transmission device 506 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 506 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0115] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 50 .

[0116] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0118] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0119] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. they can be located in one place or distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0120] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0121] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0122] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method for recommending operator packages, characterized in that: include: Obtaining a first operator package subscription behavior sequence of a first user within a first time period; The first operator package subscription behavior sequence is analyzed using a pre-trained operator package recommendation model to obtain a target operator package recommended to the first user at the next moment in the first time period, wherein the operator package recommendation model is trained using multiple target training sample data subsets whose contribution is higher than a preset threshold value and the number of samples is not higher than a preset quantity threshold, and the contribution is determined using the Shapley value method.

2. The method according to claim 1, characterized in that The training process of the operator package recommendation model includes: Obtaining an initial training sample data set, wherein the initial training sample data set includes multiple sets of training sample data, and each set of the training sample data includes: a sequence of subscription behaviors of a second user to a second operator package within a third time period and a subscription behavior of the second user to a second operator package at a next moment in the third time period; The initial training sample data set is divided according to a preset division rule to obtain a plurality of training sample data subsets, and the plurality of training sample data subsets are randomly combined to obtain a plurality of first data set alliances, wherein the division rule includes one of the following: dividing the sample data set based on features, dividing the sample data set based on model prediction results, and dividing the sample data set based on sample usage scenarios; Determine the preset machine learning model; Using the plurality of first dataset alliances to perform multiple iterative training on the machine learning model, and evaluating the model performance indicators of the machine learning model after each training, and using the Shapley value method to analyze the model performance indicators corresponding to each first dataset alliance to obtain the contribution of each of the training sample data subsets; For each of the training sample data subsets, determining whether the contribution of the current training sample data subset is higher than the preset threshold value, and determining whether the number of samples in the current training sample data subset is higher than the preset number threshold value; In a case where the contribution of the current training sample data subset is higher than the preset threshold value and the number of samples is not higher than the number threshold value, using the current training sample data subset as the target training sample data subset; The machine learning model is iteratively trained using each of the target training sample data subsets to obtain the operator package recommendation model.

3. The method according to claim 2, characterized in that The machine learning model is trained multiple times using the first dataset alliances, and the model performance indicators of the machine learning model after each training are evaluated, including: For each of the first data set alliances, the machine learning model is iteratively trained using the first data set alliance, and the trained machine learning model is tested using a preset test sample data set to obtain a model performance indicator of the trained machine learning model, wherein the model performance indicator includes at least one of the following: recall rate, accuracy rate, area under the receiver operating characteristic ROC curve, and area under the precision-recall rate PR curve.

4. The method according to claim 2, characterized in that The contribution degree at least includes a marginal contribution degree, wherein the contribution degree of each of the training sample data subsets is determined based on the model performance indicators corresponding to each of the first data set alliances, including: For each of the training sample data subsets, determining a plurality of first target data set alliances including at least the current training sample data subset; Traversing each of the first target dataset alliances, determining the number of members in the current first target dataset alliance, and determining the model performance index after the current first target dataset alliance removes the current training sample data subset to train the machine learning model; calculating a first factorial value of a first difference obtained by subtracting one from the number of members, calculating a second factorial value of a second difference obtained by subtracting the number of members from the number of sets of the multiple training sample data subsets, and calculating a quotient obtained by dividing the product of the first factorial value and the second factorial value by a fourth factorial value of the number of sets of the multiple training sample data subsets; multiplying the quotient by the difference between the model performance index corresponding to the current first target dataset alliance and the model performance index corresponding to the current first target dataset alliance after removing the current training sample data subset, to obtain a contribution of the current training sample data subset in the current first target dataset alliance; The contribution of the current training sample data subset in each of the first target data set alliances is summed to obtain the marginal contribution of the current training sample data subset.

5. The method according to claim 4, characterized in that The contribution degree also includes: a comprehensive contribution degree, wherein the contribution degree of each of the training sample data subsets is determined based on the model performance indicators corresponding to each of the first data set alliances, and further includes: For each of the training sample data subsets, the marginal contribution of the current training sample data subset is multiplied by the model performance index corresponding to the first target data set alliance that only contains the current training sample data subset to obtain the comprehensive contribution of the current training sample data subset.

6. The method according to claim 2, characterized in that The method further comprises: When the contribution of the current training sample data subset is higher than the preset threshold value and the number of samples is higher than the number threshold, the current training sample data subset is divided according to the division rule, and the multiple training sample data subsets obtained by the division are randomly combined to obtain multiple second data set alliances.

7. The method according to claim 2, characterized in that The method further comprises: When the contribution of each of the training sample data subsets is not higher than the preset threshold value, the initial training sample data set is re-divided according to the division rule, and the multiple new training sample data subsets obtained by the re-division are randomly combined to obtain multiple new first data set alliances.

8. An operator package recommendation device, characterized in that: include: An acquisition module, configured to acquire a sequence of subscription behaviors of a first user to a first operator's package within a first time period; A recommendation module is used to analyze the first operator package subscription behavior sequence using a pre-trained operator package recommendation model to obtain a target operator package recommended to the first user at the next moment in the first time period, wherein the operator package recommendation model is trained using multiple target training sample data subsets whose contribution is higher than a preset threshold value and the number of samples is not higher than a preset quantity threshold, and the contribution is determined using the Shapley value method.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the operator package recommendation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the operator package recommendation method according to any one of claims 1 to 7 through the computer program.

Images