Model variable screening method and device based on distributed parallel computing AUC

Abstract

The invention relates to a model variable screening method and device based on distributed parallel computing AUC, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: determining model variables to be screened; dividing a sample set corresponding to the model variable into a plurality of sample subsets; distributing the plurality of sample subsets to a plurality of nodes of a Spark distributed system, and carrying out parallel binning calculation to obtain a local AUC; summarizing the local AUC of the plurality of nodes to a central node of the Spark distributed system for addition processing, and generating a global AUC corresponding to the model variable; and determining the discrimination capability of model variables on positive and negative tags through the global AUC, thereby screening the model variables. According to the method, the dependence of sensitive data stored by a single data source or a center is reduced by adopting distributed calculation, the efficient data analysis requirement can be met, and the potential safety hazard of data leakage can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer information processing, and in particular, to a model variable screening method and device based on distributed parallel computing AUC, an electronic device, a computer readable medium and a computer program product. BACKGROUND

[0002] In the process of model construction and data analysis, variable screening is an important step to improve model accuracy and reduce redundant information. With the increase of data dimension, selecting effective variables becomes increasingly critical. There are various existing variable screening methods, mainly including methods based on statistical tests, information theory, model evaluation and dimension reduction techniques. These methods provide different perspectives and means for screening model input variables, and can effectively identify variables with significant distinguishing characteristics to improve the prediction effect and computational efficiency of the model.

[0003] Methods based on statistical tests, such as variance selection, correlation analysis and chi-square test, screen effective variables by the statistical relationship between variables and target values; methods based on information theory use information gain, mutual information and entropy to measure the contribution of variables to target values. In addition, model evaluation methods (such as feature importance of decision tree and Lasso regression) automatically select variables during model fitting, making the screening process more concise and efficient; while dimension reduction techniques (such as PCA, LDA and SVD) reduce redundant information by reducing data dimension, ensuring the strong generalization ability of the model under high-dimensional data.

[0004] To further optimize the variable screening effect, heuristic algorithms (such as genetic algorithm and particle swarm optimization) and recursive feature elimination are also widely used in variable selection, which iteratively searches for the optimal variable combination to improve screening accuracy. However, with the increasing amount and complexity of data, these traditional methods may face problems such as excessive consumption of computing resources and slow speed when dealing with large-scale data. Therefore, how to achieve efficient and accurate variable screening while ensuring accuracy has become a technical problem to be solved.

[0005] Therefore, a new model variable screening method, device, electronic device, computer readable medium and computer program product based on distributed parallel computing AUC are needed.

[0006] The above information disclosed in the background section is only used to strengthen the understanding of the background of the present application, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] Therefore, the application provides a model variable screening method and device based on distributed parallel computing AUC, an electronic device, a computer readable medium and a computer program product.

[0008] Other features and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0009] According to an aspect of the present application, a model variable screening method based on distributed parallel computing AUC is provided, which comprises: determining a model variable to be screened; dividing a sample set corresponding to the model variable into a plurality of sample subsets; distributing the plurality of sample subsets to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUC; aggregating the local AUC of the plurality of nodes to a central node of the Spark distributed system for summation processing to generate global AUC corresponding to the model variable; and determining the discriminative ability of the model variable to positive and negative labels through the global AUC, thereby screening the model variable.

[0010] Optionally, determining the model variable to be screened comprises: determining input variables according to target variables of a model; scoring the input variables; and extracting input variables with scores lower than a variable threshold as the model variable to be screened.

[0011] Optionally, scoring the input variables comprises: scoring the input variables according to the correlation between target variables and input variables; and / or scoring the importance of the input variables through a random forest or gradient boosting model; and / or scoring the importance of the input variables through stability and information gain features.

[0012] Optionally, dividing the sample set corresponding to the model variable into a plurality of sample subsets comprises: constructing a sample set corresponding to each model variable respectively; and dividing the sample set into a plurality of sample subsets through stratified sampling.

[0013] Optionally, distributing the plurality of sample subsets to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUC comprises: converting data in the plurality of sample subsets into RDD or Dataset form; distributing the sample subsets to a plurality of nodes of a Spark distributed system in a broadcast manner; and performing parallel binning calculation of the model in each node based on the sample subsets to generate local AUC.

[0014] Optionally, the model is calculated in parallel in each node based on the sample subsets and a local AUC is generated, including: inputting the sample subsets into the model in each node to obtain a model calculation result; performing a binning operation on the model calculation result; and calculating the local AUC value by integrating the TPR and FPR of each interval using the rectangular approximation method based on the calculation result after the binning operation.

[0015] Optionally, the binning operation on the model calculation result includes adjusting the number of bins based on calculation accuracy and calculation efficiency.

[0016] Optionally, the local AUCs of the multiple nodes are aggregated to a center node of the Spark distributed system for summation processing to generate a global AUC corresponding to the model variable, including: collecting the local AUC values of each node by the center node of the Spark through an aggregation operation; determining a weight based on the sample quantity of the sample subsets; and generating the global AUC corresponding to the model variable by the weight and the local AUC value of each sample subset.

[0017] Optionally, the discrimination ability of the model variable for positive and negative labels is determined based on the global AUC, so as to perform screening of the model variable, including: sorting the model variables according to the numerical size of the global AUC corresponding to the model variables; determining the discrimination ability of the model variable for positive and negative labels based on the sorting; and extracting the model variable whose discrimination ability is less than a threshold.

[0018] According to an aspect of the present application, a model variable screening device based on distributed parallel AUC calculation is provided, which includes: a variable module configured to determine a model variable to be screened; a division module configured to divide a sample set corresponding to the model variable into multiple sample subsets; a distribution module configured to distribute the multiple sample subsets to multiple nodes of a Spark distributed system for parallel binning calculation to obtain local AUCs; a summation module configured to aggregate the local AUCs of the multiple nodes to a center node of the Spark distributed system for summation processing to generate a global AUC corresponding to the model variable; and a screening module configured to determine the discrimination ability of the model variable for positive and negative labels based on the global AUC, so as to perform screening of the model variable.

[0019] According to an aspect of the present application, an electronic device is provided, which includes: one or more processors; a storage device configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0020] According to an aspect of the present application, a computer readable medium having a computer program stored thereon is provided, and the program is executed by a processor to implement the method as described above.

[0021] According to an aspect of the present application, a computer program product is provided, comprising computer programs / instructions which, when executed by a processor, implement the method as in the foregoing.

[0022] According to the model variable screening method and device based on distributed parallel computing AUC, the electronic equipment, the computer readable medium and the computer program product provided by the present application, the model variables to be screened are determined; the sample set corresponding to the model variables is divided into a plurality of sample subsets; the plurality of sample subsets are distributed to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUC; the local AUC of the plurality of nodes is aggregated to a center node of the Spark distributed system for summation processing to generate global AUC corresponding to the model variables; the ability of the model variables to distinguish positive and negative labels is determined through the global AUC, so that the screening of the model variables is performed. In this way, the dependence on a single data source or central storage of sensitive data is reduced by using distributed computing, the centralized leakage risk of financial transaction data in transmission or storage is reduced, the efficient data analysis demand can be met, and the data leakage risk is reduced.

[0023] It should be understood that the foregoing general description and the following detailed description are only examples and are not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0025] Figure 1 FIG. 1 is a flowchart illustrating a model variable screening method based on distributed parallel computing AUC according to an example embodiment.

[0026] Figure 2 FIG. 2 is a flowchart illustrating a model variable screening method based on distributed parallel computing AUC according to another example embodiment.

[0027] Figure 3 FIG. 3 is a flowchart illustrating a model variable screening method based on distributed parallel computing AUC according to another example embodiment.

[0028] Figure 4 FIG. 4 is a block diagram of a model variable screening device based on distributed parallel computing AUC according to an example embodiment.

[0029] Figure 5 FIG. 5 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0030] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the several views and the description of the figures.

[0031] Figure 1 is a flow chart of a model variable screening method based on distributed parallel computing AUC according to an example embodiment. The model variable screening method based on distributed parallel computing AUC 10 comprises at least steps S102 to S108.

[0032] As shown in S102, the model variables to be screened are determined. For example, the input variables can be determined according to the target variable of the model; the input variables are scored; and the input variables with scores lower than the variable threshold are extracted as the model variables to be screened. Figure 1

[0033] In one embodiment, scoring the input variables comprises scoring the input variables by the correlation between the target variable and the input variables; more specifically, each input variable can be scored by analyzing the correlation between the input variable and the target variable (e.g., calculating the Pearson correlation coefficient, the Spearman correlation coefficient, etc.). The correlation score can directly reflect the degree of influence of the variable on the target result.

[0034] In one embodiment, scoring the input variables comprises scoring the importance of the input variables by a random forest or a gradient boosting model; the contribution of the input variables can be evaluated by constructing a model such as a random forest or a gradient boosting tree, and using the feature importance scores generated by the model. The importance scores of different variables by the model can reveal which variables have greater discriminability in the prediction process.

[0035] In one embodiment, scoring the input variables comprises scoring the importance of the input variables by stability and information gain features. The contribution of each input variable to the target variable can be judged by calculating the information gain, and the stability of the input variable in the data is evaluated to eliminate variables that are too volatile and unstable. This feature-based scoring method can further ensure that the screened variables have strong robustness in model prediction.

[0036] ​Finally, a score threshold can be set to mark variables with scores below the threshold as screening variables to reduce the interference of noise variables on the model effect.

[0037] In S104, the sample set corresponding to the model variable is divided into multiple sample subsets. For example, a sample set corresponding to each model variable can be constructed respectively, and the sample set is divided into multiple sample subsets by stratified sampling.

[0038] For example, a sample set related to each model variable can be generated, and the sample set contains positive and negative sample data for calculating AUC.

[0039] To ensure the proportion of different categories in each subset is consistent, stratified sampling can be performed according to the category proportion. For example, in the case of a positive-negative sample ratio of 1:1, it is ensured that each subset also maintains this ratio.

[0040] More specifically, using stratified sampling technology, each sample set is divided into multiple subsets according to a certain proportion to ensure that the samples in each subset can uniformly cover positive and negative labels while maintaining the consistency of the overall data distribution. The multiple sample subsets processed in this way facilitate subsequent independent calculation in a distributed system and ensure sample representativeness and calculation efficiency.

[0041] In S106, the multiple sample subsets are distributed to multiple nodes of a Spark distributed system for parallel binning calculation to obtain local AUC. For example, the data in the multiple sample subsets can be converted into RDD or Dataset form; the sample subsets are distributed to multiple nodes of the Spark distributed system in a broadcast manner; and the model is calculated based on the sample subsets in parallel binning in each node to generate local AUC.

[0042] The data of the sample subsets is converted into RDD (Resilient Distributed Dataset) or Dataset form supported by Spark, so as to utilize the Spark framework to perform efficient data processing in a distributed environment.

[0043] More specifically, the broadcast mechanism of Spark can be used to distribute the sample subsets to different nodes, so that each node can independently perform data processing. During distribution, after the sample subsets are distributed to each node, each node loads the corresponding sample data for calculation.

[0044] On each node, binning calculation is performed based on the sample subsets, the AUC curve is approximated as a superposition of multiple rectangular regions, and the local AUC value is generated by summing the areas of the rectangles. Binning operation can significantly reduce the calculation complexity and improve the calculation efficiency.

[0045] In S108, the local AUCs of the plurality of nodes are aggregated to the center node of the Spark distributed system for summation processing to generate the global AUC corresponding to the model variable. The center node of the Spark may, for example, collect the local AUC values of the nodes through an aggregation operation; determine the weights according to the sample sizes of the sample subsets; and generate the global AUC corresponding to the model variable through the weights and the local AUC values of the respective sample subsets.

[0046] More specifically, the center node of the Spark collects the local AUC values from the nodes, which can be completed using a reduce or aggregate operation. The weight of each local AUC is calculated according to the sample size of the respective sample subset. The weight can be determined by the ratio of the sample size of the subset to the total sample size to ensure that the subset with a larger sample size contributes relatively more to the global AUC.

[0047] The global AUC is obtained by weighted averaging using the respective local AUC values and their corresponding weights, as the discrimination performance indicator of the model variable on the overall data.

[0048] In S110, the discrimination ability of the model variable for the positive and negative labels is determined through the global AUC, thereby performing the screening of the model variable. For example, the model variables can be sorted according to the numerical values of their corresponding global AUCs; the discrimination ability of the model variable for the positive and negative labels is determined according to the sorting; and the model variables with a discrimination ability less than a threshold value are extracted.

[0049] The respective model variables are sorted according to the global AUC values, and the higher the AUC value of a variable, the stronger the discrimination ability of the variable for the target variable.

[0050] According to the preset AUC threshold value, the variables with a discrimination ability less than the threshold value can be screened out and removed from the model input variables, thereby retaining the variables with the strongest discrimination effect and improving the prediction ability and efficiency of the model.

[0051] According to the model variable screening method based on distributed parallel computing AUC of the present application, the model variable to be screened is determined; the sample set corresponding to the model variable is divided into a plurality of sample subsets; the plurality of sample subsets are distributed to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUCs; the local AUCs of the plurality of nodes are aggregated to the center node of the Spark distributed system for summation processing to generate the global AUC corresponding to the model variable; and the discrimination ability of the model variable for the positive and negative labels is determined through the global AUC, thereby performing the screening of the model variable. The distributed computing reduces the dependence on a single data source or central storage of sensitive data, reduces the risk of centralized leakage of financial transaction data in transmission or storage, meets the efficient data analysis demand, and reduces the data leakage risk.

[0052] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teachings of the present disclosure, these principles can be applied to many other embodiments.

[0053] Figure 2 is a flow chart of a model variable screening method based on distributed parallel computing AUC according to another exemplary embodiment. Figure 2 The flow 20 shown is for Figure 1 Detailed description of S106 "distributing the plurality of sample subsets into a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUC" in the flow shown.

[0054] As Figure 2 As shown in S202, the sample subsets are distributed in a broadcast manner into a plurality of nodes of a Spark distributed system. For example, the data in the plurality of sample subsets can be converted into RDD or Dataset form for distribution into a plurality of nodes of a Spark distributed system.

[0055] Each sample subset is converted into a distributed data format supported by Spark, such as RDD (Resilient Distributed Dataset) or Dataset. RDD is suitable for data that needs to be fault-tolerant and cannot be processed in real time in distributed computing, while Dataset has stronger data structure and SQL functions, and is suitable for operations that need to be structured queries.

[0056] The large data set is distributed to each computing node through a broadcast variable (Broadcast Variable), avoiding frequent data transmission. In this way, the sample subsets can be broadcast to each node, so that each node has the same sample subset data, so as to perform independent calculation locally, thereby improving the transmission efficiency of data and the overall computing performance.

[0057] After the sample subsets are distributed, each node loads the local data and starts calculation. Using the distributed architecture of Spark, each node only needs to process the data assigned to itself, thereby reducing the memory and computing burden, and achieving parallel acceleration of tasks.

[0058] In S204, parallel binning calculation is performed in each node based on the sample subsets. For example, the sample subsets can be input into the model in each node to obtain the model calculation results, and the model calculation results can be subjected to binning operation.

[0059] On each node, the sample subset is input into the distributed model for computation, and the prediction result of each sample data is obtained, including the prediction score or category of the target variable.

[0060] The computation result of the model is binned. The binning operation converts continuous values into multiple intervals (i.e., bins), divides the sample prediction results according to certain intervals (e.g., 0.1, 0.2, etc.), and calculates the true positive rate (TPR) and false positive rate (FPR) in each interval, facilitating the subsequent calculation of AUC.

[0061] More specifically, the number of bins can be adjusted by calculating the accuracy and computational efficiency. To balance the accuracy and efficiency, the number of bins can be adjusted according to the distribution of the data and the requirement for accuracy. The more bins, the higher the accuracy of the calculated AUC, but it will also increase the computational burden; the fewer bins, the higher the calculation speed, but the accuracy may decrease. Therefore, in actual operation, the appropriate number of bins can be selected according to the requirements to achieve the ideal balance between accuracy and computational efficiency.

[0062] In S206, the TPR and FPR of each interval are integrated by the rectangular approximation method based on the computation result after the binning operation, and the local AUC value is calculated.

[0063] Calculate the TPR and FPR of the interval: In the binning operation of each node, the true positive rate (TPR) and false positive rate (FPR) in each interval are calculated. The calculation of TPR and FPR is based on the number of positive and negative samples in each interval, for example, TPR is the proportion of correctly predicted positive samples, and FPR is the proportion of misjudged negative samples as positive samples.

[0064] Each bin interval is regarded as a small rectangle, the height of the rectangle corresponds to the TPR value of the current interval, and the width corresponds to the FPR increment of the adjacent interval. By adding up the areas of the rectangles, the integral approximation of AUC is realized. This method can effectively approximate the area under the ROC curve, thereby obtaining the local AUC.

[0065] The areas of all rectangles are added up to obtain the local AUC value of each node. The local AUC value represents the discrimination ability of the model variable for the target label under the sample subset of the current node. The local AUC value will be passed back to the center node for aggregation to calculate the global AUC of the model variable.

[0066] Figure 3 is a flowchart of a model variable screening method based on distributed parallel computation of AUC according to another exemplary embodiment. Figure 3 The flowchart 30 shown is a method for screening model variables based on distributed parallel computation of AUC. Figure 1The detailed description of S108 in the flow shown, "aggregate the local AUC of multiple nodes to the center node of the Spark distributed system for summation processing to generate the global AUC corresponding to the model variable".

[0067] As shown in Figure 3 In S302, the center node of Spark collects the local AUC values of each node through aggregation operation. In the Spark distributed system, the computing task is distributed to multiple computing nodes, and each node generates a local AUC value after processing a subset of samples. The center node of Spark uses aggregation operations such as reduce or aggregate functions to collect these local AUC values.

[0068] In a distributed environment, the transmission cost of data is high, so the aggregation operation of Spark will reduce the communication cost through optimization mechanism (such as local aggregation in task, or recursive collection through tree structure between nodes), so as to realize efficient and fast collection of local AUC results of each node.

[0069] Spark has data fault tolerance mechanism. If a node is interrupted due to failure, the center node can trigger the re-computation mechanism to ensure the accuracy and integrity of the final collected local AUC value.

[0070] In S304, the weight is determined according to the sample size of the sample subset. In order to fairly reflect the contribution of each subset to the global AUC, a weight needs to be assigned to each local AUC value.

[0071] Each sample subset has determined its sample size before being distributed to the node. The center node calculates the weight for each subset according to these sample sizes, usually using the proportion of the sample size of the subset to the total sample size as the weight value.

[0072] Different sample subsets may contain different number of samples, and through weight balancing, the influence of small sample size subsets on the global result can be avoided, so that the overall calculation result is more consistent with the overall data distribution.

[0073] In S306, the global AUC corresponding to the model variable is generated through the weight and local AUC value of each sample subset. Through weighted aggregation, it is ensured that the final generated global AUC value can reflect the contribution of all sample subsets, and the evaluation effect of the overall model variable will not be affected by the small sample size of a certain subset or the deviation of a certain sample.

[0074] The global AUC value as the discriminant ability index of the model variable on the overall data can be used for the subsequent model variable screening step. In this way, it can be ensured that the selected variable has stable discriminant effect on the overall data and is suitable for further model optimization and performance improvement.

[0075] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above-mentioned functions defined by the method provided in the present application are executed. The program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0076] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0077] The following is a device embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0078] Figure 4 is a block diagram of a model variable screening device based on distributed parallel computing AUC according to an exemplary embodiment. As shown in Figure 4 The model variable screening device based on distributed parallel computing AUC 40 includes a variable module 402, a division module 404, a distribution module 406, a summation module 408, and a screening module 410.

[0079] The variable module 402 is configured to determine the model variables to be screened; the variable module 402 is further configured to determine the input variables according to the target variables of the model; score the input variables; and extract the input variables with scores lower than the variable threshold as the model variables to be screened.

[0080] The division module 404 is configured to divide the sample set corresponding to the model variables into multiple sample subsets; the division module 404 is further configured to construct the sample set corresponding to each model variable respectively; and divide the sample set into multiple sample subsets by stratified sampling.

[0081] The distribution module 406 is configured to distribute the multiple sample subsets to multiple nodes of a Spark distributed system for parallel binning calculation to obtain local AUC; the distribution module 406 is further configured to convert the data in the multiple sample subsets into RDD or Dataset form; distribute the sample subsets to multiple nodes of the Spark distributed system in a broadcast manner; and perform parallel binning calculation based on the sample subsets in each node, and generate local AUC.

[0082] The adding module 408 is configured to add the local AUCs of the plurality of nodes to the central node of the Spark distributed system for addition processing to generate the global AUC corresponding to the model variable; the adding module 408 is also configured to collect the local AUCs of the nodes by the central node of the Spark through an aggregation operation; determine the weight according to the sample size of the sample subset; and generate the global AUC corresponding to the model variable through the weight and the local AUC corresponding to each sample subset.

[0083] The screening module 410 is configured to determine the discrimination ability of the model variable to the positive and negative labels through the global AUC, thereby screening the model variable. The screening module 410 is also configured to sort the model variables according to the numerical size of the global AUC corresponding to the model variables; determine the discrimination ability of the model variable to the positive and negative labels according to the sorting; and extract the model variable whose discrimination ability is less than a threshold.

[0084] According to the model variable screening device based on distributed parallel computing AUC, the model variable to be screened is determined; the sample set corresponding to the model variable is divided into a plurality of sample subsets; the plurality of sample subsets are distributed to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUCs; the local AUCs of the plurality of nodes are added to the central node of the Spark distributed system for addition processing to generate the global AUC corresponding to the model variable; and the discrimination ability of the model variable to the positive and negative labels is determined through the global AUC, thereby screening the model variable. In this way, the distributed computing reduces the dependence on a single data source or central storage of sensitive data, reduces the centralized leakage risk of financial transaction data in transmission or storage, can meet the efficient data analysis demand, and can reduce the data leakage risk.

[0085] As shown in Figure 5 The electronic device provided by the embodiment of the present application includes a processor 510, a memory 520 and a bus, wherein the processor 510 and the memory 520 complete mutual communication through the bus 540;

[0086] The memory 620 is configured to store a computer program.

[0087] The processor 610 is configured to execute the program stored in the memory 620 to implement the model variable screening method based on distributed parallel computing AUC of any of the above embodiments.

[0088] The communication interface 620 is configured to communicate between the above electronic device and other devices.

[0089] The memory 620 can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory 620. Optionally, the memory 620 can also be at least one storage device located away from the aforementioned processor 610.

[0090] The above-described methods in the present application, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes of the above-described embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-described various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc.

[0091] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the model variable screening method based on distributed parallel computing AUC of any one of the above-described embodiments. For example, the model variable to be screened can be determined; the sample set corresponding to the model variable is divided into a plurality of sample subsets; the plurality of sample subsets are distributed to a plurality of nodes of a Spark distributed system for parallel binning calculation to obtain local AUC; the local AUC of the plurality of nodes is summarized to a center node of the Spark distributed system for summation processing to generate global AUC corresponding to the model variable; the ability of the model variable to distinguish positive and negative labels is determined through the global AUC, so that the screening of the model variable is performed.

[0092] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A model variable screening method based on distributed parallel computing AUC, characterized in that: include: Determine the model variables to be screened; Dividing the sample set corresponding to the model variables into multiple sample subsets; Distributing the multiple sample subsets to multiple nodes of the Spark distributed system to perform parallel binning calculations to obtain local AUCs; Aggregate the local AUCs of multiple nodes to the central node of the Spark distributed system for summation to generate the global AUC corresponding to the model variables; The global AUC is used to determine the ability of the model variables to discriminate between positive and negative labels, thereby screening the model variables.

2. The method according to claim 1, wherein Identify the model variables to be screened, including: Determine the input variables based on the target variable of the model; scoring the input variables; Input variables with scores lower than the variable threshold are extracted as the model variables to be screened.

3. The method according to claim 2, wherein Scoring the input variables includes: Scoring input variables based on their correlation with the target variable; and / or Scoring the importance of the input variables using a random forest or gradient boosting model; and / or The importance of the input variables is scored by stability and information gain features.

4. The method according to claim 1, wherein The sample set corresponding to the model variables is divided into multiple sample subsets, including: Construct the corresponding sample set for each model variable; The sample set is divided into multiple sample subsets by stratified sampling.

5. The method according to claim 1, wherein Distributing the multiple sample subsets to multiple nodes in the Spark distributed system for parallel binning calculation to obtain local AUC, including: Convert the data in the plurality of sample subsets into RDD or Dataset format; Distributing the sample subset to multiple nodes of the Spark distributed system in a broadcast manner; In each node, the sample subsets are divided into parallel bins and the local AUC is generated.

6. The method according to claim 5, wherein In each node, the sample subsets are divided into parallel bins and the local AUC is generated, including: Input the sample subset into the model in each node to obtain the model calculation results; Performing a binning operation on the model calculation results; Based on the calculation results after the binning operation, the TPR and FPR of each interval are integrated using the rectangular approximation method to calculate the local AUC value.

7. The method according to claim 6, wherein The model calculation results are binned, including: The number of bins is adjusted based on computational accuracy and efficiency.

8. The method according to claim 1, wherein The local AUCs of multiple nodes are aggregated to the central node of the Spark distributed system for summation to generate the global AUC corresponding to the model variables, including: Spark's central node collects the local AUC values ​​of each node through aggregation operations; Determine the weights based on the sample size of the sample subset; The global AUC corresponding to the model variable is generated by the weights and local AUC values ​​corresponding to each sample subset.

9. The method according to claim 1, wherein The global AUC is used to determine the ability of the model variable to distinguish positive and negative labels, thereby screening the model variables, including: Sort the model variables according to the numerical values ​​of their corresponding global AUC; Determine the ability of the model variables to discriminate between positive and negative labels based on the ranking; Extract model variables whose discriminative ability is less than the threshold.

10. A model variable screening device based on distributed parallel computing AUC, characterized in that: include: A variable module is used to determine the model variables to be screened; A partitioning module, configured to partition the sample set corresponding to the model variables into a plurality of sample subsets; A distribution module is used to distribute the multiple sample subsets to multiple nodes of the Spark distributed system to perform parallel binning calculations to obtain local AUCs; The summation module is used to aggregate the local AUCs of multiple nodes to the central node of the Spark distributed system for summation processing to generate the global AUC corresponding to the model variables; The screening module is used to determine the discrimination ability of the model variable for positive and negative labels through the global AUC, thereby screening the model variable.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images