A modeling method, device and electronic equipment of a data set and a storage medium

Abstract

The application relates to a data set modeling method and device, electronic equipment and a storage medium, and belongs to the modeling technical field.The method comprises the following steps: establishing a business wide table of a data set, performing Pearson test on all fields in the business wide table to obtain main features, and performing modeling according to the main features to obtain a first model; important types are selected from the first model as significant features of the feature importance degrees of the splitting times and the gains; the root nodes of each tree of the first model are circularly traversed, the root node features are deduplicated, and the deduplicated root node features are summarized; key features are selected, and the weights of the key features are set; a score is calculated according to the key features and the weights; a parameter sample is determined according to the score; and a second modeling is performed according to the parameter sample and the main features corresponding to the parameter sample to obtain a second model. The application adopts a pre-filtering mode, and only parameter samples with high probabilities are used to perform modeling to obtain the second model, so that the model can focus more on high-probability users for calculation and derivation.

Description

Technical Field

[0001] This invention belongs to the field of modeling technology, and in particular relates to a modeling method, apparatus, electronic device and storage medium for a dataset. Background Technology

[0002] When modeling large-scale imbalanced datasets, the overall cardinality is large and the proportion of positive samples is very small, resulting in a lack of information about positive samples in both the training and test sets, leading to very low accuracy in the evaluation results.

[0003] Currently, algorithms with keywords related to large-scale datasets, imbalanced positive and negative samples, tree models, and mathematical modeling mainly include the following aspects: For large-scale datasets, the approach can be to transform big data problems into easily solvable small data problems, cluster large-scale datasets, sequentially complete the mining of small data, and finally merge the mining results to obtain the final result; or to construct multiple classifiers with the greatest differences and achieve efficient integrated decision-making of multiple classifiers.

[0004] In dealing with imbalanced positive and negative samples, SMOTE oversampling is used to increase the number of minority samples and undersampling is used to reduce the number of majority samples; or random sampling with replacement is used to dilute the negative class samples by sampling multiple times, and the negative samples and all positive samples are combined into multiple new sample sets so that the number of positive and negative samples in each set is equal.

[0005] Regarding pre-filtering, when setting thresholds, some methods are too simplistic and cannot comprehensively measure the value of all indicators. These methods are too limited to fully utilize geometric knowledge, delineate a circular domain as the extraction range of the secondary boundary vector, and use the data sample points found from the secondary boundary vector set to add to the new training set until the sample points are balanced.

[0006] Regarding the root node and feature importance attributes of the tree model, how can we accurately and automatically find a reasonable root node according to preset rules, or focus on calculating the feature importance of the tree model by introducing the node depth of the decision tree, or use the random forest algorithm to calculate the importance of each dimension and standardize the distance by weighting the dimension importance values? The formula for calculating the dimension importance is the decrease in the rate of change of information entropy when splitting, which means that it uses information about the information gain of feature splitting, but does not consider information about the number of splits. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of the invention is to provide a modeling method, device, electronic device and storage medium for datasets. The invention adopts a pre-filtering method, which can effectively filter out samples with low probability and obtain a second lightGBM model by modeling the input parameter samples with higher probability. The second lightGBM model reduces the total number of samples to be predicted and can focus more on high-probability users for calculation and derivation.

[0008] A first aspect of the present invention proposes a method for modeling a dataset, comprising: A business wide table is created for the dataset. Pearson test is performed on all fields in the business wide table to obtain the main features. The first lightGBM model is obtained by performing the first lightGBM model based on the main features. From the first lightGBM model, significant features of importance were selected based on two characteristics: the number of splits and the gain. Iterate through the root nodes of each tree in the first lightGBM model, remove duplicate features from the root nodes, and summarize the deduplicated root node features. Analyze the main features, significant features, and root node features after deduplication to select key features, and set the weights of the key features; Scores are calculated based on key features and their weights. The input sample is determined based on the scores; The second lightGBM model is obtained by performing a second lightGBM modeling based on the input parameter samples and their corresponding main features.

[0009] Furthermore, in the above-mentioned dataset modeling method, a business wide table is established, and Pearson tests are performed on all fields in the business wide table to obtain the main features, including: Based on the business situation and the corresponding dataset, create a wide business table containing multiple independent variable fields and one dependent variable field; Perform Pearson tests on all fields of the business wide table to analyze the Pearson correlation coefficient of each field with the dependent variable; Fields with a Pearson correlation coefficient greater than a first preset threshold are selected as the main features.

[0010] Furthermore, in the modeling method for the aforementioned dataset, significant features of importance, namely the number of splits and the gain, are selected from the first lightGBM model. These features include: Call the feature importance attribute of the first lightGBM model and check the importance of model features with the importance type being number of splits and the importance type being gain. The importance of model features is ranked separately for the number of splits and the gain. Output the model feature importance of the number of splits and the model feature importance of the gain for the top N ranked numbers, respectively, to obtain salient features with two feature importance types: number of splits and gain.

[0011] Furthermore, in the above-mentioned dataset modeling method, the root node of each tree in the first lightGBM model is iterated through, and the root node features are deduplicated. The deduplicated root node features are then summarized, including: Extract the tree structure information of each tree in the first lightGBM model; Traverse and summarize the root node of each tree; The root nodes are deduplicated after being summarized and arranged in their original order to obtain the root node features.

[0012] Furthermore, in the above-mentioned dataset modeling method, key features are selected by analyzing the main features, salient features, and deduplicated root node features, and the weights of the key features are set, including: Key features are selected based on the overlap between the main features, salient features, and the root node features after deduplication, and the weights of the key features are set.

[0013] Furthermore, in the above-mentioned modeling method for a dataset, the scoring is calculated based on key features and their weights, including: The key features are preprocessed sequentially, followed by z-standardization and 0-1 standardization to obtain the standardized results. The score is obtained by multiplying the standardized result by the corresponding weight.

[0014] Furthermore, in the above-mentioned modeling method for a dataset, determining the input parameter samples based on the scores includes: The input parameter samples are determined based on the score and the magnitude of the second preset threshold.

[0015] A second aspect of the present invention also provides a modeling apparatus for a dataset, comprising: The first modeling module is used to build a business wide table for the dataset, perform Pearson test on all fields in the business wide table to obtain the main features, and perform the first lightGBM modeling based on the main features to obtain the first lightGBM model. Selection module: Used to select salient features from the first lightGBM model based on the importance of two types of features: number of splits and gain. The summary module is used to iterate through the root nodes of each tree in the first lightGBM model, remove duplicate features from the root nodes, and summarize the deduplicated root node features. Analysis and settings modules: These are used to analyze the main features, significant features, and root node features after deduplication, select key features, and set the weights of key features. Calculation module: Used to calculate scores based on key features and their weights; Determine module: Used to determine the input parameter sample based on the score; The second modeling module is used to perform a second lightGBM modeling based on the input sample and its corresponding main features to obtain the second lightGBM model.

[0016] A third aspect of the present invention also provides an electronic device comprising: a processor and a memory; The processor executes a modeling method for a dataset, such as one of the above, by calling programs or instructions stored in memory.

[0017] In a fourth aspect, the present invention also provides a computer-readable storage medium that stores a program or instructions that cause a computer to perform a modeling method for a dataset as described in any of the preceding claims.

[0018] The beneficial effects of this invention are as follows: This invention utilizes two feature selection processes, a pre-filtering approach with scoring, and two LightGBM modeling steps to process large-scale imbalanced datasets. During the pre-filtering feature selection, it integrates three parts: the Pearson test, the internal feature importance attribute of the model, and the root node of the tree model. This allows for the examination of the overlap of the three sets of features. The feature importance attribute is calculated by comprehensively measuring both the number of splits and the information gain. The root node of the tree model is analyzed as the key feature range, effectively utilizing the split gain information within the model to comprehensively evaluate the fields and improve the identification ability of the evaluation system. During the pre-filtering, a multi-dimensional index and corresponding weight evaluation system is established, rather than filtering solely based on the static number of single features or the dynamic ratio of two features. A second LightGBM model is obtained by modeling input samples with higher probabilities. The second LightGBM model reduces the total number of samples to be predicted, allowing for greater focus on high-probability users for calculation and derivation. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0020] Figure 1 A diagram illustrating a dataset modeling method provided in an embodiment of the present invention; Figure 2 A method diagram for obtaining key features provided in an embodiment of the present invention; Figure 3A method diagram for obtaining significant features provided in an embodiment of the present invention; Figure 4 A diagram illustrating a method for obtaining root node features provided in an embodiment of the present invention; Figure 5 A diagram of a dataset modeling apparatus provided in an embodiment of the present invention; Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0023] In the description of this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the present invention.

[0025] Before introducing the embodiments of the present invention, the technical terms involved in the present invention will be introduced first.

[0026] The lightGBM algorithm is similar in principle to the GBDT and XGBoost algorithms, using the negative gradient of the loss function as an approximation of the residual of the current decision tree to fit a new decision tree. It boasts faster training efficiency, lower memory usage, higher accuracy, support for parallel learning, the ability to handle large-scale data, and support for direct use of class features. The algorithm is based on a leaf-wise decision tree growth strategy, histogram algorithm, histogram difference acceleration, parallel learning, one-sided gradient sampling (GOSS) algorithm, and mutually exclusive feature binding (EFB) algorithm.

[0027] The Pearson correlation test is used to test whether a linear correlation exists between two variables. It is a statistical method used to measure the strength and direction of the linear relationship between two continuous variables. It measures the correlation based on the covariance between the variables and standardizes it to obtain the correlation coefficient. The Pearson correlation coefficient ranges from -1 to 1.

[0028] Root node in a tree model: Exists in tree models such as decision trees, random forests, and lightGBM. The root node is a component of a tree, also called the tree root. Every non-empty binary tree has one and only one root node. In short, it is the ancestor of all nodes in the same tree except itself, and has no parent node.

[0029] This invention proposes a modeling method, device, electronic device, and storage medium for datasets. By employing a pre-filtering approach, low-probability samples can be effectively screened and filtered, allowing only high-probability samples to be input into the algorithm model. This reduces the total number of samples that need to be predicted, enabling the model to focus more on high-probability users for calculation and derivation, thus providing a powerful auxiliary role for the model.

[0030] Method Implementation Examples Figure 1 A diagram illustrating a dataset modeling method provided in an embodiment of the present invention.

[0031] In a first aspect, the present invention proposes a method for modeling datasets, combining... Figure 1 ,include: S1: Create a business wide table for the dataset, perform Pearson test on all fields in the business wide table to obtain the main features, and perform the first lightGBM modeling based on the main features to obtain the first lightGBM model.

[0032] Specifically, in this embodiment of the invention, the method of establishing a business wide table and performing Pearson tests on all fields in the business wide table to obtain the principal features will be described in detail below. Here, the principal features applied for the first time in lightGBM modeling are those with high Pearson correlation coefficients.

[0033] The code for the first lightGBM model obtained by performing the first lightGBM model based on the main features is as follows: model_lgb=lgb.train(params=params,train_set=lgb_train,num_boost_round=100,valid_sets=lgb_valid,early_stopping_rounds=5) S2: Select significant features from the first lightGBM model, based on the importance of two features: number of splits and gain.

[0034] Specifically, in this embodiment of the invention, the method for selecting significant features of importance type two from the first lightGBM model, namely the number of splits and the gain, is described in detail below.

[0035] It should be understood that when selecting the feature importance parameters within the LightGBM model, both split count and information gain are considered. Significant features with an importance type of information gain (gain) contribute significantly to the total information gain after splitting, while those with an importance type of split contribute significantly to the total number of splits. Therefore, if a feature sample is highly discriminative and only requires one split to effectively separate the sample, while other features will continue to split many times, the split count importance of this feature that only splits once will be extremely low, even though the feature itself is very important. In this case, due to its high discriminativeness, the feature brings a large information gain, so its information gain importance will be high and significant. Therefore, selecting significant features with both split count and information gain importance types can avoid extreme cases where some features have high discriminativeness but few splits, or low discriminativeness but many splits, thus more comprehensively filtering out important features and preventing the omission of important features due to extreme cases of a single type being strong.

[0036] S3: Iterate through the root node of each tree in the first lightGBM model, remove duplicate features from the root node features, and summarize the deduplicated root node features.

[0037] Specifically, in this embodiment of the invention, the method of iterating through the root node of each tree in the first lightGBM model, removing duplicate features from the root node, and summarizing the deduplicated root node features will be described in detail below.

[0038] It should be understood that exploring the root node within the tree model as the key feature range can effectively utilize the splitting results gain within the tree model, enabling comprehensive analysis of the fields. In the lightGBM model, each tree first splits from the root node, and then at each split, it seeks the optimal branch point, selecting the node with the greatest splitting gain from all leaf nodes. Since only one leaf node can split at each level, it means that the optimal feature, the optimal splitting point, and the leaf node with the greatest gain need to be selected. The root node is the initially selected node, which is the optimal feature and the feature with the greatest gain for each tree. Therefore, using the root node of each tree as the candidate range of key nodes can more effectively distinguish samples.

[0039] S4: Analyze the main features, significant features, and root node features after deduplication to select key features and set the weights of the key features.

[0040] Specifically, in this embodiment of the invention, a comprehensive analysis is conducted on the high relevance of the Pearson test, the high importance of the features, and the overlap of the root node features. Key features are selected, key feature weights are set, and an indicator evaluation system is established.

[0041] S5: Calculate the score based on the key features and their weights.

[0042] Specifically, in this embodiment of the invention, the method for calculating scores based on key features and the weights of key features will be described in detail below.

[0043] S6: Determine the input sample based on the scores.

[0044] Specifically, in this embodiment of the invention, the input parameter samples are determined based on the range of the scores and the size of the second preset threshold, and the samples whose score range is greater than the second preset threshold are determined as input parameter samples.

[0045] S7: Based on the input sample and the main features corresponding to the input sample, perform a second lightGBM modeling to obtain the second lightGBM model.

[0046] Specifically, in this embodiment of the invention, a pre-filtering method is used to effectively filter low-probability samples to obtain input parameters. Thus, only high-probability input parameters are used for the second lightGBM modeling to obtain the second lightGBM model, which reduces the total number of samples that need to be predicted. This allows the model to focus more on high-probability users for calculation and derivation, providing a strong auxiliary role for the model.

[0047] Figure 2 A diagram illustrating a method for obtaining key features according to an embodiment of the present invention.

[0048] Furthermore, in the aforementioned dataset modeling method, a business wide table is established, and Pearson tests are performed on all fields in the business wide table to obtain the principal features, combined with... Figure 2 It includes three steps, S21 to S23: S21: Based on the business situation and the corresponding dataset, create a wide business table containing multiple independent variable fields and one dependent variable field; S22: Perform Pearson tests on all fields of the business wide table and analyze the Pearson correlation coefficient of each field with the dependent variable; S23: Select fields with Pearson correlation coefficients greater than the first preset threshold as the main features.

[0049] Specifically, in this embodiment of the invention, a business wide table containing multiple independent variable fields and one dependent variable field is established based on the business situation and the corresponding dataset. Pearson correlation tests are performed on all fields of the business wide table to analyze the influence strength of each field on the dependent variable, i.e., the Pearson correlation coefficient of the dependent variable. Based on the business analysis, fields with Pearson correlation greater than a first preset threshold are selected as the main features.

[0050] The code to analyze the Pearson correlation coefficient of each field with respect to the dependent variable is as follows: corr_value=data.corr(method='pearson').

[0051] For example, a wide table is created using all 40 business fields and dependent variable labels from the original 10 million samples, and the Pearson correlation coefficient is analyzed for all business fields in the wide table.

[0052] Figure 3 A diagram illustrating a method for obtaining significant features according to an embodiment of the present invention.

[0053] Furthermore, in the aforementioned dataset modeling method, significant features of importance, namely the number of splits and the gain, are selected from the first lightGBM model, and combined with... Figure 3 It includes three steps, S31 to S33: S31: Call the feature importance attribute of the first lightGBM model and check the feature importance of the model with the importance type being number of splits and the importance type being gain.

[0054] Specifically, in this embodiment of the invention, the feature_importance attribute of the first lightGBM model is called to sequentially check the importance of model features with the importance type 'split' and the importance type 'gain', as shown in the following code: df['impt_s']=model_lgb.feature_importance(importance_type='split') df['impt_g']=model_lgb.feature_importance(importance_type='gain') S32: Sort the importance of model features with the number of splits as the importance type and the gain as the importance type, respectively.

[0055] Specifically, the importance of model features with the number of splits as the importance type and the gain as the importance type can be sorted in descending order.

[0056] S33: Output the model feature importance of the number of splits and the model feature importance of the gain for the top N sorted items, respectively, to obtain significant features with two feature importance types: number of splits and gain.

[0057] Specifically, in this embodiment of the invention, the importance of the model features of the gain of the top 10 ranked items and the importance of the model features of the number of splits of the top 10 ranked items are output respectively.

[0058] The code to output the feature importance of the top 10 gain values ​​is as follows: gain_top10=df.sort_values('impt_g',ascending=False).iloc[0:10,:] The code that outputs the feature importance of the model features based on the number of splits in the top 10 sorted items is as follows: split_top10=df.sort_values('impt_s',ascending=False).iloc[0:10,:] Figure 4 This is a diagram illustrating a method for obtaining root node features according to an embodiment of the present invention.

[0059] Furthermore, in the aforementioned dataset modeling method, the root node of each tree in the first LightGBM model is iterated through, and the root node features are deduplicated. The deduplicated root node features are then summarized and combined with... Figure 4 It includes three steps: S41 to S43. S41: Extract the tree structure information of each tree in the first lightGBM model.

[0060] Specifically, in this embodiment of the invention, the code for extracting tree structure information is as follows: tree_info_list = tree_info_dict['tree_info'] S42: Traverse and summarize the root node of each tree.

[0061] Specifically, in this embodiment of the invention, the code for traversing and summarizing the root nodes of each tree is as follows: root_node=[] for i in range(0,len(tree_info_list)): #key is the root node number of each tree. key=tree_info_list[i]['tree_structure']['split_feature'] #Find the corresponding value from the feature dictionary feature=tree_info_dict['feature_names'][key] root_node.append(feature) S43: Remove duplicates from the summarized root nodes and arrange them in the original order to obtain the root node features.

[0062] Specifically, in this embodiment of the invention, the code for obtaining the root node features by deduplicating the summarized root nodes and arranging them in their original order is as follows: root_node_unique= sorted(set(root_node),key=root_node.index) print(root_node_unique) Furthermore, in the above-mentioned dataset modeling method, key features are selected by analyzing the main features, salient features, and deduplicated root node features, and the weights of the key features are set, including: Key features are selected based on the overlap between the main features, salient features, and the root node features after deduplication, and the weights of the key features are set.

[0063] Specifically, in this embodiment of the invention, if a feature appears in the ranges of the main features, significant features, and deduplicated root node features, it indicates that the feature is very important in the main features, significant features, and deduplicated root node features. If the Pearson test shows high correlation and the significant features within the first lightGBM model are also present in the root node of the first lightGBM model, then the overlap is high. If the feature appears in two sets of feature ranges in the main features, significant features, and deduplicated root node features, the overlap is medium. If the feature appears in only one set of the main features, significant features, and deduplicated root node features, the overlap is low. Key features and corresponding weights are selected based on the overlap of the three sets of features. Higher overlap makes it easier to select, and the corresponding weight will also be higher.

[0064] Furthermore, in the above-mentioned modeling method for a dataset, the scoring is calculated based on key features and their weights, including: The key features are preprocessed sequentially, followed by z-standardization and 0-1 standardization to obtain the standardized results. The score is obtained by multiplying the standardized result by the corresponding weight.

[0065] Specifically, in this embodiment of the invention, key features are multiplied by corresponding weights, and the scoring range is between -100 and 100.

[0066] Furthermore, in the above-mentioned modeling method for a dataset, determining the input parameter samples based on the scores includes: The input parameter samples are determined based on the score and the magnitude of the second preset threshold.

[0067] Specifically, for example, based on the score distribution, 50 points is used as the second preset threshold, and samples with scores above 50 points are selected as input samples for the second lightGBM modeling, totaling 1.3 million. Of the original 10 million samples, 8.7 million samples were filtered out after pre-filtering, with a screening rate of 87%. The filtered samples were almost all negative samples with an excessively high proportion. Pre-filtering reduced the proportion of negative samples.

[0068] Device Examples Figure 5 A diagram of a dataset modeling apparatus provided in an embodiment of the present invention.

[0069] A second aspect of the invention also proposes a modeling apparatus for datasets, combined with Figure 5 ,include: First modeling module 51: Used to build a business wide table for the dataset, perform Pearson test on all fields in the business wide table to obtain the main features, and perform the first lightGBM modeling based on the main features to obtain the first lightGBM model.

[0070] Specifically, in this embodiment of the invention, a business wide table is established, and Pearson test is performed on all fields in the business wide table to obtain the main features. The first modeling module 51 performs the first lightGBM modeling based on the main features to obtain the first lightGBM model.

[0071] The code for the first lightGBM model obtained by the first modeling module 51 based on the main features is as follows: model_lgb=lgb.train(params=params,train_set=lgb_train,num_boost_round=100,valid_sets=lgb_valid,early_stopping_rounds=5) Selection module 52: Used to select salient features from the first lightGBM model based on the importance of two features: number of splits and gain.

[0072] Specifically, in this embodiment of the invention, the selection module 52 selects significant features from the first lightGBM model that are of two important types: the number of splits and the gain.

[0073] Summary Module 53: Used to iterate through the root nodes of each tree in the first lightGBM model, remove duplicate features from the root nodes, and summarize the deduplicated root node features.

[0074] Specifically, in this embodiment of the invention, the summarization module 53 iterates through the root nodes of each tree in the first lightGBM model, removes duplicate features from the root nodes, and summarizes the deduplicated root node features.

[0075] Analysis module 54 and setting module 55 are used to analyze the main features, significant features, and root node features after deduplication, select key features, and set the weights of key features.

[0076] Specifically, in this embodiment of the invention, the analysis module 54 comprehensively analyzes the Pearson test results for high relevance, high feature importance, and overlap of root node features to select key features, and the setting module 55 sets the weights of key features to establish an indicator evaluation system.

[0077] Calculation module 56: Used to calculate scores based on key features and their weights.

[0078] Specifically, in this embodiment of the invention, the calculation module 56 calculates the score based on the key features and the weights of the key features.

[0079] Determine module 57: Used to determine the input parameter sample based on the score.

[0080] Specifically, in this embodiment of the invention, the determining module 57 determines the input parameter samples based on the range of the scores and the size of the second preset threshold, and determines the samples whose score range is greater than the second preset threshold as input parameter samples.

[0081] Second modeling module 58: used to perform a second lightGBM modeling based on the input sample and the main features corresponding to the input sample to obtain the second lightGBM model.

[0082] Specifically, in this embodiment of the invention, a pre-filtering method is adopted, which can effectively filter out samples with low probability to obtain input parameters. The second modeling module 58 only uses the input parameters with higher probability to perform a second lightGBM modeling to obtain a second lightGBM model, which reduces the total number of samples that need to be predicted, allowing the model to focus more on high-probability users for calculation and derivation, and playing a strong auxiliary role in the model.

[0083] A third aspect of the present invention also provides an electronic device comprising: a processor and a memory; The processor executes a modeling method for a dataset, such as one of the above, by calling programs or instructions stored in memory.

[0084] In a fourth aspect, the present invention also provides a computer-readable storage medium that stores a program or instructions that cause a computer to perform a modeling method for a dataset as described in any of the preceding claims.

[0085] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention.

[0086] like Figure 6 As shown, the electronic device includes at least one processor 601, at least one memory 602, and at least one communication interface 603. The various components of the electronic device are coupled together via a bus system 604. The communication interface 603 is used for information transmission with external devices. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 604.

[0087] It is understood that the memory 602 in this embodiment may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0088] In some implementations, memory 602 stores elements such as executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0089] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. A program implementing any method in the dataset modeling method provided in this embodiment of the invention can be included in the application programs.

[0090] In this embodiment of the invention, the processor 601 executes the steps of various embodiments of the dataset modeling method provided in this embodiment of the invention by calling the program or instructions stored in the memory 602, specifically, the program or instructions stored in the application program.

[0091] A business wide table is created for the dataset. Pearson test is performed on all fields in the business wide table to obtain the main features. The first lightGBM model is obtained by performing the first lightGBM model based on the main features. From the first lightGBM model, significant features of importance were selected based on two characteristics: the number of splits and the gain. Iterate through the root nodes of each tree in the first lightGBM model, remove duplicate features from the root nodes, and summarize the deduplicated root node features. Analyze the main features, significant features, and root node features after deduplication to select key features, and set the weights of the key features; Scores are calculated based on key features and their weights. The input sample is determined based on the scores; The second lightGBM model is obtained by performing a second lightGBM modeling based on the input parameter samples and their corresponding main features.

[0092] Any method in the dataset modeling method provided in this embodiment of the invention can be applied to, or implemented by, processor 601. Processor 601 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of processor 601 or by instructions in software form. Processor 601 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0093] The steps of any method in the dataset modeling method provided in this embodiment of the invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software units in the decoding processor. The software units can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 602, and processor 601 reads the information in memory 602 and combines it with its hardware to complete the steps of the method.

[0094] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0095] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention. All such modifications and variations fall within the scope defined by the appended claims. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0097] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for modeling a dataset, characterized in that, include: A business wide table for the dataset is established. Pearson test is performed on all fields in the business wide table to obtain the main features. The first lightGBM model is obtained by performing the first lightGBM model based on the main features. From the first lightGBM model, salient features of importance are selected based on two feature importance types: number of splits and gain. Iterate through the root node of each tree in the first lightGBM model, remove duplicate features from the root node, and summarize the deduplicated root node features. Analyze the main features, the significant features, and the deduplicated root node features to select key features, and set the weights of the key features; A score is calculated based on the key features and their weights. The input parameter sample is determined based on the aforementioned score; Based on the input parameter samples and the main features corresponding to the input parameter samples, a second lightGBM model is obtained by performing a second lightGBM modeling. From the first lightGBM model, salient features of importance based on two feature importance types: number of splits and gain are selected, including: Call the feature importance attribute of the first lightGBM model and check the importance of model features with the importance type being number of splits and the importance type being gain. The importance of model features is ranked separately for the number of splits and the gain. Output the model feature importance of the number of splits and the model feature importance of the gain for the top N sorted items, respectively, to obtain salient features with two feature importance types: number of splits and gain. The analysis of the main features, the salient features, and the deduplicated root node features selects key features and sets the weights of the key features, including: Key features are selected and weights are set based on the overlap between the main features, the salient features, and the deduplicated root node features.

2. The dataset modeling method according to claim 1, characterized in that, The business wide table used to build the dataset is used to perform a Pearson test on all fields in the business wide table to obtain the main features, including: Based on the business situation and the corresponding dataset, create a wide business table containing multiple independent variable fields and one dependent variable field. Perform Pearson tests on all fields of the business wide table to analyze the Pearson correlation coefficient of each field with the dependent variable; Fields with a Pearson correlation coefficient greater than a first preset threshold are selected as the main features.

3. The method for modeling a dataset according to claim 1, characterized in that, The process iterates through the root nodes of each tree in the first lightGBM model, removes duplicate features from the root nodes, and summarizes the deduplicated root node features, including: Extract the tree structure information of each tree in the first lightGBM model; Traverse and summarize the root node of each tree; The root nodes are deduplicated after being summarized and arranged in their original order to obtain the root node features.

4. The dataset modeling method according to claim 1, characterized in that, The calculation of the score based on the key features and their weights includes: The key features are preprocessed sequentially, followed by z-standardization and 0-1 standardization to obtain the standardized result. The score is obtained by multiplying the standardized result by the corresponding weight.

5. The method for modeling a dataset according to claim 1, characterized in that, The process of determining the input parameter sample based on the score includes: The input parameter samples are determined based on the magnitude of the score and the second preset threshold.

6. A modeling apparatus for a dataset, characterized in that, include: First modeling module: Used to build a business wide table of the dataset, perform Pearson test on all fields in the business wide table to obtain the main features, and perform the first lightGBM modeling based on the main features to obtain the first lightGBM model; Selection module: used to select significant features from the first lightGBM model with importance based on two feature importance types: number of splits and gain. The summary module is used to iterate through the root nodes of each tree in the first lightGBM model, remove duplicate features from the root nodes, and summarize the deduplicated root node features. Analysis module and setting module: used to analyze the main features, the significant features, and the deduplicated root node features to select key features, and set the weights of the key features; Calculation module: used to calculate a score based on the key features and their weights; Determining module: used to determine the input parameter sample based on the score; The second modeling module is used to perform a second lightGBM modeling based on the input parameter samples and the main features corresponding to the input parameter samples to obtain a second lightGBM model. From the first lightGBM model, salient features of importance based on two feature importance types: number of splits and gain are selected, including: Call the feature importance attribute of the first lightGBM model and check the importance of model features with the importance type being number of splits and the importance type being gain. The importance of model features is ranked separately for the number of splits and the gain. Output the model feature importance of the number of splits and the model feature importance of the gain for the top N sorted items, respectively, to obtain salient features with two feature importance types: number of splits and gain. The analysis of the main features, the salient features, and the deduplicated root node features selects key features and sets the weights of the key features, including: Key features are selected and weights are set based on the overlap between the main features, the salient features, and the deduplicated root node features.

7. An electronic device, characterized in that, include: Processor and memory; The processor executes a dataset modeling method as described in any one of claims 1 to 5 by calling programs or instructions stored in the memory.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform a modeling method for a dataset as described in any one of claims 1 to 5.

Images