Method, device and electronic device for extracting business data from database

Abstract

The present invention discloses a method, device and electronic device for extracting business data from a database, the method comprising: obtaining first data generated in a database according to a business processing process; extracting data features from the first data according to the field type of the first data; using a preset first screening model to screen the data features, using multiple first features that pass the screening and an initial second feature to train a preset second screening model, using the second screening model to screen other features that have not completed the screening, and using all the features that pass the screening as business data. Each time the present invention uses the first combined feature to train the first screening model, it analyzes and determines whether the screening ability of the first screening model is improved, ensuring that the quality of the features that pass the screening is higher, and the training effect after the combination of features is better than that of a single feature, that is, ensuring that the second model ultimately used for screening has a higher screening ability, and improving the overall effect of business data mining.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing, and in particular to a method, device, electronic device and computer-readable medium for extracting business data from a database. Background Art

[0002] Features refer to some useful information extracted from big data. Data feature mining refers to the technology of identifying understandable and applicable information from large-scale data. It has important applications in various fields. For example, a large amount of data is often generated in the process of business processing. Some of the features reflect high-quality business information, which can be mined and utilized as business data.

[0003] Generally speaking, for low-dimensional, easy-to-understand data, manually designed features are often more effective. However, for data with a large number of dimensions or data that cannot be manually seen in terms of their relationship with the target, automated mining is required, that is, a mining system that automatically performs data preprocessing, feature generation, feature screening, and model training generates a large number of deep (complex) features, and then screens them to construct features.

[0004] Therefore, there is a need to provide a technical solution for automatically mining business data to ensure that high-quality business data is mined. Summary of the invention

[0005] The present invention aims to provide a technical solution for automatically mining business data, which can ensure that high-quality business data is mined.

[0006] In order to solve the above technical problems, the first aspect of the present invention proposes a method for extracting business data from a database, comprising: obtaining first data generated in the database according to a business processing process; extracting data features from the first data according to the field type of the first data; using a preset first screening model to screen the data features, specifically comprising: extracting a first feature from a feature group that has not completed screening; combining the first feature with a preset second feature that has passed screening to obtain a first combined feature, and using the first combined feature to train the first screening model; if the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, using the first combined feature to update the second feature, and re-extracting the first feature and training the first screening model, if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, re-extracting the first feature and training the first screening model until the second feature is updated a preset number of times; determining that multiple first features related to the latest second feature have passed screening; using multiple first features that have passed screening and the initial second feature to train a preset second screening model, using the second screening model to screen other features that have not completed screening, and using all features that have passed screening as the business data.

[0007] According to a preferred embodiment of the present invention, before using the preset first screening model to screen the data features, it also includes: using a preset third screening model to screen the data features, the parameter amount of the third screening model is lower than the preset first amount, specifically including: combining multiple features in the feature group with the second feature to obtain multiple second combination features, and using the multiple second combination features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, the features corresponding to the second combination features used for training are removed from the feature group.

[0008] According to a preferred embodiment of the present invention, after using the preset third screening model to screen the data features and before using the preset first screening model to screen the data features, it also includes: using a preset fourth screening model to screen the data features, the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically including: combining multiple features in the feature group with the second feature to obtain multiple third combination features, and using the multiple third combination features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, the features corresponding to the second combination features used for training are removed from the feature group.

[0009] According to a preferred embodiment of the present invention, before using the preset first screening model to screen the data features, it also includes: analyzing the amount of business information reflected by each feature in the data features, and removing features from the data features that reflect business information amounts below a preset level.

[0010] According to a preferred embodiment of the present invention, extracting data features from the first data according to the field type of the first data includes: setting a cutting model of the first data according to the field type of the first data, and retrieving a feature generation function from a preset feature generation library corresponding to the business; performing cutting processing on the first data using the cutting model; and processing the first data using the feature generation function to obtain the data features.

[0011] According to a preferred embodiment of the present invention, before extracting data features from the first data according to the field type of the first data, it also includes: identifying illegal data in the first data and deleting it; and / or identifying a data type in the first data whose occurrence frequency is lower than a preset frequency, and deleting all data of the data type.

[0012] In order to solve the above technical problems, the second aspect of the present invention proposes a device for extracting business data from a database, comprising: a data acquisition module, acquiring first data generated in a database according to a business processing process; a feature extraction module, extracting data features from the first data according to the field type of the first data; a first screening module, using a preset first screening model to screen the data features, specifically comprising: extracting a first feature from a feature group that has not completed screening; combining the first feature with a preset second feature that has passed screening to obtain a first combined feature, and using the first combined feature to train the first screening model; if the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, using the first combined feature to update the second feature, and re-extracting the first feature and training the first screening model, if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, re-extracting the first feature and training the first screening model until the second feature update number reaches a preset number; determining that multiple first features related to the latest second feature have passed screening; a second screening module, using multiple first features that have passed screening and the initial second feature to train a preset second screening model, using the second screening model to screen other features that have not completed screening, and using all features that have passed screening as the business data.

[0013] According to a preferred embodiment of the present invention, it also includes: a third screening module, before using the preset first screening model to screen the data features, using a preset third screening model to screen the data features, the parameter quantity of the third screening model is lower than the preset first quantity, specifically including: combining multiple features in the feature group with the second feature to obtain multiple second combination features, and using the multiple second combination features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, the features corresponding to the second combination features used for training are removed from the feature group.

[0014] According to a preferred embodiment of the present invention, it also includes: a fourth screening module, after the data features are screened using the preset third screening model and before the data features are screened using the preset first screening model, the data features are screened using a preset fourth screening model, and the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically including: combining multiple features in the feature group with the second feature to obtain multiple third combined features, and using the multiple third combined features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, the features corresponding to the second combined features used for training are removed from the feature group.

[0015] According to a preferred embodiment of the present invention, it also includes: a fifth screening module, before using the preset first screening model to screen the data features, analyzing the amount of business information reflected by each feature in the data features, and removing features that reflect business information amounts below a preset level from the data features.

[0016] According to a preferred embodiment of the present invention, the feature extraction module sets a cutting model of the first data according to the field type of the first data, and calls a feature generation function from a preset feature generation library corresponding to the business, uses the cutting model to perform cutting processing on the first data, and uses the feature generation function to process the first data to obtain the data feature.

[0017] According to a preferred embodiment of the present invention, it also includes: a preprocessing module, which identifies and deletes illegal data in the first data before extracting data features from the first data according to the field type of the first data; and / or identifies a data type in the first data whose occurrence frequency is lower than a preset frequency, and deletes all data of the data type.

[0018] In order to solve the above technical problem, the third aspect of the present invention proposes an electronic device, which includes a processor and a memory storing computer-executable instructions, and the computer-executable instructions enable the processor to execute the above method when executed.

[0019] In order to solve the above technical problem, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, the above method is implemented.

[0020] In the technical solution of the present invention, after obtaining data from the database and extracting data features, each time the first combined features are used to train the first screening model, it is analyzed and determined whether the screening ability of the first screening model is improved, and feature screening is performed on this basis to ensure that the features passing the screening are of higher quality, and the training effect of the combination of features is better than that of a single feature, that is, it is ensured that the second model ultimately used for screening has a higher screening ability, thereby improving the overall quality of the screened business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a method for extracting business data from a database according to an embodiment of the present invention;

[0023] Figure 2 is a flow chart of a method for extracting business data from a database according to an embodiment of the present invention;

[0024] Figure 3 is a flow chart of a method for extracting business data from a database according to an embodiment of the present invention;

[0025] Figure 4 is a block diagram of an apparatus for extracting business data from a database according to an embodiment of the present invention;

[0026] Figure 5 is a block diagram of an apparatus for extracting business data from a database according to an embodiment of the present invention;

[0027] Figure 6 is a block diagram of an electronic device according to an embodiment of the present invention;

[0028] Figure 7is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0030] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0031] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0032] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0033] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0035] In the technical solution of the present invention, first data generated according to a business processing process is obtained from a database; data features are extracted from the first data according to the field type of the first data; and the data features are screened using a preset first screening model, specifically including: extracting a first feature from a feature group that has not been screened; combining the first feature with a preset second feature that has passed the screening to obtain a first combined feature, and using the first combined feature to train a first screening model; if the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, using the first combined feature to update the second feature, and re-extracting the first feature and training the first screening model; if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, re-extracting the first feature and training the first screening model. The first screening model is trained until the second feature is updated a preset number of times; it is determined that multiple first features related to the latest second feature have passed the screening; the multiple first features that have passed the screening and the initial second feature are used to train the preset second screening model, and the second screening model is used to screen other features that have not completed the screening, and all the features that have passed the screening are used as business data. In the technical solution of the present invention, each time the first combined feature is used to train the first screening model, it is analyzed and determined whether the screening ability of the first screening model is improved, and feature screening is performed based on this, ensuring that the features that pass the screening are of higher quality, and the training effect after the combination of features is better than that of a single feature, that is, it is ensured that the second model ultimately used for screening has a higher screening ability, thereby improving the overall effect of business data mining.

[0036] like Figure 1 As shown, in one embodiment of the present invention, a method for extracting business data from a database is provided, the method comprising:

[0037] Step S110, obtaining first data generated in a database according to a business processing process.

[0038] In this embodiment, there is no restriction on the service type. The first data generated in the service process has a lot of content and reflects different service qualities. Therefore, it is necessary to extract high-quality service data.

[0039] Step S120: extracting data features from the first data according to the field type of the first data.

[0040] Step S130, using a preset first screening model to screen the data features, specifically as follows Figure 2 As shown, including:

[0041] Step S210: extracting the first feature from the feature group that has not been screened.

[0042] Step S220: Combine the first feature with the preset second feature that has passed screening to obtain a first combined feature, and use the first combined feature to train a first screening model.

[0043] Step S230: if the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, use the first combined feature to update the second feature, re-extract the first feature and train the first screening model; if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, re-extract the first feature and train the first screening model until the second feature update number reaches a preset number.

[0044] Step S240, determining that multiple first features related to the latest second feature have passed screening.

[0045] In this embodiment, it is assumed that the feature cluster that has been mined is A and the feature to be screened is B. i (i=0,1,2,…). In this embodiment, the GBDT (a type of machine learning algorithm based on gradient booting decision tree (regression tree), commonly used to construct big data prediction models, common ones include XGBoost, lightgbm, etc.) algorithm can be used to train the model. However, through practice, it is found that the use of the GBDT algorithm in the prior art has defects, and the effect of feature A + feature B is not necessarily better than using feature A alone. The technical solution of this embodiment is used to overcome this defect. The specific operation method is: a) Select B i , use A+B i Combine and train the GBDT model. b) If A+B i The effect is better than A. i is classified as A, and Bi is eliminated from B; if A+B i The effect is worse than A, so we can directly remove B from B. i ; c) Repeat a) and b) until the number of features included in A reaches a certain threshold. Through the above steps, multiple B iAfter combining with A, the effect is still improved. The parameter quantity of the GBDT model here is higher than a certain threshold (for example, the number of trees in the model is higher than a certain threshold), which is conducive to solving the defects of the above-mentioned GBDT algorithm.

[0046] Step S140, using the multiple first features that have passed the screening and the initial second features to train a preset second screening model, using the second screening model to screen other features that have not completed the screening, and using all the features that have passed the screening as business data.

[0047] In this embodiment, all the filtered B features are combined with the original A features for final model training, and the model is used to filter other features to obtain the final mined features as business data.

[0048] According to the technical solution of this embodiment, after obtaining data from the database and extracting data features, each time the first combined features are used to train the first screening model, it is analyzed and determined whether the screening ability of the first screening model is improved, and feature screening is performed based on this to ensure that the features that pass the screening are of higher quality, and the training effect of the combination of features is better than that of a single feature, which ensures that the second model ultimately used for screening has a higher screening ability, thereby improving the overall quality of the screened business data.

[0049] like Figure 3 As shown, in one embodiment of the present invention, a method for extracting business data from a database is provided, the method comprising:

[0050] Step S310, obtaining first data generated in a database according to a business processing process.

[0051] Step S320, identifying illegal data in the first data and deleting it; and / or identifying a data type in the first data whose occurrence frequency is lower than a preset frequency, and deleting all data of the data type.

[0052] In this embodiment, automatic data preprocessing is performed according to the field type. For example, assuming that the data field is a numeric type, the automatic data preprocessing functions called according to the field are: finding illegal data (non-numeric) and replacing it with NULL, counting the mean and variance of all data in this field, and performing filtering operations based on the mean and variance; assuming that the data field is a categorizable text type (for example, gender, province), the automatic data preprocessing functions are: finding illegal data and replacing it, counting all classifications of this field, eliminating classifications with only very few samples, and matching classified texts with discrete numerical values; assuming that the data field is a sortable text type (for example, text types that can be sorted, such as strength, size, etc.), the preprocessing function called is: finding illegal data and replacing it, counting all sorting elements, eliminating classifications with only very few samples, and matching classified texts with sortable discrete numerical values ​​(the size of the discrete numerical values ​​at this time is meaningful, while the relative size of the discrete numerical values ​​of the categorizable text is meaningless).

[0053] Step S330, according to the field type of the first data, set the cutting model of the first data, and call the feature generation function from the preset feature generation library corresponding to the business; use the cutting model to cut the first data; use the feature generation function to process the first data to obtain data features.

[0054] In this embodiment, according to the field type, the processed data is subjected to deep feature generation. The deep feature generation consists of two stages. The first is to segment the data using the data cube (a way of cutting data, for example, selecting data from 10 am to 3 pm, selecting data of male gender, etc.), and the second is to call the feature generation function in the feature generation library according to the field type to generate a large number of deep features. The two stages work together to generate a large number of data features to be screened. For example, for data with a numeric field type, data cube cutting is performed (for example, selecting data from 10 am to 2 pm, etc.), and then the numeric feature generation function is called. The following are the feature generation functions available for some numeric data:

[0055] Time-independent functions: Min, Max, median, mean, variance, stand deviation, etc.

[0056] Time-dependent but time-order-independent functions: min, max, mean, etc. Time-dependent and time-order-dependent functions: kurtosis, accumulated difference, etc.

[0057] For data whose field type is categorizable text, data cube cutting is performed, and some feature generation functions can be called:

[0058] Time-independent functions: classification

[0059] Time-related functions: entropy, std of entropy, number of occupied categories, etc.

[0060] This stage requires a lot of computing power, so it is generally carried out on the Hadoop platform (a platform for storing and processing big data under distributed computing cluster conditions).

[0061] Step S340, analyzing the amount of business information reflected by each feature in the data features, and removing features reflecting a business information amount lower than a preset level from the data features.

[0062] From here on, the large number of features generated will be screened to determine which data features will be used for model training (such as the GBDT algorithm). Conventional data screening, such as correlation calculation, variance screening, etc., are not designed for the GBDT algorithm and usually result in poor training results. Therefore, a screening method for the GBDT algorithm is needed. Generally speaking, before data mining, there will be data features that have been mined. Since how the features are combined has a greater impact on the results of the GBDT algorithm, it is necessary to ensure that the newly mined features are combined with the already mined features to still achieve good results on the GBDT algorithm.

[0063] In this embodiment, a simple screening is first performed: the variance of a certain feature, std(feature), is calculated, and features with smaller standard deviations are screened out (the smaller the standard deviation, the smaller the amount of information).

[0064] Step S350, using a preset third screening model to screen the data features, wherein the parameter quantity of the third screening model is lower than the preset first quantity, specifically includes:

[0065] Combining multiple features in the feature group with the second feature to obtain multiple second combined features, and using the multiple second combined features to train the third screening model respectively;

[0066] If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

[0067] In this embodiment, A is respectively connected to each B i Combine them and send them to the GBDT model with fewer parameters (for example, fewer trees) for training, and select the B with higher results. i, as the selected features. The parameters with lighter parameters are selected here because the GBDT algorithm usually requires a lot of computing power, and the number of features to be screened is huge, so this step is needed for preliminary GBDT screening. Each combination must be trained once, and then the one with better effect B is selected to enter the next round.

[0068] Step S360, using a preset fourth screening model to screen the data features, wherein the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically includes:

[0069] Combining multiple features in the feature group with the second feature to obtain multiple third combined features, and using the multiple third combined features to train the third screening model respectively;

[0070] If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

[0071] In this embodiment, the GBDT parameters selected here are relatively large, so the number of features to be screened has been greatly reduced, so the screening time is bearable.

[0072] Step S370: Use the preset first screening model to screen the data features.

[0073] Step S380, use the multiple first features that have passed the screening and the initial second features to train a preset second screening model, use the second screening model to screen other features that have not completed the screening, and use all the features that have passed the screening as business data.

[0074] According to the technical solution of this embodiment, automated mining from original first data to effective business data is realized, which can greatly improve efficiency and avoid complicated manual feature design processes. The technical solution of this embodiment optimizes the feature screening process of GBDT so that it can screen out more effective business data.

[0075] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments are implemented as a program executed by a data processing device (including a computer), i.e., a computer program. When the computer program is executed, the above method provided by the present invention can be implemented. Moreover, the computer program can be stored in a computer-readable storage medium, which can be a readable storage medium such as a disk, an optical disk, a ROM, a RAM, or a storage array composed of multiple storage media, such as a disk or a tape storage array. The storage medium is not limited to centralized storage, and it can also be distributed storage, such as cloud storage based on cloud computing.

[0076] The following describes an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. The details described in the apparatus embodiment of the present invention should be regarded as supplementary to the above method embodiment; details not disclosed in the apparatus embodiment of the present invention can be implemented with reference to the above method embodiment.

[0077] like Figure 4 As shown, in one embodiment of the present invention, a device for extracting business data from a database is provided, and the device includes:

[0078] The data acquisition module 410 acquires first data generated in a database according to a business processing process.

[0079] In this embodiment, there is no restriction on the service type. The first data generated in the service process has a lot of content and reflects different service qualities. Therefore, it is necessary to extract high-quality service data.

[0080] The feature extraction module 420 extracts data features from the first data according to the field type of the first data.

[0081] The first screening module 430 uses a preset first screening model to screen data features, including: extracting a first feature from a feature group that has not been screened. Combining the first feature with a preset second feature that has passed the screening to obtain a first combined feature, and using the first combined feature to train the first screening model. If the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, the second feature is updated using the first combined feature, and the first feature is re-extracted and the first screening model is trained. If the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, the first feature is re-extracted and the first screening model is trained until the second feature is updated a preset number of times. Determine whether multiple first features related to the latest second feature have passed the screening.

[0082] In this embodiment, it is assumed that the feature cluster that has been mined is A and the feature to be screened is B. i (i=0,1,2,…). In this embodiment, the GBDT (a type of machine learning algorithm based on gradient booting decision tree (regression tree), commonly used to construct big data prediction models, common ones include XGBoost, lightgbm, etc.) algorithm can be used to train the model. However, through practice, it is found that the use of the GBDT algorithm in the prior art has defects, and the effect of feature A + feature B is not necessarily better than using feature A alone. The technical solution of this embodiment is used to overcome this defect. The specific operation method is: a) Select B i , use A+B i Combine and train the GBDT model. b) If A+B iThe effect is better than A. i is classified as A, and Bi is eliminated from B; if A+B i The effect is worse than A, so we can directly remove B from B. i ; c) Repeat a) and b) until the number of features included in A reaches a certain threshold. Through the above steps, multiple B i After combining with A, the effect is still improved. The parameter quantity of the GBDT model here is higher than a certain threshold (for example, the number of trees in the model is higher than a certain threshold), which is conducive to solving the defects of the above-mentioned GBDT algorithm.

[0083] The second screening module 440 uses the multiple first features that have passed the screening and the initial second features to train a preset second screening model, uses the second screening model to screen other features that have not completed the screening, and uses all the features that have passed the screening as business data.

[0084] In this embodiment, all the filtered B features are combined with the original A features for final model training, and the model is used to filter other features to obtain the final mined features as business data.

[0085] According to the technical solution of this embodiment, each time the first combined feature is used to train the first screening model, it is analyzed and determined whether the screening ability of the first screening model is improved, and feature screening is performed based on this, ensuring that the quality of the features passing the screening is higher, and the training effect of the combination of features is better than that of a single feature, which ensures that the second model ultimately used for screening has a higher screening ability, thereby improving the overall effect of business data mining.

[0086] like Figure 5 As shown, in one embodiment of the present invention, a device for extracting business data from a database is provided, and the device includes:

[0087] The data acquisition module 510 acquires first data generated in a database according to a business processing process.

[0088] The pre-processing module 520 identifies illegal data in the first data and deletes it; and / or identifies a data type in the first data whose occurrence frequency is lower than a preset frequency, and deletes all data of the data type.

[0089] In this embodiment, automatic data preprocessing is performed according to the field type. For example, assuming that the data field is a numeric type, the automatic data preprocessing functions called according to the field are: finding illegal data (non-numeric) and replacing it with NULL, counting the mean and variance of all data in this field, and performing filtering operations based on the mean and variance; assuming that the data field is a categorizable text type (for example, gender, province), the automatic data preprocessing functions are: finding illegal data and replacing it, counting all classifications of this field, eliminating classifications with only very few samples, and matching classified texts with discrete numerical values; assuming that the data field is a sortable text type (for example, text types that can be sorted, such as strength, size, etc.), the preprocessing function called is: finding illegal data and replacing it, counting all sorting elements, eliminating classifications with only very few samples, and matching classified texts with sortable discrete numerical values ​​(the size of the discrete numerical values ​​at this time is meaningful, while the relative size of the discrete numerical values ​​of the categorizable text is meaningless).

[0090] The feature extraction module 530 sets a cutting model of the first data according to the field type of the first data, and calls a feature generation function from a preset feature generation library corresponding to the business; uses the cutting model to cut the first data; and uses the feature generation function to process the first data to obtain data features.

[0091] In this embodiment, according to the field type, the processed data is subjected to deep feature generation. The deep feature generation consists of two stages. The first is to segment the data using the data cube (a way of cutting data, for example, selecting data from 10 am to 3 pm, selecting data of male gender, etc.), and the second is to call the feature generation function in the feature generation library according to the field type to generate a large number of deep features. The two stages work together to generate a large number of data features to be screened. For example, for data with a numeric field type, data cube cutting is performed (for example, selecting data from 10 am to 2 pm, etc.), and then the numeric feature generation function is called. The following are the feature generation functions available for some numeric data:

[0092] Time-independent functions: Min, Max, median, mean, variance, stand deviation, etc.

[0093] Time-dependent but time-order-independent functions: min, max, mean, etc.

[0094] Time-related and time-order related functions: kurtosis, accumulated difference, etc.

[0095] For data whose field type is categorizable text, data cube cutting is performed, and some feature generation functions can be called:

[0096] Time-independent functions: classification

[0097] Time-related functions: entropy, std of entropy, number of occupied categories, etc.

[0098] This stage requires a lot of computing power, so it is generally carried out on the Hadoop platform (a platform for storing and processing big data under distributed computing cluster conditions).

[0099] The fifth screening module 540 analyzes the amount of business information reflected by each feature in the data features, and removes features reflecting business information amounts lower than a preset level from the data features.

[0100] From here on, the large number of features generated will be screened to determine which data features will be used for model training (such as the GBDT algorithm). Conventional data screening, such as correlation calculation, variance screening, etc., are not designed for the GBDT algorithm and usually result in poor training results. Therefore, a screening method for the GBDT algorithm is needed. Generally speaking, before data mining, there will be data features that have been mined. Since how the features are combined has a greater impact on the results of the GBDT algorithm, it is necessary to ensure that the newly mined features are combined with the already mined features to still achieve good results on the GBDT algorithm.

[0101] In this embodiment, a simple screening is first performed: the variance of a certain feature, std(feature), is calculated, and features with smaller standard deviations are screened out (the smaller the standard deviation, the smaller the amount of information).

[0102] The third screening module 550 uses a preset third screening model to screen the data features, and the parameter quantity of the third screening model is lower than the preset first quantity, specifically including:

[0103] Combining multiple features in the feature group with the second feature to obtain multiple second combined features, and using the multiple second combined features to train the third screening model respectively;

[0104] If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

[0105] In this embodiment, A is respectively connected to each B i Combine them and send them to the GBDT model with fewer parameters (for example, fewer trees) for training, and select the B with higher results. i, as the selected features. The parameters with lighter parameters are selected here because the GBDT algorithm usually requires a lot of computing power, and the number of features to be screened is huge, so this step is needed for preliminary GBDT screening. Each combination must be trained once, and then the one with better effect B is selected to enter the next round.

[0106] The fourth screening module 560 uses a preset fourth screening model to screen the data features, and the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically including:

[0107] Combining multiple features in the feature group with the second feature to obtain multiple third combined features, and using the multiple third combined features to train the third screening model respectively;

[0108] If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

[0109] In this embodiment, the GBDT parameters selected here are relatively large, so the number of features to be screened has been greatly reduced, so the screening time is bearable.

[0110] The first screening module 570 uses a preset first screening model to screen data features.

[0111] The second screening module 580 uses the multiple first features that have passed the screening and the initial second features to train a preset second screening model, uses the second screening model to screen other features that have not completed the screening, and uses all the features that have passed the screening as business data.

[0112] According to the technical solution of this embodiment, automated mining from original first data to effective business data is realized, which can greatly improve efficiency and avoid complicated manual feature design processes. The technical solution of this embodiment optimizes the feature screening process of GBDT so that it can screen out more effective business data.

[0113] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules.

[0114] The electronic device embodiment of the present invention is described below, and the electronic device can be regarded as a physical implementation of the method and device embodiments of the present invention described above. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0115] Figure 6 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0116] like Figure 7 As shown, the electronic device 200 of this exemplary embodiment is in the form of a general data processing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0117] The storage unit 220 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 210, so that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 210 may perform the following steps: Figures 1 to 3 Steps shown.

[0118] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203. The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0119] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0120] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, Bluetooth device, etc.) so that a user can interact with the electronic device 200 via these external devices 300, and / or enable the electronic device 200 to communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 250, and may also be performed through a network adapter 260 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 260 may communicate with other modules of the electronic device 200 through the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0121] Figure 7 Schematic diagram of a computer readable medium embodiment of the present invention. Figure 7As shown, the computer program can be stored on one or more computer readable media. The computer readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. When the computer program is executed by one or more data processing devices, the computer-readable medium can implement the above method of the present invention, namely: obtaining first data generated in a database according to a business processing process; extracting data features from the first data according to the field type of the first data; using a preset first screening model to screen the data features, specifically including: extracting a first feature from a feature group that has not completed screening; combining the first feature with a preset second feature that has passed screening to obtain a first combined feature, and using the first combined feature to train the first screening model; if the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, using the first combined feature to update the second feature, and re-extracting the first feature and training the first screening model; if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, re-extracting the first feature and training the first screening model until the number of updates of the second feature reaches a preset number; determining that multiple first features related to the latest second feature have passed screening; using multiple first features that have passed screening and the initial second feature to train a preset second screening model, using the second screening model to screen other features that have not completed screening, and using all features that have passed screening as business data.

[0122] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a data processing device (which can be a personal computer, a server, or a network device, etc.) to perform the above method according to the present invention.

[0123] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0124] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0125] In summary, the present invention can be implemented by a method, apparatus, electronic device or computer-readable medium that executes a computer program. In practice, a general data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all functions of the present invention.

[0126] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically mining business data, characterized in that: include: Acquire first data generated in a database according to a business processing process; Extracting data features from the first data according to the field type of the first data, including: setting a cutting model for the first data according to the field type of the first data, and retrieving a feature generation function from a preset feature generation library corresponding to the business; performing cutting processing on the first data using the cutting model; processing the first data using the feature generation function to obtain the data features; and, before using the preset first screening model to screen the data features, analyzing the amount of business information reflected by each feature in the data features, and removing data features reflecting a business information amount lower than a preset level from the data features; Use the preset first screening model to screen the data features, including: Extract the first feature from the feature group that has not been screened; Combining the first feature with a preset second feature that has passed screening to obtain a first combined feature, and using the first combined feature to train the first screening model; If the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, the second feature is updated using the first combined feature, and the first feature is re-extracted and the process of obtaining the first combined feature and training the first screening model is repeated; if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, the first feature is re-extracted and the process of obtaining the first combined feature and training the first screening model is repeated until the second feature is updated a preset number of times and the number of features updated to the second feature reaches a quantity threshold; wherein the model parameter amount of the first screening model is higher than a threshold; Determining that the latest plurality of first features related to the second feature pass screening; Using the multiple first features that have passed the screening and the initial second features to train a preset second screening model, using the second screening model to screen other features that have not completed the screening, and using all the features that have passed the screening as the business data; Before using the preset first screening model to screen the data features, a preset third screening model is also used to perform preliminary screening on the data features, wherein the parameter quantity of the third screening model is lower than the preset first quantity to ensure a large amount of computing power of the model, specifically including: Combine multiple features in the feature group with the second feature to obtain multiple second combination features, and use the multiple second combination features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, remove the features corresponding to the second combination features used for training from the feature group.

2. The method according to claim 1, characterized in that After the data features are screened using the preset third screening model, and before the data features are screened using the preset first screening model, the method further includes: The data features are screened using a preset fourth screening model, wherein the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically including: Combining a plurality of features in the feature group with the second feature to obtain a plurality of third combined features, and using the plurality of third combined features to train the third screening model respectively; If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

3. The method according to claim 1, characterized in that The cutting process of the first data includes performing data cube cutting on the data whose field type is numerical; Retrieving a feature generation function from a preset feature generation library corresponding to the business includes calling a numerical feature generation function.

4. The method according to claim 1, characterized in that: Before extracting data features from the first data according to the field type of the first data, the method further includes: Identify illegal data in the first data and delete it; and / or identify a data type in the first data whose occurrence frequency is lower than a preset frequency, and delete all data of the data type.

5. A device for automatically mining business data, characterized in that: include: A data acquisition module, acquiring first data generated in a database according to a business processing process; A feature extraction module extracts data features from the first data according to the field type of the first data, including: setting a segmentation model for the first data according to the field type of the first data, and calling a feature generation function from a preset feature generation library corresponding to the business, performing segmentation processing on the first data using the segmentation model, and processing the first data using the feature generation function to obtain the data features; A fifth screening module, before using the preset first screening model to screen the data features, analyzes the amount of business information reflected by each feature in the data features, and removes features reflecting business information amounts lower than a preset level from the data features; The first screening module uses a preset first screening model to screen the data features, specifically including: Extract the first feature from the feature group that has not been screened; Combining the first feature with a preset second feature that has passed screening to obtain a first combined feature, and using the first combined feature to train the first screening model; If the screening ability of the first screening model after training exceeds the screening ability of the first screening model before training, the second feature is updated using the first combination feature, and the first feature is re-extracted to repeatedly obtain the first combination feature and train the first screening model; if the screening ability of the first screening model after training does not exceed the screening ability of the first screening model before training, the first feature is re-extracted to repeatedly obtain the first combination feature and train the first screening model until the second feature is updated a preset number of times and the number of features updated to the second feature reaches a quantity threshold; wherein the model parameter amount of the first screening model is higher than a threshold; Determining that the latest plurality of first features related to the second feature pass screening; A second screening module uses the plurality of first features that have passed the screening and the initial second features to train a preset second screening model, uses the second screening model to screen other features that have not completed the screening, and uses all the features that have passed the screening as the business data; The third screening module uses a preset third screening model to preliminarily screen the data features before using the preset first screening model to screen the data features, and the parameter quantity of the third screening model is lower than the preset first quantity to ensure a large amount of computing power of the model. Specifically, it includes: combining multiple features in the feature group with the second feature to obtain multiple second combination features, and using the multiple second combination features to train the third screening model respectively; if the screening ability of the third screening model after training is lower than the preset ability value, the features corresponding to the second combination features used for training are removed from the feature group.

6. The device according to claim 5, characterized in that Also includes: The fourth screening module, after screening the data features using the preset third screening model and before screening the data features using the preset first screening model, screens the data features using a preset fourth screening model, wherein the parameter quantity of the fourth screening model is higher than the preset second quantity, specifically includes: Combining a plurality of features in the feature group with the second feature to obtain a plurality of third combined features, and using the plurality of third combined features to train the third screening model respectively; If the screening ability of the third screening model after training is lower than the preset ability value, the feature corresponding to the second combination feature used for training is removed from the feature group.

7. The device according to claim 5, characterized in that The cutting process of the first data includes performing data cube cutting on the data whose field type is numerical; Retrieving a feature generation function from a preset feature generation library corresponding to the business includes calling a numerical feature generation function.

8. The device according to claim 5, characterized in that Also includes: A preprocessing module, which identifies and deletes illegal data in the first data before extracting data features from the first data according to the field type of the first data; And / or identify a data type in the first data whose occurrence frequency is lower than a preset frequency, and delete all data of the data type.

9. An electronic device, comprising: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-4.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images