Feature derivation system

Abstract

The invention discloses a feature derivation system, and the system comprises a Python derivation module which is used for constructing a Python derivation logic management system and supporting the automatic processing and storage of data; the sql derivation module is used for supporting a user to directly compile and execute an SQL script on a configuration page; the universal template derivation module is used for defining a data source, a time range, processing logic and a function template and automatically executing a feature derivation process; the guide type derivation module is used for automatically processing data table reading and carrying out feature engineering; and the feature monitoring module is used for ensuring that the model performance is not influenced by data drift, and identifying and solving the data quality problem. The method supports abundant derivation modes, namely, python derivation oriented to python research and development personnel, a sparkSql derivation mode oriented to ETL personnel, a guide derivation mode oriented to data analysis personnel and a universal template derivation mode oriented to business personnel.

Description

Technical Field

[0001] The present invention belongs to the technical field of feature derivation, and in particular is a feature derivation system. Background Art

[0002] Modelers need to conduct a lot of data exploration and data preparation before starting modeling. The variable processing and derivation platform provides modelers with a large amount of prepared derivative data and a solution for the entire life cycle of features. Modelers can use it right away, saving a lot of data exploration and data preparation time, allowing modelers to focus more on the modeling itself.

[0003] At present, there are also some feature derivation solutions in the industry. Some solutions only have SQL derivation methods, and some only have template derivation methods. The derived results are placed in a large wide table, and the derived business process ends. The derived data can only be used during the model development period. The algorithm personnel need to analyze the stability of the derived features themselves, and if they need to be provided for model operation, they must be redeveloped. These solutions have more or less the following shortcomings: 1. The supported derivation methods are relatively single; 2. There is no effective monitoring method; 3. The derived results cannot provide online services. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a feature derivative system that overcomes the above problems or at least partially solves the above problems.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A feature derivation system, the system comprising:

[0007] Python derivative module, used to build Python derivative logic management system, supporting automatic data processing and storage;

[0008] SQL derivative module, which is used to support users to directly write and execute SQL scripts on the configuration page and automatically generate feature data;

[0009] The general template derivation module is used to define data sources, time ranges, processing logic and function templates, and automatically execute the feature derivation process;

[0010] Wizard-style derivation module for automating data table reading and feature engineering, and managing offline feature marts;

[0011] The feature monitoring module is used to protect model performance from data drift, identify and resolve data quality issues, and improve model accuracy by optimizing data quality.

[0012] Optionally, the system further includes a feature market module, and the feature market module includes:

[0013] Configure the basic information of the market and select features;

[0014] After the configuration is completed and saved, the features in the market information are launched as online services;

[0015] Configure the timer execution strategy according to the execution strategy;

[0016] The timer triggers the collection task regularly, and the collection task automatically generates the collection SQL script according to the selected features;

[0017] Submit the spark-submit command through spark-client to the big data platform for execution;

[0018] The big data platform calls back the execution result after the execution is completed.

[0019] Optionally, the python derivative module includes:

[0020] Write Python derivative scripts and perform derivative configuration;

[0021] Configure the timer execution strategy according to the execution strategy;

[0022] Trigger the timer and run the python derivative logic;

[0023] After the Python derivative script is run, the derivative result file is automatically uploaded to HDFS and the storage address is recorded;

[0024] After uploading HDFS, save the HDFS data to Hive.

[0025] Optionally, the sql derivation module includes:

[0026] Perform derivative configuration;

[0027] Configure the timer execution strategy;

[0028] Trigger the timer and run the SQL derivative logic;

[0029] Submit the spark-submit command through the spark-client client to execute the derived logic;

[0030] Use Spark in the script to save the derived results directly into the Hive table.

[0031] Optionally, the general template derivative module includes:

[0032] Configure information according to the page instructions;

[0033] After the configuration is completed and saved, the Cartesian product is automatically arranged and combined according to the query conditions based on the configured information, and the generated data is parsed into executable SQL statements, and the SQL statements are persisted;

[0034] Improve the parsed field information, table information and dimension information saved in the derived results;

[0035] Configure the timer execution strategy according to the execution strategy;

[0036] Trigger the timer and call the general template derived logic;

[0037] Submit the spark-submit command through the spark-client client;

[0038] The derived results are saved synchronously to the hive table configured by the user.

[0039] Optionally, the configuration information according to the page guidance includes:

[0040] Select the original data table information used for the derivation;

[0041] Select the Time Window field and configure the time window;

[0042] Configure the conditions for querying data;

[0043] Configure indicators and calculation templates.

[0044] Optionally, the wizard-style derivation module includes:

[0045] Configure derivative logic;

[0046] The configuration reference has been executed and the execution result is a successfully executed wizard-style derived instance, which is saved;

[0047] Asynchronously execute to obtain the table structure information of the derived results in the derived instance;

[0048] Improve the parsed field information, table information and dimension information saved in the derived results;

[0049] Configure the timer execution strategy according to the user's own execution strategy;

[0050] Trigger the timer and call the wizard-style derivation logic. After the wizard-style service derivation is completed, the system derivation result is configured through the Kafka callback.

[0051] After the configuration system receives the derivative results, it triggers the derivative result saving logic and stores the results in the hive table configured by the user.

[0052] Optionally, the configuration derivation logic includes:

[0053] Select the original data table information used for the derivation;

[0054] Configure data exploration information and click Execute;

[0055] Select feature engineering operators and hyperparameters, and start execution;

[0056] After execution, view the derived result data information.

[0057] Optionally, the feature monitoring module includes:

[0058] Synchronize the configuration derivative information to big data regularly;

[0059] Upload the monitoring script to the big data platform;

[0060] The monitoring service receives the calculation results, saves the calculation results and makes them available for users to view.

[0061] Optionally, uploading the monitoring script to the big data platform includes:

[0062] Get derived configuration information;

[0063] Pull the original monitoring data according to the table and field information saved in the derived results in the configuration information;

[0064] Use the original monitoring data to calculate monitoring indicators;

[0065] The monitoring indicator calculation results are saved in Hive and synchronized to the monitoring service through Kafka.

[0066] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0067] 1. The present invention supports a variety of derivation methods: there is a python derivation for python developers, a sparkSql derivation method for ETL personnel, a wizard-style derivation method for data analysts, and a general template derivation method for business personnel.

[0068] 2. The present invention makes full use of the big data platform to support the derivation tasks of massive data: the SQL derivation, general template derivation, monitoring logic operation, and market data release in the platform all use the big data platform for data processing, which greatly improves the ability to calculate large amounts of data.

[0069] 3. The present invention realizes low-code development, and most of the processes are completed by configuration. The wizard-style derivation and general template derivation in the platform are codeless derivation methods. Monitoring services and feature marts are codeless services. Except for the running scripts, other derivation links of Python derivation and SQL derivation are completed by configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a feature derivation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] See also Figure 1 , this embodiment provides a feature derivation system, the system comprising:

[0073] The Python derivative module is used to build a Python derivative logic management system, support automatic data processing and storage, and realize feature engineering automation and model input optimization through derivative operators and market management. It ensures data quality, accelerates model iteration, optimizes business decision-making processes, and improves overall operational efficiency.

[0074] The python derivative modules include:

[0075] Write Python derivative scripts and perform derivative configuration.

[0076] Configure the timer execution policy according to the user's own execution policy.

[0077] Trigger the timer to run the python derived logic by configuring the system.

[0078] After the Python derivative script is run, the derivative result file is automatically uploaded to HDFS and the storage address is recorded.

[0079] After uploading HDFS, save the HDFS data to Hive.

[0080] The entire Python derived script is mainly divided into three parts: an execution entry method needs to be provided, and this method needs to support the passing of some configuration parameters such as the file address of the final derived result; the specific derivation logic; and saving the derived results according to the address information received by the entry method.

[0081] The management system performs derivative configuration, which is divided into two steps:

[0082] Step 1: Reference the Python derivative instance and the full path information of the entry method in the instance. After the configuration is completed, save the information. The system will automatically trigger a pre-derivation. This derivation provides data structure information for the next step to obtain the table structure information of the final derivation result. The information includes: derived field code and field type;

[0083] In step 2, based on the field information derived in step 1, the user configures the table name where the derived results are saved, which can further maintain more detailed information about the fields derived in step 1 and the dimension information of the features derived from the script.

[0084] The SQL derivative module supports users to directly write and execute SQL scripts on the configuration page, automatically generate feature data, and integrate it into a comprehensive feature management system to achieve automation and efficiency in data processing. It not only optimizes the data preparation process, but also significantly improves the efficiency and quality of model building and iteration.

[0085] The sql derivative module includes:

[0086] Perform derived configuration.

[0087] Configure the timer execution policy according to your own execution policy.

[0088] Trigger the timer and run the sql derivative logic by configuring the system.

[0089] The configuration system submits the spark-submit command through the spark-client client to execute the derived logic.

[0090] Use Spark in the script to save the derived results directly into the Hive table.

[0091] The management system performs derivative configuration, which is divided into two steps:

[0092] Step 1: Configure the prepared derivative SQL script to the system. After the configuration is completed, save the information. The system will automatically trigger a pre-derivation. This derivation provides data structure information for the next step to obtain the table structure information of the final derivative result. The information includes: derived field code and field type. The execution method is to submit the spark-submit command through spark-client, and execute the SQL derivative result table structure information according to the pre-made Scala script in the system.

[0093] Step 2: Based on the field information derived in step 1, the user configures the table name where the derived results are saved, and can further maintain more detailed information of the fields derived in step 1, as well as the dimension information of the features derived by the script;

[0094] The general template derivation module and the general derivation operator can flexibly define data sources, time ranges, processing logic, and function templates through templated configuration, automatically execute feature derivation processes, generate and manage a variety of derived features, greatly optimize the data preparation cycle, enhance the efficiency, flexibility, and scalability of data applications, and provide strong support for machine learning projects.

[0095] The general template derivative module includes:

[0096] The configuration system configures information according to the page instructions.

[0097] After the configuration is completed and saved, the Cartesian product is automatically arranged and combined according to the query conditions based on the configured information, and the generated data is parsed into executable SQL statements, which are persisted for subsequent scheduled execution and derivation.

[0098] Then go to the page to complete the field information automatically parsed by the background, the table information and dimension information saved in the derived results.

[0099] Configure the timer execution policy according to your own execution policy.

[0100] Trigger the timer and derive logic from the general template by configuring the system call.

[0101] Submit the spark-submit command through the spark-client client.

[0102] The derived results are saved synchronously to the hive table configured by the user.

[0103] The configuration information according to the page guidance includes:

[0104] Select the raw data table information to be used for the derivation.

[0105] Select the Time Window field and configure the time window.

[0106] Configure the conditions for querying data. Multiple conditions can be supported, and currently a maximum of 5 conditions are supported.

[0107] Configure indicators (such as calculating the user's payment amount / loan amount, etc.) and calculation templates (such as the average / maximum / minimum value of the loan amount, etc.).

[0108] The wizard-style derivation module is used to automatically process data table reading and feature engineering (including data exploration, preprocessing, transformation, derivation, and screening) on ​​the Mars platform, manage offline feature marts, and support scheduled tasks and interface services, thereby optimizing the data preparation process and accelerating model building and iteration.

[0109] The wizard-style derivation module includes:

[0110] First, configure the derivation logic in the wizard-based derivation service system.

[0111] The configuration system can configure and save the wizard-style derived instance that references the completed execution and the execution result is a successful execution.

[0112] After saving, the background asynchronous execution obtains the table structure information of the derived results in the derived instance.

[0113] Improve the parsed field information, table information and dimension information saved in the derived results.

[0114] Configure the timer execution policy according to your own execution policy.

[0115] Trigger the timer, call the wizard-style derivation logic through the configuration system, and call back the configuration system derivation results through Kafka after the wizard-style service derivation is completed.

[0116] After the configuration system receives the derivative results, it triggers the derivative result saving logic and stores the results in the hive table configured by the user.

[0117] The configuration derivation logic includes:

[0118] Select the raw data table information to be used for the derivation.

[0119] Configure the data exploration information and click Execute.

[0120] Select feature engineering operators and hyperparameters, and start execution.

[0121] After execution, view the derived result data information.

[0122] The feature monitoring module plays a core role in machine learning and data analysis. It not only protects model performance from data drift, quickly identifies and solves data quality issues, but also improves model accuracy by optimizing data quality. At the same time, it supports continuous iteration of models and business strategy adjustments, provides accurate insights for business decisions, and is the key driving force for the success of data-driven strategies.

[0123] The feature monitoring module includes:

[0124] The configuration system periodically synchronizes the configuration's derivative information to big data.

[0125] Upload the monitoring script to the big data platform.

[0126] The monitoring service receives the calculation results, saves the calculation results and makes them available for users to view.

[0127] The uploading of the monitoring script to the big data platform includes:

[0128] Get derived configuration information.

[0129] Pull the original monitored data based on the table and field information saved in the derived results in the configuration information.

[0130] Use the original monitoring data to calculate monitoring indicators.

[0131] The monitoring indicator calculation results are saved in Hive and synchronized to the monitoring service through Kafka.

[0132] Feature Market Module, the feature market module includes:

[0133] Configure the basic information of the market, including name, code, feature market table, etc.

[0134] Features can be selected across multiple derived features as long as the dimensional information is the same.

[0135] After the configuration is completed and saved, the features in the market information are launched as online services.

[0136] Configure the timer execution policy according to your own execution policy.

[0137] The timer triggers the collection task regularly, and the collection task automatically generates the collection SQL script according to the selected features.

[0138] Submit the spark-submit command through spark-client to the big data platform for execution.

[0139] The big data platform calls back the execution result after the execution is completed.

[0140] This embodiment supports a variety of derivation methods: Python derivation for Python developers, SparkSql derivation for ETL personnel, wizard-based derivation for data analysts, and general template derivation for business personnel.

[0141] This embodiment makes full use of the big data platform to support the derivation tasks of massive data: SQL derivation, general template derivation, monitoring logic operation, and market data release in the platform all use the big data platform for data processing, which greatly improves the ability to calculate large amounts of data.

[0142] This embodiment realizes low-code development. Most of the processes are completed by configuration. The wizard-style derivation and general template derivation in the platform are codeless derivation methods. Monitoring services and feature marts are codeless services. Except for the running scripts, all other derivation links of python derivation and sql derivation are completed by configuration.

[0143] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A feature derivation system, characterized in that: The system comprises: Python derivative module, used to build Python derivative logic management system, supporting automatic data processing and storage; SQL derivative module, which is used to support users to directly write and execute SQL scripts on the configuration page and automatically generate feature data; The general template derivation module is used to define data sources, time ranges, processing logic and function templates, and automatically execute the feature derivation process; Wizard-style derivation module for automating data table reading and feature engineering, and managing offline feature marts; The feature monitoring module is used to protect model performance from data drift, identify and resolve data quality issues, and improve model accuracy by optimizing data quality.

2. A feature derivation system as claimed in claim 1, characterized in that: The system further includes a feature market module, the feature market module including: Configure the basic information of the market and select features; After the configuration is completed and saved, the features in the market information are launched as online services; Configure the timer execution strategy according to the execution strategy; The timer triggers the collection task regularly, and the collection task automatically generates the collection SQL script according to the selected features; Submit the spark-submit command through spark-client to the big data platform for execution; The big data platform calls back the execution result after the execution is completed.

3. A feature derivation system as claimed in claim 1, characterized in that: The python derivative modules include: Write Python derivative scripts and perform derivative configuration; Configure the timer execution strategy according to the execution strategy; Trigger the timer and run the python derivative logic; After the Python derivative script is run, the derivative result file is automatically uploaded to HDFS and the storage address is recorded; After uploading HDFS, save the HDFS data to Hive.

4. A feature derivation system as claimed in claim 1, characterized in that: The sql derivative module includes: Perform derivative configuration; Configure the timer execution strategy; Trigger the timer and run the SQL derivative logic; Submit the spark-submit command through the spark-client client to execute the derived logic; Use Spark in the script to save the derived results directly into the Hive table.

5. A feature derivation system as claimed in claim 1, characterized in that: The general template derivative module includes: Configure information according to the page instructions; After the configuration is completed and saved, the Cartesian product is automatically arranged and combined according to the query conditions based on the configured information, and the generated data is parsed into executable SQL statements, and the SQL statements are persisted; Improve the parsed field information, table information and dimension information saved in the derived results; Configure the timer execution strategy according to the execution strategy; Trigger the timer and call the general template derived logic; Submit the spark-submit command through the spark-client client; The derived results are saved synchronously to the hive table configured by the user.

6. A feature derivation system as claimed in claim 5, characterized in that: The configuration information according to the page guidance includes: Select the original data table information used for the derivation; Select the Time Window field and configure the time window; Configure the conditions for querying data; Configure indicators and calculation templates.

7. A feature derivation system as claimed in claim 1, characterized in that: The wizard-style derivation module includes: Configure derivative logic; The configuration reference has been executed and the execution result is a successfully executed wizard-style derived instance, which is saved; Asynchronously execute to obtain the table structure information of the derived results in the derived instance; Improve the parsed field information, table information and dimension information saved in the derived results; Configure the timer execution strategy according to the user's own execution strategy; Trigger the timer and call the wizard-style derivation logic. After the wizard-style service derivation is completed, the system derivation result is configured through the Kafka callback. After the configuration system receives the derivative results, it triggers the derivative result saving logic and stores the results in the hive table configured by the user.

8. A feature derivation system as claimed in claim 7, characterized in that: The configuration derivation logic includes: Select the original data table information used for the derivation; Configure data exploration information and click Execute; Select feature engineering operators and hyperparameters, and start execution; After execution, view the derived result data information.

9. A feature derivation system as claimed in claim 1, characterized in that: The feature monitoring module includes: Synchronize the configuration derivative information to big data regularly; Upload the monitoring script to the big data platform; The monitoring service receives the calculation results, saves the calculation results and makes them available for users to view.

10. A feature derivation system as claimed in claim 9, characterized in that: The uploading of the monitoring script to the big data platform includes: Get derived configuration information; Pull the original monitoring data according to the table and field information saved in the derived results in the configuration information; Use the original monitoring data to calculate monitoring indicators; The monitoring indicator calculation results are saved in Hive and synchronized to the monitoring service through Kafka.

Images