Feature data generation method and apparatus, storage medium, and electronic device

Abstract

The application discloses a feature data generation method and device, a storage medium and electronic equipment. The method comprises the following steps: reading a feature group and feature group logic of the feature group in a database to obtain feature logic of each feature in the feature group; taking each feature in the feature group as a current feature, generating an execution statement of the current feature according to the feature logic of the current feature, and adding a task ID of an execution statement of a feature having a dependency relationship with the current feature into the execution statement of the current feature; and executing the execution statement of each feature in the feature group to obtain feature data of the feature group. The application solves the technical problem of low efficiency of generating feature data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular, to a feature data generation method and device, a storage medium, and an electronic device. BACKGROUND

[0002] The training of a model requires the support of feature data. As the use of models becomes more and more widespread, the use of feature data is also becoming more and more extensive and more and more frequent.

[0003] In the prior art, when generating feature data, a worker usually collates fields in a data source, writes an SQL statement that conforms to feature logic, and then executes the SQL statement to calculate and generate feature data. However, this method is inefficient in generating feature data. SUMMARY

[0004] The embodiments of the present application provide a feature data generation method and device, a storage medium, and an electronic device to at least solve the technical problem of low efficiency in generating feature data.

[0005] According to one aspect of the embodiments of the present application, a feature data generation method is provided, including: reading a feature group in a database and feature group logic of the feature group, to obtain feature logic of each feature in the feature group; taking each feature in the feature group as a current feature, generating an execution statement of the current feature according to the feature logic of the current feature, and adding a task ID of an execution statement of a feature that has a dependency relationship with the current feature to the execution statement of the current feature; and executing the execution statement of each feature in the feature group to obtain feature data of the feature group.

[0006] According to another aspect of the embodiments of the present application, a feature data generation device is provided, including: a first reading module configured to read a feature group in a database and feature group logic of the feature group, to obtain feature logic of each feature in the feature group; a generating module configured to take each feature in the feature group as a current feature, generate an execution statement of the current feature according to the feature logic of the current feature, and add a task ID of an execution statement of a feature that has a dependency relationship with the current feature to the execution statement of the current feature; and a first execution module configured to execute the execution statement of each feature in the feature group to obtain feature data of the feature group.

[0007] As an optional example, the generating module is further configured to: read an operator, a data source, an output column, and a grouping basis of the current feature in the feature logic; and splice the execution statement of the current feature according to the operator, the data source, the output column, and the grouping basis.

[0008] As an optional example, the generating module is further configured to: obtain a first task ID of an execution statement of an upstream dependent feature of the current feature and a second task ID of an execution statement of a downstream dependent feature of the current feature; and write the first task ID and the second task ID into the execution statement of the current feature.

[0009] As an optional example, the executing module includes a processing unit configured to execute each execution statement as a current statement, and perform the following operations on the current statement: in a case where the current statement has an upstream statement with a dependency relationship, if the upstream statement is successfully executed, the current statement is executed; and in a case where the upstream statement fails to be executed, it is determined that the current statement fails to be executed, and a failure reason is recorded.

[0010] As an optional example, the executing module further includes an executing unit configured to, in a case where the current statement has a downstream statement with a dependency relationship, execute the downstream statement of the current statement in a case where the current statement is successfully executed.

[0011] As an optional example, the device further includes an updating module configured to update a survival heartbeat timestamp of an execution statement of the feature group when any execution statement is executed; a second reading module configured to read the survival heartbeat timestamp after the execution statement of the feature group is executed; and a second executing module configured to re-execute the execution statement of the feature group in a case where a time length of the survival heartbeat timestamp from a current time is greater than a first threshold value.

[0012] As an optional example, the device further includes an inputting module configured to input a data source, an operator, a feature library, and a feature group into a database before reading the feature group and feature group logic of the feature group in the database, wherein the feature group includes a plurality of features, the data source is source data used by each feature in the feature group, the operator is a calculation logic of each feature in the feature group, and the feature library is used to define a range of users that can share the feature group.

[0013] According to still another aspect of the embodiments of the present application, a storage medium is provided, in which a computer program is stored, and the computer program is run by a processor to execute the feature data generation method.

[0014] According to still another aspect of the embodiments of the present application, an electronic device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the feature data generation method by using the computer program.

[0015] In the embodiment of the present application, the feature group in the database and the feature group logic of the feature group are read, the feature logic of each feature in the feature group is obtained, each feature in the feature group is taken as a current feature, the execution statement of the current feature is generated according to the feature logic of the current feature, the task ID of the execution statement of the feature having a dependency relationship with the current feature is added to the execution statement of the current feature, and the execution statement of each feature in the feature group is executed to obtain the feature data of the feature group. In the method, the feature group in the database and the feature group logic of the feature group are read in the process of generating the feature data of the feature, the execution statement of the feature group is generated according to the read logic, and the execution statement is executed to obtain the feature data of the feature group, so that the staff no longer needs to write the corresponding SQL according to the feature, the feature data generation efficiency is improved, and the technical problem of low feature data generation efficiency is solved. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0017] Figure 1 is a flow chart of an optional feature data generation method according to an embodiment of the present application;

[0018] Figure 2 is a system framework diagram of an optional feature data generation method according to an embodiment of the present application;

[0019] Figure 3 is a system framework diagram of another optional feature data generation method according to an embodiment of the present application;

[0020] Figure 4 is a structural schematic diagram of an optional feature data generation device according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the present application.

[0023] It is to be understood that the terminology "first", "second", and the like used in the specification and the claims of the application and the above drawings is merely intended to distinguish between similar objects and not necessarily to describe a particular sequential or chronological order. It is to be understood that data so used can be interchanged, where appropriate, so that the embodiments of the application described herein can be carried out in other than the order shown or described herein. Furthermore, the terms "comprise" and "include" and variations thereof as used in the specification and the claims of the application are intended to cover both the express stated features or steps and also those that are equivalent in function, result, or effect. In addition, the term "comprising" and "including" and variations thereof as used in the specification and the claims of the application are intended to cover the inclusions "consisting of" and "consisting essentially of" where appropriate.

[0024] According to a first aspect of the embodiments of the present application, there is provided a feature data generation method, which can be used in the process of generating feature data used in model training, as shown in the following. Figure 1 The method comprises the following steps:

[0025] S102, reading the feature group and the feature group logic of the feature group in the database to obtain the feature logic of each feature in the feature group;

[0026] S104, taking each feature in the feature group as a current feature, generating an execution statement of the current feature according to the feature logic of the current feature, and adding the task ID of the execution statement of the feature having a dependency relationship with the current feature to the execution statement of the current feature;

[0027] S106, executing the execution statement of each feature in the feature group to obtain the feature data of the feature group.

[0028] Optionally, the feature data generation method can be applied in the process of generating feature data used in model training. The database is a data source, and the source data, i.e., the features, are stored in the database. The type of the database is not limited. The features in the database can be calculated to obtain the feature data of the features, which can be used for model training.

[0029] Figure 2 is a system block diagram of the embodiment. The system mainly comprises four parts. The first part is a feature registration and configuration platform, the second part is an operation and evaluation center, the third part is a task splitting service and scheduling platform, and the fourth part is a task execution engine. The feature registration and configuration platform is used for managing the registration and configuration of the features and materials of the database. The operation and evaluation center is used for manual operation, offline start and stop, task query, and evaluation report of the features and feature group tasks. The task splitting service and scheduling platform is used for analyzing and optimizing the multiple features in the feature group and distributing them to the executors for execution. The task execution engine is used for executing the specific calculation task, evaluating the calculation, recording the log, and feeding back the state.

[0030] Among them, the feature registration and configuration platform comprises:

[0031] 1) Operator configuration module: An operator is a unit of encapsulating a feature calculation process, such as summation, average, or a complex calculation process encapsulated in a function. After the operator is encapsulated, for example, operator 1 is to calculate the average of the feature, then in the calculation process of the feature, operator 1 is called, and the average of the feature is calculated to obtain the feature data of the feature. The operator configuration module is used to register the operator logic required by the feature, that is, which features are calculated by which operators. It is realized by uploading a user defined function (UDF) or defining a structured query language (SQL) statement, which will be integrated into a task SQL in the task splitting service later.

[0032] 2) Data source configuration module: The data source is a data table provided by the upstream. It supports data sources of multiple databases such as Hive, Impala, Mysql, etc.

[0033] 3) Feature library configuration module: The feature library is a unit of sharing range of features and feature groups, that is, all users in the feature library can share features and feature groups; features and feature groups cannot be shared between different feature libraries. The feature library is set for project management of different teams. For example, user 1 and user 2 can share features and feature groups in feature library 1, and user 3 and user 4 can share features and feature groups in feature library 2. If user 1 wants to use feature data to train a model, the feature data can be generated by using the features in feature library 1.

[0034] 4) Feature group configuration module: The feature group is the outlet of the feature data output. A feature group contains multiple features, and multiple features generated by an offline task are combined into an entity table in the feature group, which is prepared for the downstream use of the feature table. In this embodiment, a part of the features can be configured into a feature group. Then, the features configured into a feature group are output to an entity table after calculation.

[0035] 5) Feature configuration module: The main module for configuring feature calculation logic, including upstream dependent data (data source or other current feature library under the feature), operator and operator parameter instantiation, output column configuration, other parameters (such as where condition, groupby, etc. Common SQL fields)

[0036] The operation and evaluation center comprises:

[0037] 1) Job running query platform: responsible for displaying the current running task state, log and other information on the user interface;

[0038] 2) Offline task start-stop service: responsible for providing task offline running configuration and start-stop in the user interface, managed by feature group;

[0039] 3) Feature evaluation report: whenever a feature task is completed, the data evaluation of the current feature will be automatically run, and the evaluation results will be stored in the database. Users can view the evaluation report center front-end visualization report. The viewable content includes the number, mean, variance, extreme value, and histogram distribution information;

[0040] 4) Feature group evaluation report: whenever a feature group task is completed, the data evaluation of the current feature group will be automatically run, and the evaluation results will be stored in the database. Users can view the evaluation report center front-end visualization report. The viewable content is the number of current feature groups, and the correlation coefficient of all features.

[0041] Task splitting service and scheduling platform includes a scheduler for analyzing and optimizing multiple features in a feature group and distributing them to an executor for execution.

[0042] Task execution engine includes an executor for specific calculation task execution, evaluation calculation, log recording, and state feedback.

[0043] In this embodiment, the features in the database can be configured with operators, i.e., the calculation method of the features, the configuration of the feature group, the feature library, and the dependency relationship of the features.

[0044] When a feature data training model is to be obtained, the feature group to be used can be determined, the feature logic of the feature group is identified, and the logic of the features in the feature group is obtained. The logic of the features records the dependency relationship between the features. The execution statement of the features is generated according to the read feature logic, and the execution statement of all features in the feature group is obtained. Since there is a dependency relationship between the features, the obtained execution statement also has a dependency relationship. The execution statement of the feature group is executed according to the dependency relationship, and the feature data of the feature group is obtained.

[0045] In the above method, in the process of generating the feature data of the features, the feature group and the feature group logic of the feature group in the database can be read, the execution statement of the feature group is generated according to the read logic, and the execution statement is executed to obtain the feature data of the feature group, so that the staff no longer needs to write the corresponding SQL according to the features, and the effect of improving the feature data generation efficiency is achieved.

[0046] Optionally, in the embodiment, for each feature in the feature group, the feature is taken as a current feature. When generating the execution statement of the current feature, it can be checked which features have a dependency relationship with the current feature. The dependency relationship can be an upstream dependency relationship and / or a downstream dependency relationship. The upstream dependency relationship means that the execution statement of the current feature is executed after the execution statement of the upstream feature is executed. The downstream dependency relationship means that the execution statement of the downstream feature is executed after the execution statement of the current feature is executed. After the execution statement of the current feature is generated, the task IDs of the execution statements of the features having the dependency relationship with the current feature are added to the execution statement of the current feature.

[0047] As an optional example, generating the execution statement of the current feature according to the feature logic of the current feature comprises:

[0048] reading the operator, the data source, the output column and the grouping basis of the current feature in the feature logic;

[0049] splicing the execution statement of the current feature according to the operator, the data source, the output column and the grouping basis.

[0050] Optionally, in the embodiment, when the execution statement of the current feature is generated, the operator, the data source, the output column and the grouping basis of the current feature can be obtained, and then the execution statement of the current feature is spliced according to the operator, the data source, the output column and the grouping basis of the current feature.

[0051] As an optional example, adding the task IDs of the execution statements of the features having the dependency relationship with the current feature to the execution statement of the current feature comprises:

[0052] obtaining a first task ID of the execution statement of the upstream dependent feature of the current feature and a second task ID of the execution statement of the downstream dependent feature of the current feature;

[0053] writing the first task ID and the second task ID into the execution statement of the current feature.

[0054] Optionally, in the embodiment, the execution statement of the upstream dependent feature of the current feature can be a first execution statement, the task ID of the first execution statement is the first task ID, the execution statement of the downstream dependent feature of the current feature can be a second execution statement, and the task ID of the second execution statement is the second task ID. The number of the upstream dependent features or the downstream dependent features of the current feature can be one or more. If the number of the dependent features is more than one, the task IDs also correspond to more than one.

[0055] As an optional example, the execution statements of the features in the feature group have a dependency relationship, and executing the execution statement of each feature in the feature group to obtain the feature data of the feature group comprises:

[0056] execute the current statement in the case that the current statement has a downstream statement with a dependency relationship, and the current statement is executed successfully.

[0057] execute the current statement in the case that the current statement has a downstream statement with a dependency relationship, and the current statement is executed successfully.

[0058] In the case that the upstream statement is executed unsuccessfully, it is determined that the current statement is executed unsuccessfully, and the failure reason is recorded.

[0059] As an optional example, the above method further comprises:

[0060] execute the downstream statement of the current statement in the case that the current statement has a downstream statement with a dependency relationship, and the current statement is executed successfully.

[0061] Optionally, for the execution statement of the feature with a dependency relationship, there is a corresponding execution order. The execution statement of the upstream dependent feature of the current feature is executed first, then the execution statement of the current feature is executed, and then the execution statement of the downstream dependent feature of the current feature is executed. If there are multiple upstream dependent features of the current feature, the execution statement of the current feature needs to be executed after the execution of the execution statements of the multiple upstream dependent features is completed. If there are multiple downstream dependent features of the current feature, the execution of each downstream dependent feature of the current feature needs to be performed after the execution of the current feature is completed.

[0062] In the execution process, the output result of the execution statement of the upstream dependent feature can be used as the input of the execution statement of the current feature. The output result of the execution statement of the current feature can be used as the input of the execution statement of the downstream dependent feature.

[0063] If the execution statement of the upstream dependent feature of the current feature is executed unsuccessfully, the current feature dependent on the upstream dependent feature is also recorded as being executed unsuccessfully, and the failure reason is that the execution statement of the upstream dependent feature is executed unsuccessfully.

[0064] As an optional example, the above method further comprises:

[0065] updating the survival heartbeat timestamp of the execution statement of the feature group when any one of the execution statements is executed;

[0066] reading the survival heartbeat timestamp after the execution of the execution statement of the feature group is completed;

[0067] re-executing the execution statement of the feature group in the case that the time length of the survival heartbeat timestamp from the current time is greater than a first threshold.

[0068] Optionally, in the embodiment, a survival heartbeat timestamp can also be set. The time recorded by the survival heartbeat timestamp can be updated. The survival heartbeat timestamp can be updated once after each execution statement of a feature in the feature group is executed. If all execution statements of the feature group are executed, the recorded survival heartbeat timestamp can be read. If the time length between the survival heartbeat timestamp and the current time is greater than the first threshold, it indicates that the survival heartbeat timestamp has not been updated for a long time, and further indicates that the execution statements of the features in the feature group that are relatively late are not executed. Therefore, the execution statements of the feature group can be re-executed to ensure the generation of feature data.

[0069] As an optional example, before reading the feature group and the feature group logic in the database, the above method further includes:

[0070] entering the data source, the operator, the feature library, and the feature group into the database, wherein the feature group contains multiple features, the data source is the source data used by each feature in the feature group, the operator is the calculation logic of each feature in the feature group, and the feature library is used to define the range of users that can share the feature group.

[0071] Optionally, in the embodiment, the data source, the operator, the feature library, and the feature group can be entered into the database, so as to configure the feature group, the feature library, the operator for feature calculation in the feature group, and the data source used by the feature.

[0072] Figure 3 is another system framework diagram of the embodiment.

[0073] The feature registration and configuration platform enters the resource data such as the data source, the operator, the feature library, the feature, and the feature group into the database. The operation and maintenance center starts the task and is divided into manual and offline timing tasks. The difference is that the manual task reads the feature group and the feature logic from the feature group and the feature table in the database, and the offline timing task reads the feature group and the feature logic from the task offline snapshot, and then splits the read content into executable SQL statements for execution. When the platform starts the task, the SQL logic of multiple features under the feature group name is obtained by reading the feature group logic; the feature group and the feature logic are assembled into multiple tree structures (there is an upstream and downstream dependency relationship) of executable SQL statements, and are distributed to the task executor for execution of specific SQL. In the configuration of the feature logic, the dependent feature relationship of the current feature is set, that is, the data read by the current feature comes from the output result of the upstream dependent feature. Therefore, the features have an upstream and downstream dependency relationship. The feature group is the end point of all feature dependency relationships.

[0074] The executable SQL task split by the task splitting service includes the following structure:

[0075] current task id;

[0076] Build table SQL instance;

[0077] Current task execution SQL instance;

[0078] Result evaluation SQL instance;

[0079] Current task status;

[0080] Upstream dependent task ID list;

[0081] Downstream dependent task ID list;

[0082] Survival heartbeat timestamp;

[0083] The task splitting service reads all logical elements (operators, data sources, output columns, groupby, etc.) from the feature, feature group relationship of the database, assembles them into SQL, saves the current task to the database by the scheduling platform, and obtains the task id; according to the dependent data source and feature relationship, the upstream dependent task ID is queried; when saving the upstream dependent task of the current task, the current task ID is also added as the downstream dependency of the upstream task. In this way, the task relationship table of the upstream and downstream tree structure is completed.

[0084] The task executor receives the SQL task and starts to execute the SQL, and after saving the result in the corresponding result table, updates the state and log to the database and the storage system redis corresponding to the task id. In this step, the platform encapsulates all feature subtasks into a task structure and submits them to the MQ task queue. When the task is executed, the scheduling platform monitors the running state of each task in real time, reads the downstream task of the current task when a subtask is completed, and attempts to start the next subtask; when a subtask fails, the downstream task dependency is updated to fail immediately, and the running is abandoned.

[0085] The execution engine service, after starting, pulls the task structure by listening to the MQ task queue; the execution engine obtains the task, parses the task structure, reads the upstream dependent task ID list, and constantly polls the upstream dependent task state from the database or redis. When all upstream dependent task states are complete, the current task can enter the execution process; when there is a failure in the upstream dependent task, the current task is marked as failed and the failure reason is saved, and the current task exits.

[0086] The execution engine reads the table building statement, the execution statement and the evaluation statement instance in the task structure after entering the execution flow, sequentially calls the big data computing engine such as hive / spark to execute, and records the execution log to the database at any time. If the execution fails, the current execution flow state is identified as failure and exits; if the execution succeeds, the current execution flow state is identified as completion and exits. When all the feature and feature group tasks are completed or failed, the corresponding task state is changed, and the user is notified of the result.

[0087] The scheduling platform tracks the redis task at any time: if there is a task running, the running state of all feature subtasks under the task is traversed and the task progress is calculated. When all the subtasks are completed or failed, the task state is updated and the current redis task is emptied. At the same time, the scheduling platform obtains the heartbeat survival time stamp of the current subtask, and when the heartbeat time is more than 1 minute away from the current time (i.e. the execution engine has not updated the heartbeat for 1 minute), it is determined that the subtask is abnormal, and is resubmitted to the MQ task queue.

[0088] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0089] According to another aspect of the embodiments of the present application, a feature data generation apparatus is also provided, as shown in Figure 4 , comprising:

[0090] The first reading module 402 is configured to read the feature group and the feature group logic of the feature group in the database, and obtain the feature logic of each feature in the feature group.

[0091] The generation module 404 is configured to take each feature in the feature group as a current feature, generate an execution statement of the current feature according to the feature logic of the current feature, and add the task ID of the execution statement of the feature having a dependency relationship with the current feature into the execution statement of the current feature.

[0092] The first execution module 406 is configured to execute the execution statement of each feature in the feature group, and obtain the feature data of the feature group.

[0093] Optionally, the feature data generation device described above can be applied in the process of generating feature data used in the model training process. A database is a data source, and the database stores source data, i.e., features. The type of database is not limited. The features in the database can be calculated to obtain feature data of the features, and the feature data can be used for training of the model.

[0094] Figure 2 is a system block diagram of the embodiment. The system mainly includes four parts. The first part is a feature registration and configuration platform, the second part is an operation and evaluation center. The third part is a task splitting service and scheduling platform. The fourth part is a task execution engine. Among them, the feature registration and configuration platform is used to manage the registration and configuration of features and materials in the database. The operation and evaluation center is used for manual operation, offline start and stop, task query, and evaluation report of features and feature group tasks. The task splitting service and scheduling platform is used to analyze and optimize multiple features in a feature group and distribute them to the executor for execution. The task execution engine is used for specific calculation task execution, evaluation calculation, log recording, and state feedback.

[0095] Among them, the feature registration and configuration platform includes:

[0096] 1) Operator configuration module: An operator is a unit of encapsulation of a feature calculation process, such as summation, average value, or a complex calculation process encapsulated in a function. After the operator is encapsulated, for example, operator 1 is to calculate the average value of the feature, then in the calculation process of the feature, operator 1 is called, and the average value of the feature is calculated to obtain the feature data of the feature. The operator configuration module is used to register the operator logic required by the feature, i.e., which features to use which operators for calculation. It is realized by uploading a user defined function (UDF) or defining a structured query language (SQL), which will be integrated into a task SQL in the task splitting service later.

[0097] 2) Data source configuration module: The data source is a data table provided by the upstream. It supports data sources of multiple databases such as Hive, Impala, Mysql, etc.

[0098] 3) Feature library configuration module: The feature library is a unit of sharing range of features and feature groups, i.e., all users in the feature library can share features and feature groups; features and feature groups cannot be shared between different feature libraries. The feature library is set for project management of different teams. For example, user 1 and user 2 can share features and feature groups in feature library 1, and user 3 and user 4 can share features and feature groups in feature library 2. If user 1 wants to use feature data to train a model, the features in feature library 1 can be used to generate feature data.

[0099] 4) Feature group configuration module: Feature group is the output of feature data. The same feature group contains multiple features, and the offline task generates multiple features and combines them into an entity table in the feature group, which is prepared for downstream feature table use. In this embodiment, a part of the features can be configured into a feature group. Then, the features configured into a feature group are output to an entity table after calculation.

[0100] 5) Feature configuration module: The main module for configuring feature calculation logic, including upstream dependent data (data source or other features under the current feature library), operator and operator parameter instantiation, output column configuration, and other parameters (such as where condition, groupby, and other common SQL fields)

[0101] The operation and evaluation center includes:

[0102] 1) Job running query platform: responsible for displaying the current running task state, log and other information on the user interface;

[0103] 2) Offline task start-stop service: responsible for providing task offline running configuration and start-stop on the user interface, through feature group management;

[0104] 3) Feature evaluation report: whenever the feature task is completed, the data evaluation of the current feature will be automatically run, and the evaluation result will be stored in the database. Users can view the evaluation report center front-end visual report. The viewable content includes the number, mean, variance, extreme value, and histogram distribution information;

[0105] 4) Feature group evaluation report: whenever the feature group task is completed, the data evaluation of the current feature group will be automatically run, and the evaluation result will be stored in the database. Users can view the evaluation report center front-end visual report. The viewable content is the current feature group number and all feature two-by-two correlation coefficients.

[0106] The task splitting service and scheduling platform includes a scheduler, which is used to analyze and optimize multiple features in a feature group, and distribute them to an executor for execution.

[0107] The task execution engine includes an executor, which is used for specific task execution, evaluation calculation, log recording, and state feedback.

[0108] In this embodiment, the features in the database can be configured with operators, that is, the calculation method of the features, the feature group, the feature library, and the dependent relationship of the features.

[0109] When a feature data training model is to be obtained, a feature group to be used can be determined, feature logic of the feature group is identified, and logic of features in the feature group is obtained. The logic of the features records dependency relationships between the features. Execution statements of the features are generated according to the read feature logic, and execution statements of all features in the feature group are obtained. Since there are dependency relationships between the features, the obtained execution statements also have dependency relationships. The execution statements of the feature group are executed according to the dependency relationships, and feature data of the feature group is obtained.

[0110] In the above method, in the process of generating feature data of a feature, a feature group and feature group logic of the feature group in a database can be read, execution statements of the feature group are generated according to the read logic, and the execution statements are executed to obtain feature data of the feature group, so that staff no longer needs to write corresponding SQL according to the feature, and the effect of improving feature data generation efficiency is achieved.

[0111] Optionally, in the embodiment, for each feature in a feature group, the feature is taken as a current feature. When generating an execution statement of the current feature, it can be checked which features have dependency relationships with the current feature. The dependency relationships can be upstream dependency relationships and / or downstream dependency relationships. The upstream dependency relationship means that the execution statement of the current feature is executed after the execution statement of an upstream feature is executed. The downstream dependency relationship means that the execution statement of a downstream feature is executed after the execution statement of the current feature is executed. After the execution statement of the current feature is generated, task IDs of the execution statements of the features having the dependency relationships with the current feature are added to the execution statement of the current feature.

[0112] As an optional example, the generating module is further configured to:

[0113] read the operator, the data source, the output column, and the grouping basis of the current feature in the feature logic, and splice the execution statement of the current feature according to the operator, the data source, the output column, and the grouping basis.

[0114] Optionally, in the embodiment, when the execution statement of the current feature is generated, the operator, the data source, the output column, and the grouping basis of the current feature can be obtained, and then the execution statement of the current feature is spliced according to the operator, the data source, the output column, and the grouping basis.

[0115] As an optional example, the generating module is further configured to:

[0116] obtain a first task ID of an execution statement of an upstream dependent feature of the current feature and a second task ID of an execution statement of a downstream dependent feature of the current feature;

[0117] write the first task ID and the second task ID into the execution statement of the current feature.

[0118] Optionally, in the embodiment, the execution statement of the upstream dependent feature of the current feature can be a first execution statement, the task ID of the first execution statement can be a first task ID, the execution statement of the downstream dependent feature of the current feature can be a second execution statement, and the task ID of the second execution statement can be a second task ID. The number of the upstream dependent features or the downstream dependent features of the current feature can be one or more. If the number of the dependent features is more than one, the task IDs also correspond to multiple task IDs.

[0119] As an optional example, the execution module includes:

[0120] The processing unit is configured to execute the following operations on each execution statement as a current statement:

[0121] If the current statement has an upstream statement with a dependent relationship, the current statement is executed if the upstream statement is successfully executed.

[0122] If the upstream statement fails to be executed, it is determined that the current statement fails to be executed, and the failure reason is recorded.

[0123] As an optional example, the execution module further includes:

[0124] The execution unit is configured to execute the downstream statement of the current statement if the current statement has a downstream statement with a dependent relationship and the current statement is successfully executed.

[0125] Optionally, in the embodiment, the execution statements of the features with dependent relationships have corresponding execution sequences. The execution statement of the upstream dependent feature of the current feature is executed first, the execution statement of the current feature is executed second, and the execution statement of the downstream dependent feature of the current feature is executed third. If there are multiple upstream dependent features of the current feature, the execution statement of the current feature is executed after the execution statements of the multiple upstream dependent features are all executed. If there are multiple downstream dependent features of the current feature, each downstream dependent feature of the current feature needs to be executed after the current feature is executed.

[0126] During the execution, the output result of the execution statement of the upstream dependent feature can be used as the input of the execution statement of the current feature. The output result of the execution statement of the current feature can be used as the input of the execution statement of the downstream dependent feature.

[0127] If the execution statement of the upstream dependent feature of the current feature fails to be executed, the current feature dependent on the upstream dependent feature also records the execution failure, and the reason for the execution failure is that the execution statement of the upstream dependent feature fails to be executed.

[0128] As an optional example, the apparatus further comprises:

[0129] an updating module configured to update the alive heartbeat timestamp of the execution statement of the feature group when executing any one of the execution statements;

[0130] a second reading module configured to read the alive heartbeat timestamp after the execution of the execution statement of the feature group is completed;

[0131] a second execution module configured to re-execute the execution statement of the feature group if the time length of the alive heartbeat timestamp from the current time is greater than the first threshold.

[0132] Optionally, in the embodiment, an alive heartbeat timestamp can be further set. The time recorded by the alive heartbeat timestamp can be updated. The alive heartbeat timestamp can be updated once after the execution of the execution statement of one feature of the feature group is completed. If all the execution statements of the feature group are executed, the recorded alive heartbeat timestamp can be read. If the time length of the alive heartbeat timestamp from the current time is greater than the first threshold, it indicates that the alive heartbeat timestamp has not been updated for a long time, and further indicates that the execution statements of the features in the feature group that are relatively late are not executed. Therefore, the execution statements of the feature group can be re-executed to ensure the generation of the feature data.

[0133] As an optional example, the apparatus further comprises:

[0134] an inputting module configured to input the data source, the operator, the feature library and the feature group into the database before reading the feature group and the feature group logic of the feature group in the database, wherein the feature group comprises a plurality of features, the data source is the source data used by each feature in the feature group, the operator is the calculation logic of each feature in the feature group, and the feature library is used to define the range of users that can share the feature group.

[0135] Optionally, in the embodiment, the data source, the operator, the feature library and the feature group can be inputted into the database, so as to configure the feature group, the feature library, the operator used in the calculation of the features in the feature group, and the data source used by the features.

[0136] Other examples of the embodiment can refer to the above examples, which will not be described herein.

[0137] Figure 5 is a structural block diagram of an optional electronic device according to the embodiment of the application, as shown in Figure 5 includes a processor 502, a communication interface 504, a memory 506 and a communication bus 508, wherein the processor 502, the communication interface 504 and the memory 506 complete the communication among each other through the communication bus 508, wherein,

[0138] a memory 506 for storing a computer program;

[0139] a processor 502 for implementing the following steps when executing the computer program stored in the memory 506:

[0140] reading the feature group and the feature group logic of the feature group in the database to obtain the feature logic of each feature in the feature group;

[0141] taking each feature in the feature group as a current feature, generating an execution statement of the current feature according to the feature logic of the current feature, and adding the task ID of the execution statement of the feature having a dependency relationship with the current feature into the execution statement of the current feature;

[0142] executing the execution statement of each feature in the feature group to obtain the feature data of the feature group.

[0143] Optionally, in the embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, Figure 5 In the figure, only one thick line is used to represent the communication bus, but it does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the electronic device and other devices.

[0144] The memory can include a RAM and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0145] As an example, the memory 506 can include, but is not limited to, the first reading module 402, the generating module 404, and the first execution module 406 in the feature data generating device. In addition, other module units in the request processing device can also be included, but not limited to, which will not be described in detail in this example.

[0146] The processor can be a general processor, which can include but is not limited to: a CPU (Central Processing Unit), a NP (Network Processor), etc.; or can be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0147] Optionally, the specific examples in the embodiments can refer to the examples described in the above embodiments, and the embodiments will not be described here.

[0148] Those skilled in the art can understand that, Figure 5 The structure shown is only schematic, and the device for implementing the above feature data generation method can be a terminal device, which can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device can further include more or less components (such as a network interface, a display device, etc.) than Figure 5 or have a different configuration from Figure 5 The structure shown.

[0149] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware of the terminal device by a program, which can be stored in a computer readable storage medium, and the storage medium can include: a flash disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0150] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which stores a computer program, wherein the computer program is run by a processor to perform the steps in the above feature data generation method.

[0151] Optionally, in the embodiment, all or part of the steps in the various methods of the above embodiments can be understood by those skilled in the art that the terminal device related hardware can be instructed by a program, and the program can be stored in a computer readable storage medium, and the storage medium can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0152] The serial numbers of the above embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0153] The integrated units in the above embodiments, if realized in the form of software function units and sold or used as independent products, can be stored in the above computer readable storage medium. Based on such understanding, the technical solutions of the application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, and the computer software product stored in the storage medium includes a plurality of instructions for causing one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the application.

[0154] In the above embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0155] In several embodiments provided in the present application, it should be understood that the disclosed client can be implemented by other manners. Among them, the apparatus embodiment described above is only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0156] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0157] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0158] The above description is only the preferred embodiment of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A feature data generation method characterized by comprising: The method comprises the following steps: reading a feature group in a database and feature group logic of the feature group, to obtain feature logic of each feature in the feature group; taking each feature in the feature group as a current feature, generating an execution statement of the current feature according to the feature logic of the current feature, and adding task IDs of execution statements of features having a dependency relationship with the current feature to the execution statement of the current feature; executing the execution statement of each feature in the feature group to obtain feature data of the feature group; wherein the step of adding the task IDs of the execution statements of the features having the dependency relationship with the current feature to the execution statement of the current feature comprises the following steps: obtaining a first task ID of an execution statement of an upstream dependent feature of the current feature and a second task ID of an execution statement of a downstream dependent feature of the current feature; and writing the first task ID and the second task ID into the execution statement of the current feature.

2. The method of claim 1, wherein, The step of generating the execution statement of the current feature according to the feature logic of the current feature comprises the following steps: reading an operator, a data source, an output column and a grouping basis of the current feature in the feature logic; splicing the execution statement of the current feature according to the operator, the data source, the output column and the grouping basis.

3. The method of claim 1, wherein, The execution statements of the feature group have a dependency relationship, and the step of executing the execution statement of each feature in the feature group to obtain the feature data of the feature group comprises the following steps: taking each execution statement as a current statement, and performing the following operations on the current statement: in the case that the current statement has an upstream statement having a dependency relationship, executing the current statement in the case that the upstream statement is successfully executed; in the case that the upstream statement fails to be executed, determining that the current statement fails to be executed, and recording a failure reason.

4. The method of claim 3, wherein, The method further comprises the following steps: in the case that the current statement has a downstream statement having a dependency relationship, executing the downstream statement of the current statement in the case that the current statement is successfully executed.

5. The method of claim 1, wherein, The method further comprises the following steps: updating a survival heartbeat timestamp of the execution statement of the feature group when executing any execution statement; reading the survival heartbeat timestamp after the execution statement of the feature group is executed; in the case that a time length of the survival heartbeat timestamp from a current time is greater than a first threshold value, re-executing the execution statement of the feature group.

6. The method according to any one of claims 1 to 5, characterized in that, Before the step of reading the feature group in the database and the feature group logic of the feature group, the method further comprises the following steps: recording a data source, an operator, a feature library and a feature group into the database, wherein the feature group contains a plurality of features, the data source is source data used by each feature in the feature group, the operator is calculation logic of each feature in the feature group, and the feature library is used to define a range of users that can share the feature group.

7. A feature data generating apparatus characterized by comprising: The method comprises the following steps: a first reading module is configured to read a feature group in a database and feature group logic of the feature group, to obtain feature logic of each feature in the feature group; The generating module is configured to take each feature in the feature group as a current feature, generate an execution statement of the current feature according to feature logic of the current feature, and add task IDs of execution statements of features having a dependency relationship with the current feature into the execution statement of the current feature. The first executing module is configured to execute the execution statement of each feature in the feature group to obtain feature data of the feature group. The adding of the task IDs of the execution statements of the features having the dependency relationship with the current feature into the execution statement of the current feature includes: obtaining a first task ID of an execution statement of an upstream dependent feature of the current feature and a second task ID of an execution statement of a downstream dependent feature of the current feature; and writing the first task ID and the second task ID into the execution statement of the current feature.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The computer program is configured to execute the method in any one of claims 1 to 6 when run by the processor.

9. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store the computer program, and the processor is configured to execute the method in any one of claims 1 to 6 by using the computer program. The memory is configured to store the computer program, and the processor is configured to execute the method in any one of claims 1 to 6 by using the computer program.

Images