System and method for managing data modeling

Abstract

A method for managing data modeling is provided, comprising: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; (C) configuring modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following: a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task; and (D) launching the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project. Through the above-described method, the processing, data, and resources involved in data modeling can be effectively managed.

Description

Technical Field

[0001] The present invention generally relates to data modeling technology, and more particularly to a system and method for managing data modeling. Background Art

[0002] In recent years, with the generation of massive amounts of data in various fields, data mining technology has gradually become more widely used. It is used to analyze the potential meaning of data and reveal the inherent laws of business, thereby helping people better carry out practical activities such as production and operation. However, the application of data mining technology not only requires relevant personnel to have professional knowledge in machine learning or statistical learning, but also requires the use of large amounts of data samples in various formats and contents. Therefore, in practice, problems such as data management, personnel coordination, and modeling level often make it difficult to effectively carry out data modeling to solve business problems.

[0003] Several systems and devices exist for data modeling in the existing technology. These systems and devices can help users complete the data modeling process and perform corresponding data analysis. However, these existing systems and devices can only train models based on imported features, without integrating data modeling project processes, and are unable to achieve effective and systematic data modeling processing. Summary of the Invention

[0004] Exemplary embodiments of the present invention are directed to overcoming the drawback of existing data modeling systems that lack a systematic modeling process.

[0005] According to one aspect of an exemplary embodiment of the present invention, a method for managing data modeling is provided, comprising: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; (C) configuring modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task; and (D) starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project.

[0006] In the method, step (A) may further include: specifying at least one user who participates in data modeling under the established modeling project, wherein the at least one user may be set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

[0007] In the method, the at least one user may include a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

[0008] In the method, the modeling project participating users may be configured to be able to share the system resources and data resources of the modeling project main user under the modeling project.

[0009] In the method, in step (B), the at least one modeling plan can be established by copying an already established modeling plan; or, in step (C), the modeling tasks involved in the corresponding data modeling activity can be configured by copying an already established modeling task.

[0010] In the method, in step (C), a DAG graph corresponding to the established modeling plan may be displayed, wherein the DAG graph may include interactive structural units for respectively configuring the modeling tasks.

[0011] In the method, the interactive structure unit may include at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

[0012] In the method, the modeling task configuration entry and the modeling task progress indicator can be displayed in the same area of the interactive structure unit in a multiplexed manner.

[0013] In the method, the modeling project established in step (A) can be a rapid modeling project; and, in step (B), a rapid modeling plan can be automatically established under the rapid modeling project, and in step (C), after the input data records are configured according to the user's input operation under the rapid modeling plan, the corresponding feature extraction tasks and model training tasks can be automatically configured, and in step (D), the rapid modeling plan can be automatically started.

[0014] In the method, in step (C), the feature extraction task and the model training task can be automatically configured using preset feature extraction configuration items and model training parameters, wherein the feature extraction configuration items can be used to define how to extract predetermined features from data records.

[0015] In the method, in step (C), when configuring the feature extraction task, feature extraction configuration items can be generated based on the input operations performed by the user on the page for setting feature extraction configuration items, wherein the feature extraction configuration items can be used to define how predetermined features are extracted from data records.

[0016] In the method, the page for setting feature extraction configuration items can be a graphical user interface, which may include a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

[0017] In the method, the feature extraction configuration item of each predetermined feature may include a source field item and a processing method item, the source field item can be used to limit the field of the data record involved in each predetermined feature as the source field, and the processing method item can be used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function can be used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item to run the feature extraction task when the modeling plan is started.

[0018] In the method, step (D) may further include: downloading the saved results generated by the at least one modeling plan according to a predetermined percentage or a predetermined number of rows.

[0019] In the method, in step (D), after starting the model training task of the at least one modeling plan, the model coefficients generated during the execution of the model training task can be distributed and stored in multiple parameter servers.

[0020] The method may further include: (E) displaying an evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started in correspondence with the corresponding model training task and / or modeling plan.

[0021] In the method, in step (C), the model application task can be configured as a manual application mode and / or an automatic application mode, wherein, in the manual application mode, the model application can be started according to the user's operation, and in the automatic application mode, the model application can be started according to a preset time interval.

[0022] According to another aspect of an exemplary embodiment of the present invention, a system for managing data modeling is provided, comprising: a project establishment module for establishing a modeling project for managing data modeling; a plan establishment module for establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; a task configuration module for configuring modeling tasks involved in corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task; and a plan startup module for starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project.

[0023] In the system, the project establishment module may further specify at least one user who participates in data modeling under the established modeling project, wherein the at least one user may be set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

[0024] In the system, the at least one user may include a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

[0025] In the system, modeling project participating users can be configured to share system resources and data resources of the modeling project main user under the modeling project.

[0026] In the system, the plan establishment module may establish the at least one modeling plan by copying an already established modeling plan; or, the task configuration module may configure the modeling tasks involved in the corresponding data modeling activity by copying an already established modeling task.

[0027] In the system, the task configuration module may display a DAG diagram corresponding to the established modeling plan, wherein the DAG diagram may include interactive structural units for respectively configuring the modeling tasks.

[0028] In the system, the interactive structure unit may include at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

[0029] In the system, the modeling task configuration entry and the modeling task progress indicator can be displayed in the same area of the interactive structure unit in a multiplexed manner.

[0030] In the system, the modeling project established by the project establishment module can be a rapid modeling project; and the plan establishment module can automatically establish a rapid modeling plan under the rapid modeling project. The task configuration module can automatically configure the corresponding feature extraction tasks and model training tasks after configuring the input data records according to the user's input operations under the rapid modeling plan, and the plan startup module can automatically start the rapid modeling plan.

[0031] In the system, the task configuration module can automatically configure feature extraction tasks and model training tasks using preset feature extraction configuration items and model training parameters, wherein the feature extraction configuration items can be used to define how to extract predetermined features from data records.

[0032] In the system, the task configuration module can generate feature extraction configuration items according to the input operations performed by the user on the page for setting feature extraction configuration items when configuring the feature extraction task, wherein the feature extraction configuration items can be used to define how to extract predetermined features from data records.

[0033] In the system, the page for setting feature extraction configuration items can be a graphical user interface, which may include a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

[0034] In the system, the feature extraction configuration item of each predetermined feature may include a source field item and a processing method item, the source field item can be used to limit the field of the data record involved in each predetermined feature as a source field, and the processing method item can be used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function can be used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item to run the feature extraction task when the modeling plan is started.

[0035] In the system, the plan starting module may further download the results generated by the at least one saved modeling plan according to a predetermined percentage or a predetermined number of rows.

[0036] In the system, after the plan initiation module initiates the model training task of the at least one modeling plan, the model coefficients generated during the execution of the model training task can be stored in a distributed manner in multiple parameter servers.

[0037] The system may further include: a presentation module for displaying an evaluation report of a data model generated when a model evaluation task under the at least one modeling plan is started, corresponding to the corresponding model training task and / or modeling plan.

[0038] In the system, the task configuration module can configure the model application task as a manual application mode and / or an automatic application mode, wherein, in the manual application mode, the model application can be started according to the user's operation, and in the automatic application mode, the model application can be started according to a preset time interval.

[0039] According to another aspect of an exemplary embodiment of the present invention, a computing device for managing data modeling is provided, comprising a storage component and a processor, wherein a set of computer-executable instructions is stored in the storage component, and when the set of computer-executable instructions is executed by the processor, the following steps are performed: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; (C) configuring modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks; and (D) starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project.

[0040] In the computing device, step (A) may further include: specifying at least one user who participates in data modeling under the established modeling project, wherein the at least one user may be set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

[0041] In the computing device, the at least one user may include a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

[0042] In the computing device, modeling project participating users may be configured to share system resources and data resources of the modeling project main user under the modeling project.

[0043] In the computing device, in step (B), the at least one modeling plan can be established by copying an already established modeling plan; or, in step (C), the modeling tasks involved in the corresponding data modeling activity can be configured by copying an already established modeling task.

[0044] In the computing device, in step (C), a DAG graph corresponding to the established modeling plan may be displayed, wherein the DAG graph may include interactive structural units for respectively configuring the modeling tasks.

[0045] In the computing device, the interactive structure unit may include at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

[0046] In the computing device, the modeling task configuration entry and the modeling task progress indicator may be displayed in a multiplexed manner in the same area of the interactive structure unit.

[0047] In the computing device, the modeling project established in step (A) can be a rapid modeling project; and, in step (B), a rapid modeling plan can be automatically established under the rapid modeling project, and in step (C), after the input data records are configured according to the user's input operation under the rapid modeling plan, the corresponding feature extraction tasks and model training tasks can be automatically configured, and in step (D), the rapid modeling plan can be automatically started.

[0048] In the computing device, in step (C), the feature extraction task and the model training task can be automatically configured using preset feature extraction configuration items and model training parameters, wherein the feature extraction configuration items can be used to define how to extract predetermined features from the data records.

[0049] In the computing device, in step (C), when configuring the feature extraction task, feature extraction configuration items can be generated based on the input operations performed by the user on the page for setting feature extraction configuration items, wherein the feature extraction configuration items can be used to define how predetermined features are extracted from data records.

[0050] In the computing device, the page for setting feature extraction configuration items can be a graphical user interface, which may include a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

[0051] In the computing device, the feature extraction configuration item of each predetermined feature may include a source field item and a processing method item, the source field item can be used to limit the field of the data record involved in each predetermined feature as a source field, and the processing method item can be used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function can be used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item to run the feature extraction task when the modeling plan is started.

[0052] In the computing device, step (D) may further include: downloading the saved results generated by the at least one modeling plan according to a predetermined percentage or a predetermined number of rows.

[0053] In the computing device, in step (D), after starting the model training task of the at least one modeling plan, the model coefficients generated during the execution of the model training task can be distributedly stored in multiple parameter servers.

[0054] In the computing device, when the computer executable instruction set is executed by the processor, the following steps may also be performed: (E) the evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started is displayed corresponding to the corresponding model training task and / or modeling plan.

[0055] In the computing device, in step (C), the model application task can be configured as a manual application mode and / or an automatic application mode, wherein, in the manual application mode, the model application can be started according to the user's operation, and in the automatic application mode, the model application can be started according to a preset time interval.

[0056] In the system and method for managing data modeling according to an exemplary embodiment of the present invention, it is not only possible to help users complete the data modeling process, but also possible to effectively perform systematic data processing, process processing and / or model processing, thereby truly helping users find solutions to practical problems based on big data technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] These and / or other aspects and advantages of the present invention will become more apparent and more readily understood from the following detailed description of embodiments of the present invention in conjunction with the accompanying drawings, in which:

[0058] Figure 1 A block diagram illustrating a data modeling management system according to an exemplary embodiment of the present invention;

[0059] Figure 2 A flowchart showing a data modeling management method according to an exemplary embodiment of the present invention;

[0060] Figure 3 shows an example of a configuration page for a modeling plan according to an exemplary embodiment of the present invention;

[0061] Figure 4 An example of an operation item list of an interactive structure unit according to an exemplary embodiment of the present invention is shown;

[0062] Figure 5A shows an example of a graphical user interface for configuring a feature extraction task according to an exemplary embodiment of the present invention;

[0063] Figure 5B The exemplary embodiment of the present invention is shown in FIG. Figure 5A An example of a portion of a graphical user interface that displays a list of processing methods to a user while a single field in a left area of is selected by the user;

[0064] Figure 5C The exemplary embodiment of the present invention is shown in FIG. Figure 5AAn example of a portion of a graphical user interface that displays a list of processing methods to a user while a plurality of fields in a left area of the display are selected by the user;

[0065] Figure 6 An example of an exemplary graphical user interface having an area capable of text editing of feature extraction configuration items according to an exemplary embodiment of the present invention is shown;

[0066] Figure 7 An example of a page for downloading a result file according to an exemplary embodiment of the present invention is shown;

[0067] Figure 8 An example of a page for creating a new modeling project according to an exemplary embodiment of the present invention is shown;

[0068] Figure 9 An example of a page for rapid modeling according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the present invention, exemplary embodiments of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] Exemplary embodiments of the present invention provide a system for managing data modeling. This system can be implemented entirely in software via a computer program, with specialized hardware, or through a combination of hardware and software. This system not only assists users in completing the data modeling process but also effectively and systematically processes data, processes, and / or models, thereby truly helping users find solutions to practical problems based on big data technology.

[0071] Figure 1 A block diagram of a data modeling management system according to an exemplary embodiment of the present invention is shown. Specifically, the data modeling management system proposes a processing architecture based on "modeling project-modeling plan-modeling task", wherein the modeling project is aimed at data modeling management, and the modeling plan is a modeling activity that can be started under the modeling project, and the modeling activity involves at least one modeling task (for example, a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task), so that each time a modeling activity is started, one or more complete data modeling processes and / or partial data modeling processes are completed, and the intermediate result data and / or final result data generated by such a data modeling process can be saved under the modeling project.

[0072] like Figure 1As shown, the project establishment module 10 is used to establish a modeling project for managing data modeling. For example, a corresponding modeling project can be established for a predetermined modeling goal, modeling team, modeling data source, etc. Here, the modeling project can be established according to user instructions, allowing the user to manage data, processes, participating users, and / or models under the modeling project.

[0073] The plan establishment module 20 is used to establish at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities. Here, the modeling plan refers to the data modeling activities that can be started under the modeling project, and the data modeling activities involve at least one modeling task (for example, data input task, data splicing task, feature extraction task, model training task, model evaluation task, model application task, etc.), so that each time a modeling activity is started, one or more complete data modeling processes and / or partial data modeling processes are executed, thereby completing the experimental work of at least one modeling link. The process and / or results of these experimental works will be saved under the modeling project.

[0074] The task configuration module 30 is used to configure the modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks may include at least one of the following items: data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks.

[0075] Specifically, the data input task is used to input the original data resources for model training; the data splicing task is used to splice specific fields of the same or different input tables of the original data resources when necessary to obtain data records from which features can be extracted; the feature extraction task is used to extract features and target values for model training from the data records; the model training task is used to train the model based on the extracted features and corresponding target values; the model evaluation task is used to use test data to evaluate the model effect; the model application task is used to apply new data samples to the trained model to obtain prediction results.

[0076] It should be noted that according to exemplary embodiments of the present invention, configurable modeling tasks may include one or more of the above-mentioned modeling tasks, and it is not limited to that all modeling tasks must be in a configurable state.

[0077] Here, the task configuration module 30 can configure one or more modeling tasks under each modeling plan. These configured modeling tasks can constitute one or more complete data modeling processes and / or partial data modeling processes, so that when each modeling plan is started, the corresponding configured modeling tasks under the modeling plan can be executed.

[0078] The plan activation module 40 is used to activate the at least one modeling plan and save the results generated by the at least one modeling plan under the modeling project. Here, the plan activation module 40 can activate the at least one established modeling plan one by one and / or in batches. When the modeling plan is activated, the modeling tasks configured thereunder are executed in a predetermined order and generate corresponding execution results. Accordingly, the plan activation module 40 can save the execution results corresponding to each modeling task under the modeling plan, so that the intermediate results and / or final results generated by each related modeling plan can be saved under the modeling project.

[0079] Existing data modeling systems only allow for the configuration of each step in a single data modeling process, based on data input and output. However, data modeling technology requires significant expertise, and the data and operations involved are complex. Therefore, users (e.g., business personnel) struggle to directly achieve good modeling results when operating existing modeling systems, and are unable to effectively adjust or improve the modeling process, making it difficult to conveniently utilize data modeling technology to solve practical problems.

[0080] According to an exemplary embodiment of the present invention, by executing a modeling plan configured with one or more modeling tasks and saving the execution results of each modeling task under the modeling plan, multiple complete modeling experiments or phased modeling experiments of different links can be carried out under the same modeling project, and the various experimental results or experimental configurations can be used to effectively adjust or improve the data modeling project.

[0081] The following reference Figure 2 Here, as an example, the data modeling management method according to the exemplary embodiment of the present invention is described. Figure 2 The method shown can be Figure 1 It should be noted that the data management system shown in the figure can also be executed by a computing device with a specific configuration. Figure 2 The method shown.

[0082] As shown in the figure, in step S10, the project establishment module 10 establishes a modeling project for managing data modeling. As mentioned above, under the established modeling project, an activateable modeling plan can be further established, wherein the modeling plan involves one or more modeling tasks. Accordingly, the results generated after the modeling plan is activated are saved under the modeling project to which it belongs.

[0083] Here, as an example, the project establishment module 10 may detect a user clicking on the "New Project" tab on the project management page and create a new modeling project based on the user's click. Optionally, the project establishment module 10 may configure the established modeling project based on the user's operation, such as configuring project participating users and available project data.

[0084] Here, as a preferred method, at least one user who participates in data modeling can be specified under a newly created modeling project, wherein the at least one user is set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task. As described above, according to an exemplary embodiment of the present invention, a modeling plan that can be independently started is established under each modeling project, and one or more modeling tasks can be configured under each modeling plan. Therefore, in this way, not only can multi-user collaborative modeling be achieved, but also, when collaboratively modeling, each user can operate relatively independently under the same modeling project, thereby further ensuring the independence and reference of each user during collaboration.

[0085] For example, at least one user participating in a modeling project may include a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

[0086] As described above, the project establishment module 10 can establish a corresponding modeling project according to the user's instructions. In this case, as an example, the user who instructs to establish the modeling project can be designated as the main user of the modeling project, and at least a portion of the data resources owned by the main user of the modeling project can be allocated to the modeling project. In addition, at least a portion of the system resources (for example, computing resources, storage resources, etc.) of the main user of the modeling project can also be allocated to the modeling project. In other words, the main user of the modeling project bears all the expenses of the modeling project. Accordingly, the participating users of the modeling project can be set to be able to share the system resources and data resources of the main user of the modeling project under the modeling project. Here, the sharing permissions of the participating users of the modeling project can be specified by the main user of the modeling project or set by the system default. As an example, only the main user of the modeling project is configured to have the right to delete or modify the established modeling project and its configuration items. For example, the main user of the modeling project can delete or modify the entire modeling project, delete, modify or add the original data resources (for example, input tables) that can be used by the modeling project, etc. In addition, users participating in the modeling project may be allowed to process the results of the modeling project (for example, intermediate results (such as sample tables) or final results (such as trained models)), but are prohibited from performing any processing on the modeling project itself or its configuration items.

[0087] As can be seen, according to exemplary embodiments of the present invention, a modeling project master user can implement resource allocation and personnel deployment for data modeling through a modeling project. For example, the project establishment module 10 can modify the configuration of a modeling project (including data resources, system resources, or participating personnel), delete an established modeling project, and so on, based on the instructions of the modeling project master user.

[0088] In step S20, the plan creation module 20 creates at least one modeling plan under the established modeling project, wherein the modeling plan is used to execute data modeling activities. As described above, the modeling plan is an activatable object, and the data modeling activities executed upon activation can be considered as a modeling experiment, which can correspond to a complete data modeling process or a portion of the data modeling process.

[0089] Here, as an example, a list of established modeling plans can be displayed on the page of the established modeling project. In addition, a button such as "New Plan" is provided. When the user clicks the "New Plan" button, the plan establishment module 20 can create a new blank modeling plan and add it to the list.

[0090] As another example, the at least one modeling plan can be created by copying an already created modeling plan. For example, a list of already created modeling plans can be displayed on the page for creating the modeling project. Next to each modeling plan listed in the list, a button such as "Copy Plan" can be provided. When the user clicks the "Copy Plan" button, the configuration content of the corresponding modeling plan is copied.

[0091] Additionally, you can copy the current modeling plan from the configuration page. Figure 3 An example of a configuration page for a modeling plan according to an exemplary embodiment of the present invention is shown. Figure 3 An operation item (eg, icon, button, etc.) for copying a modeling plan is set on the page shown, and the configuration content of the current modeling plan is copied according to the operation performed by the user on the operation item.

[0092] Here, as an example, the configuration content may include relevant configuration items of all modeling tasks under the modeling plan. As a preferred method, the plan establishment module 20 may automatically rename the copied modeling plan name, modeling task name, output table name, model name, etc. according to preset naming rules.

[0093] As an example, the modeling plan obtained after copying can be established under the same modeling project by default. In this case, when the user clicks the operation item (for example, an icon, button, etc.) used to copy a specific modeling plan, the new modeling plan obtained after copying can be automatically displayed under the modeling project to which the modeling plan belongs.

[0094] Here, the plan establishment module 20 can establish respective modeling plans according to the instructions of each user. Here, as an example, for the established modeling plan, only the main user of the modeling project and / or the modeling project participating users who established the modeling plan can be allowed to modify, delete, and other operations on the modeling plan. In addition, all users can also be allowed to modify, delete, and other operations on the modeling plan.

[0095] In step S30, the task configuration module 30 configures the modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: data input task, data splicing task, feature extraction task, model training task, model evaluation task, and model application task.

[0096] Here, the configurable modeling tasks can be any one or any combination of data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks. Accordingly, the modeling tasks involved in the data modeling activities can be at least one configurable modeling task.

[0097] As an example, the configurable modeling tasks can be set to include only feature extraction tasks and model training tasks. In this case, the feature extraction task can be configured to directly extract the features and target values of the training samples from the data records of the input table as the original data resource. In addition, if model evaluation and model application are required, model evaluation can be performed independently after the model is trained (that is, model evaluation is performed independently of the modeling plan). Similarly, model application can also be independent of the modeling plan, so that model training and model application can be run separately on two independent platforms.

[0098] As another example, the configurable modeling tasks can be set to include the six modeling tasks mentioned above: data input task, data splicing task, feature extraction task, model training task, model evaluation task, and model application task. Here, any parameters or items related to the modeling task can be configured under each modeling task. As an example, one or more raw data resources can be configured in the data input task; the method of splicing fields for the input table of the raw data resource to obtain data records can be configured in the data splicing task; how to obtain the features and target values of the training samples (i.e., the sample table) from the data records can be configured in the feature extraction task; model training parameters such as model algorithm, model size, number of training rounds, and learning rate can be configured in the model training task; parameters such as evaluation indicators can be configured in the model evaluation task; and items such as application method and result data download can be configured in the model application task.

[0099] It should be noted that the above is only an example. In practice, any combination of data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks can be selected as configurable modeling tasks as needed, and the specific configuration content can be adaptively adjusted.

[0100] Here, as an example, the task configuration module 30 may configure each modeling task under each modeling plan based on operations performed by the user within each modeling plan's page. For example, a new modeling task may be created using a tab provided on the page for creating each modeling task, and the specific configuration of the modeling task may be completed on the configuration page corresponding to the newly created modeling task.

[0101] As a preferred manner, according to an exemplary embodiment of the present invention, the configuration of the modeling task can be achieved with good interaction by reflecting the modeling plan process. Specifically, the task configuration module 30 can display a DAG diagram corresponding to the established modeling plan, wherein the DAG diagram includes interactive structural units for respectively configuring the modeling tasks. The above-mentioned DAG diagram can be displayed in the page of the modeling plan, and the page can also be provided with buttons for creating various new modeling tasks. As an example, when the user clicks such a button, the corresponding modeling task configuration page will be directly entered. After the user completes the specific configuration of the newly created modeling task in the modeling task configuration page, the interactive structural unit corresponding to the modeling task can be displayed on the DAG diagram. As another example, when the user clicks the above-mentioned button, the interactive structural unit corresponding to the newly created modeling task can be first displayed on the DAG diagram. At this time, the specific configuration of the modeling task can be completed by performing operations on the interactive unit.

[0102] As an example, in Figure 3 The page shown may include a DAG diagram corresponding to the current modeling plan according to an exemplary embodiment of the present invention. The DAG diagram may include interactive structural units for allocating and configuring various modeling tasks.

[0103] Here, in order to enhance the interactivity of the modeling task configuration, the interactive structural unit may be designed to include at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

[0104] by Figure 3 Taking the interactive structure unit "Data Splicing Task 1" as an example, the modeling task icon, modeling task name, and modeling task configuration entry are displayed from left to right. Here, the modeling task configuration entry serves as the entrance to the modeling task configuration page.

[0105] As an example, the modeling task configuration entry can be designed as a button for directly entering the modeling task configuration page. When the user clicks such a button, the user can enter the modeling task configuration page to perform specific configuration of the modeling task or modify the existing configuration of the modeling task.

[0106] Furthermore, as another example, the modeling task configuration portal can be designed as a button for displaying a list of operation items. Here, in addition to including an operation item (e.g., "Modify") for entering the modeling task configuration page, the list can also include other operation items to effectively complete related operations under the modeling plan. For example, the list can further include an operation item for copying the current modeling task, an operation item for creating a new downstream modeling task, and an operation item for deleting the current modeling task.

[0107] Figure 4 An example of an operation item list of an interactive structural unit according to an exemplary embodiment of the present invention is shown. Specifically, when the user clicks Figure 3 When configuring the modeling task entry on the interactive structure unit "Feature Splicing Task 1" shown in the figure, you can Figure 4 As shown, a corresponding list of operation items is displayed near the interactive structure unit "Feature Stitching Task 1", which may include operation items such as modification (used to modify the configuration content of the current modeling task), copy (used to copy the current modeling task), feature extraction (used to create a new downstream feature extraction task), model training (used to create a new downstream model training task), and deletion (used to delete the current modeling task). The user can click on each operation item to perform corresponding configuration or other operations for the modeling task.

[0108] The interactive structure unit may also include a modeling task progress indicator for indicating the progress of the modeling task represented by the interactive structure unit when the modeling plan is started. Here, as a preferred embodiment, the modeling task configuration entry and the modeling task progress indicator are displayed in the same area of the interactive structure unit in a multiplexed manner.

[0109] like Figure 3As shown, after starting the modeling plan, when the modeling task represented by the interactive structural unit (for example, a model training task) is run, the modeling task configuration entry on the interactive structural unit is converted to a modeling task progress indication. As an example, the modeling task progress indication can indicate the running progress of the modeling task in the form of a percentage. After the modeling task is successfully run or fails to run, the modeling task progress indication will be converted to the modeling task configuration entry again. That is to say, when the modeling task has not yet run and the modeling task has been completed (that is, the run is successful or failed), the interactive structural unit displays the modeling task configuration entry so that the corresponding modeling task can be configured or other operations can be performed. During the running of the modeling task, the interactive structural unit displays the modeling task progress indication, which on the one hand indicates the running progress of the modeling task, and on the other hand can also prohibit operations such as configuring the modeling task. Here, as a preferred method, in order to further distinguish between modeling tasks that have not yet run, failed to run, and successfully run, the filling style of the interactive structural unit can be used to distinguish them. For example, for a modeling task that has not yet been executed, its interactive structure unit may not be filled with any content (e.g., a colored area); for a modeling task that has been successfully executed, its interactive structure unit may be filled with predetermined content (e.g., a green area); and for a modeling task that has failed, its interactive structure unit may be filled with another predetermined content (e.g., a red area). In addition, as an example, for a modeling task that is in progress, its interactive structure unit may be filled with content according to the percentage indicated by the modeling task progress.

[0110] It can be seen that the above-mentioned interactive structural unit can effectively express the attributes and running status of the modeling task, and can also effectively configure or operate the corresponding modeling task, thereby enhancing the user experience.

[0111] In addition, as an example, in step S30, the modeling tasks involved in the corresponding data modeling activity can be configured by copying the established modeling tasks. Figure 4 The page shown has an operation item for copying a modeling task (e.g., a "Copy" option in the list). The configuration content of the corresponding modeling task is copied based on the user's operation on the operation item. Here, as an example, the configuration content may include relevant configuration items for the modeling task. Preferably, the task configuration module 30 may automatically rename the copied modeling task name, output table name, model name, etc. according to a preset naming rule.

[0112] For example, a copied modeling task may be placed under the same modeling plan by default. In this case, after the user selects the action item for copying the modeling task (e.g., the "Copy" option in the list), the copied modeling task will automatically be displayed under the modeling plan to which it belongs. For example, within the overall flow of the modeling plan displayed on the DAG graph, the modeling task may be displayed at the same stage as the copied modeling task, meaning that both are successors to the same upstream modeling task.

[0113] Here, the task configuration module 30 can configure each modeling task according to the instructions of each user. Here, as an example, for the configured modeling task, only the main user of the modeling project and / or the participating users of the modeling project who configured the modeling task can be allowed to modify, delete, and other operations on the modeling task. In addition, all users can also be allowed to modify, delete, and other operations on the modeling task.

[0114] Furthermore, according to an exemplary embodiment of the present invention, feature engineering can be implemented based on the user's manual operation. Specifically, the feature extraction task can be configured according to the user's input to transform and define data records to form training features that can represent the problem to be determined.

[0115] For example, when configuring a feature extraction task, the task configuration module 30 may generate feature extraction configuration items based on input operations performed by the user on a page for setting feature extraction configuration items, wherein the feature extraction configuration items are used to define how to extract predetermined features from data records.

[0116] When the modeling task configured under the modeling plan includes a data splicing task, the above-mentioned data records may come from the output of the data splicing task; when the modeling task configured under the modeling plan only includes a data input task but does not include a data splicing task, the above-mentioned data records may come directly from the output of the data input task; when the modeling task configured under the modeling plan includes neither a data input task nor a data splicing task, the above-mentioned data records may come directly from the input table configured by the user as the original data resource in the feature extraction task.

[0117] Specifically, the feature extraction configuration item of each predetermined feature may include a source field item and a processing method item, the source field item is used to limit the field of the data record involved in each predetermined feature as a source field, and the processing method item is used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function is used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item when the modeling plan is started to run the feature extraction task.

[0118] Accordingly, the page for setting feature extraction configuration items can be a graphical user interface, which includes a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

[0119] The following describes, in conjunction with the accompanying drawings, an example of a user configuring a feature extraction task through a graphical user interface according to an embodiment of the present invention. It should be noted that the graphical user interface here is merely an example, and the present invention may also employ any other form of input interface. As an example, the feature extraction configuration items set through the interface may be used to form a corresponding configuration file so that each feature extraction configuration item can be subsequently read from the configuration file. Alternatively, the feature extraction configuration items set through the interface may be directly applied to the feature extraction main program without generating any configuration file.

[0120] Figure 5A An example of a graphical user interface 200 for configuring a feature extraction task according to an exemplary embodiment of the present invention is shown, wherein an input table 201 bank basic data may indicate the original data of a bank, a target value 202 y indicates the target value of a training sample, and an output table 203 bankdata_out indicates an extracted feature table.

[0121] In the above graphical user interface 200, at least the fields of the data record that can be used as source fields and the feature extraction configuration items of the set predetermined features can be displayed. In addition, as an example, other information about the data source or data output can also be displayed. Specifically, if Figure 5A As shown, the left area shows the various fields of the data record in the input table, including the field name 204 and the field attribute 205; the right area shows the configuration page for configuring the feature. As an example, the configuration page may include a selection input interface for displaying content options of feature extraction configuration items for manual selection, wherein each row is for a specific feature, and the source item 206, processing method 207 and feature name 208 of the feature are configured accordingly.

[0122] As an example, the various feature configuration items set by the user may be displayed in the right area according to the user's setting operations on the various fields displayed in the left area. In one example, the user may manually edit the configuration items displayed in the right area.

[0123] Specifically, the various fields of the data record can be first displayed on the graphical user interface (for example, the left area). When the user selects (for example, by clicking to select) one or some of the displayed fields, the field selected by the user is set as the set source field in the configuration page, and when the source field is selected, the processing method list is displayed on the graphical user interface. Here, as an example, the processing method list can be displayed near the source field selected by the user so that the user can select the processing method to be displayed in the configuration page from it; here, in the processing method list, all processing methods can be in an activated state; or, only processing methods that can be applied to the selected source field item can be included; or, all processing methods can be included but the processing methods that can be applied are displayed as activated and the processing methods that cannot be applied are displayed as disabled.

[0124] Figure 5B An example of a portion of a graphical user interface 300 is shown in which a single field (e.g., the "age" field) 301 in the left area is selected by the user, and a list of processing methods 302 is displayed to the user. For example, when the user clicks on the "age" field 301, a list of processing methods 302 pops up on the right side near the "age" field for selection. The list of processing methods 302 may list all processing methods, with the user's currently selected processing method highlighted. Alternatively, the list of processing methods 302 may only display processing methods applicable to the selected "age" field, or only activate processing methods applicable to the selected "age" field (e.g., display them as selectable or highlighted) in the list of processing methods 302, while other processing methods may be displayed as disabled.

[0125] Figure 5C An example of a portion of a graphical user interface 400 is shown, showing a processing method list 404 displayed to the user upon selection of multiple fields 401, 402, and 403 in the left area. This indicates that the user can select one or more source fields 401, 402, and 403 on the left side, and accordingly, a processing method list 404 pops up, allowing the user to select a processing method to be applied to these source fields. Similarly, the processing method list 404 can be popped up using any suitable method, and does not necessarily need to include all processing methods. Accordingly, the processing methods displayed in the processing method list 404 can be dynamically adjusted based on the source field selected on the left side.

[0126] In addition to the above-mentioned selection input interface that displays the content options of feature extraction configuration items for manual selection (for example, by clicking the mouse), other forms of interfaces for setting feature extraction configuration items can also be used, such as a text editing interface for manually editing configuration files, so that users can directly write "configuration files" in the text editing interface. Since the configuration file itself has repetitive content, the "configuration file" can be quickly written through text editing operations (for example, copying, pasting, dragging, etc.).

[0127] Figure 6 An exemplary graphical user interface 500 is shown with an area where text editing of feature extraction configuration items can be performed. Figure 5B and Figure 5C The graphical user interface shown is similar, except that the right area of the graphical user interface 500 shows a text editing interface 501 for manually editing the configuration file. The user can manually edit the feature extraction configuration items in the text editing interface 501, including the configuration feature item name, source field item, processing method item, etc. By performing text editing operations (e.g., copying, pasting, dragging, etc.) in the text editing interface, the user can efficiently set the feature extraction configuration items.

[0128] The above two graphical user interfaces can be displayed on the screen at the same time, or they can be displayed separately on the screen according to the user's choice. For example, in response to the user's interface switching operation input to switch between the text editing interface and the selection input type interface (display switching or activation switching), the feature extraction configuration item setting results in the interface before switching are synchronously displayed in the interface after switching. Accordingly, the user can take advantage of the operational convenience of the two configuration interfaces to more effectively set multiple feature extraction methods. For example, the user can first complete the representative feature extraction configuration by clicking and selecting the input method in the selection input type interface, and then switch to the text editing interface. Since the results of the previous settings will be synchronously displayed in the text editing interface, the user can quickly complete the extraction item settings of a large number of features in combination with operations such as copying and pasting.

[0129] In the existing field of data modeling, in order to be able to train, test or apply models based on a large amount of structured or unstructured data, it is often necessary to spend a lot of manpower in the feature engineering stage. For example, programmers are required to write the extraction code for each feature in advance according to specific feature extraction rules. Accordingly, in modeling products for customer use, such as modeling platforms, it is often necessary to input the extracted training data (i.e., extracted feature vectors) into the modeling platform, and it is difficult for users to flexibly set or adjust the objects and rules for feature extraction, which limits the use of the modeling platform. However, according to an exemplary embodiment of the present invention, the feature extraction task can be conveniently configured in the above manner, which fully expands the applicability of data modeling.

[0130] Further, according to an exemplary embodiment of the present invention, when configuring a model application task, the model application task can be configured as a manual application mode and / or an automatic application mode, wherein, in the manual application mode, the model application is started according to the user's operation, and in the automatic application mode, the model application is started according to a preset time interval.

[0131] Here, as an example, in the page for configuring model applications, a common model batch estimation application or a model batch estimation application that runs automatically at a scheduled time can be configured, and the application results can be called or downloaded through an interface.

[0132] Specifically, in manual application configuration, you can enter or modify the name of the model application, such as "2015 User Credit Risk Control Modeling Application".

[0133] In addition, the source of the model application data to be applied to the trained model can be determined based on the user's selection, for example, an available data table, an HDFS (Hadoop Distributed File System) data source, a local file, etc. After the source of the application data is determined, a list of corresponding optional data can be presented to the user for the user to select the model application data from.

[0134] In addition, it is also possible to determine which items of the model application data (i.e., original fields or related features) are included in the model application results displayed to the user based on the user's operation. For example, a pop-up box for item selection may be provided to the user, which includes two items: "Keep all item results" and "Customize item results". When the user selects "Customize item results", all items of the model application data (including the target value predicted by the model) may be displayed to the user for the user to check the final displayed items, wherein the predicted target value may be set as the output item by default and cannot be modified, and the remaining items may be checked or unchecked. In addition, an "Invert" button may be set to invert the selection result.

[0135] Furthermore, the output order of the model application results can also be determined based on the user's operation. Here, as an example, three selection buttons for output order can be provided to the user, such as "original order", "ascending order by predicted value", "descending order by predicted value", etc.

[0136] In addition to the above options, you can also configure the "Scheduled Application Task Run Period," "Scheduled Start Time," and "Scheduled End Method" based on user input when configuring a scheduled application. The scheduled end time can be set to "Continuously Run," "End After a Predetermined Number of Model Predictions," or a specific end time.

[0137] The configuration of scheduled applications can effectively expand the application scenarios of prediction models, which is particularly suitable for online applications of prediction models.

[0138] Refer again Figure 2 In step S40, the plan starting module 40 starts at least one established modeling plan, and saves the results generated by the at least one modeling plan under the modeling project. Here, when the plan starting module 40 starts a modeling plan among the at least one modeling plan, the modeling tasks configured under the modeling plan are executed in sequence, and corresponding intermediate result data and / or final result data are obtained, for example, the complete input table obtained when the data splicing task is executed, the training sample table obtained when the feature extraction task is executed, the prediction model obtained when the model training task is executed, the evaluation report obtained when the model evaluation task is executed, the prediction result obtained when the model application task is executed, etc. These result data can all be saved under the modeling plan, so as to facilitate unified processing under the modeling project to which it belongs.

[0139] As described above, for example, a list of established modeling plans may be displayed on the page of the established modeling project, wherein a button for "Launch Modeling Plan" may be provided near each modeling plan. In this way, the user can select the modeling plan to be launched on the page of the modeling project.

[0140] Alternatively, a button for starting the current modeling plan may be set in the DAG diagram page corresponding to the established modeling plan, so that when the user presses the button, the plan starting module 40 starts the current modeling plan to sequentially execute the various modeling tasks configured in the DAG.

[0141] Here, in step S40, after the model training task of the at least one modeling plan is started, the model coefficients generated during the execution of the model training task can be distributed and stored in multiple parameter servers. In this way, the capability of model training can be further improved.

[0142] In addition, the results generated by the at least one modeling plan can be downloaded according to a predetermined percentage or a predetermined number of rows. For example, when the model application task is executed, a prediction result file will be generated. Figure 7 An example of a page for downloading result files according to an exemplary embodiment of the present invention is shown. In this regard, when a user clicks a button for downloading result files in a page of a modeling project or a page of a current modeling plan, a page such as the following may be displayed to the user: Figure 7 The pop-up box shown in the figure allows users to choose whether to download all result data or the first few rows of all result data. Figure 7 The displayed pages are only examples and are not limiting. For example, according to an exemplary embodiment of the present invention, you may also choose to download a predetermined percentage of the total result data.

[0143] also, Figure 2 The method may further include: displaying the evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started in correspondence with the corresponding model training task and / or modeling plan. Specifically, according to an exemplary embodiment of the present invention, the display entry of the evaluation report of the data model can be set to correspond to the model training task and / or modeling plan to which the data model belongs. In this way, the user can conveniently adjust the model training task or other related modeling tasks under the modeling plan after viewing the evaluation report of the model.

[0144] Combination of the above Figure 2 This article describes an example of data modeling management according to an exemplary embodiment of the present invention. It can be seen that the exemplary embodiment of the present invention not only helps users complete the data modeling process, but also effectively performs systematic data processing, process processing, and / or model processing, thereby truly helping users find solutions to practical problems based on big data technology.

[0145] Preferably, under the modeling system according to the exemplary embodiment of the present invention, the rapid modeling process can be effectively configured so that users who are not familiar with the modeling process can quickly obtain a desired data model.

[0146] Specifically, the modeling project established in step S10 is a quick modeling project. Here, the quick modeling project can be established according to the user's selection of the "Quick Modeling Project" tab.

[0147] Figure 8 A page for creating a new modeling project according to an exemplary embodiment of the present invention is shown. Figure 8 On the page shown, when the user clicks the "Quick Modeling" button or the "Quick Modeling" tab, a quick modeling project will be created.

[0148] Accordingly, after the rapid modeling project is established, in step S20, a rapid modeling plan is automatically established under the rapid modeling project. In step S30, after the input data records are configured according to the user's input operations under the rapid modeling plan, the corresponding feature extraction tasks and model training tasks are automatically configured, and in step S40, the rapid modeling plan is automatically started.

[0149] For example, in step S30, the user can be provided with an entry for directly selecting an input table, allowing the user to select the original training data and target values for rapid modeling. After the user configures the input data records, the feature extraction and model training tasks can be automatically configured using pre-set feature extraction configuration items and model training parameters. The feature extraction configuration items define how to extract predetermined features from the data records.

[0150] Here, the feature extraction configuration item can be set in advance to use the default processing method (for example, direct extraction) to process all table items (i.e., fields) of the input table to obtain the various features of the sample. In addition, the model training task can be configured using pre-set model training parameters, or the model training parameters can be adaptively and automatically set by analyzing the characteristics of the input data records.

[0151] As a preferred method, users can also choose to manually set model training parameters during the rapid modeling process. Specifically, the default method can be set to use the preset model training parameters to configure the model training task, but users can also choose to set the model training parameters themselves and manually set the desired model training parameters.

[0152] Figure 9 An example of a page for rapid modeling according to an exemplary embodiment of the present invention is shown. Specifically, Figure 9 In the quick modeling page shown, users can manually set the model training parameters by selecting "More Settings". Otherwise, they can train the model according to the predetermined feature extraction configuration items and model training parameters for the input table and target value.

[0153] It should be noted that the above-mentioned data modeling management system can completely rely on the operation of computer programs to realize corresponding functions, that is, each module corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding data modeling management function.

[0154] on the other hand, Figure 1The modules shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the corresponding operations may be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operations by reading and running the corresponding program code or code segments.

[0155] Here, an exemplary embodiment of the present invention can also be implemented as a computing device, which includes a storage component and a processor, wherein a computer executable instruction set is stored in the storage component, and when the computer executable instruction set is executed by the processor, the above-mentioned data modeling management method is executed.

[0156] Specifically, the computing device may be deployed in a server or client, or may be deployed on a node device in a distributed network environment. In addition, the computing device may be a PC, tablet device, personal digital assistant, smart phone, web application, or other device capable of executing the above-mentioned instruction set.

[0157] Here, the computing device is not necessarily a single computing device, but may be any collection of devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0158] In the computing device, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0159] Some operations described in the above data modeling management method can be implemented through software, some operations can be implemented through hardware, and in addition, these operations can also be implemented through a combination of software and hardware.

[0160] The processor may execute instructions or codes stored in one of the memory components, wherein the memory component may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0161] The storage component can be integrated with the processor, for example, by integrating RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the storage component can include a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The storage component and the processor can be operatively coupled or can communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor can access files stored in the storage component.

[0162] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0163] The operations involved in the above-mentioned data modeling management method can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operate according to non-precise boundaries.

[0164] Specifically, as described above, a computing device for managing data modeling according to an exemplary embodiment of the present invention may include a storage component and a processor, wherein a set of computer-executable instructions is stored in the storage component, and when the set of computer-executable instructions is executed by the processor, the following steps are performed: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; (C) configuring modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks; (D) starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project.

[0165] It should be noted that the above has been combined Figure 2 The processing details of the data modeling management method according to the exemplary embodiment of the present invention have been described, and the processing details when the computing device executes each step will not be repeated here.

[0166] While various exemplary embodiments of the present invention have been described above, it should be understood that the foregoing description is merely illustrative and not exhaustive, and that the present invention is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for managing data modeling, comprising: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein the modeling plan is used to perform data modeling activities; (C) configuring, under each established modeling plan, modeling tasks involved in the corresponding data modeling activities, wherein the modeling tasks include at least one of the following: a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task; (D) starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project; In step (C), after the user completes the specific configuration of the newly created modeling task on the modeling task configuration page, the interactive structural unit corresponding to the modeling task is displayed, or the interactive structural unit corresponding to the newly created modeling task is displayed, and the specific configuration of the modeling task is completed by performing operations on the interactive structural unit; When configuring a feature extraction task, a preset feature extraction configuration item is used or a feature extraction configuration item is generated according to an input operation performed by a user on a page for setting the feature extraction configuration item, wherein the feature extraction configuration item is used to define how to extract predetermined features from data records; Among them, the feature extraction configuration item of each predetermined feature includes a source field item and a processing method item, the source field item is used to limit the field of the data record involved in each predetermined feature as the source field, and the processing method item is used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function is used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item when the modeling plan is started to run the feature extraction task.

2. The method according to claim 1, wherein Step (A) also includes: specifying at least one user who participates in data modeling under the established modeling project, wherein the at least one user is set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

3. The method according to claim 2, wherein: The at least one user includes a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

4. The method according to claim 3, wherein: Modeling project participating users are configured to be able to share the system resources and data resources of the modeling project main user under the modeling project.

5. The method according to claim 1, wherein In step (B), the at least one modeling plan is established by copying an already established modeling plan; Alternatively, in step (C), the modeling tasks involved in the corresponding data modeling activity are configured by copying the established modeling tasks.

6. The method of claim 1, wherein: In step (C), a DAG graph corresponding to the established modeling plan is displayed, wherein the DAG graph includes the interactive structural units for respectively configuring the modeling tasks.

7. The method according to claim 6, wherein: The interactive structural unit includes at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

8. The method of claim 7, wherein: The modeling task configuration entry and the modeling task progress indicator are displayed in the same area of the interactive structure unit in a reused manner.

9. The method of claim 1, wherein: The modeling project established in step (A) is a rapid modeling project; and, in step (B), a rapid modeling plan is automatically established under the rapid modeling project. In step (C), after the input data records are configured according to the user's input operations under the rapid modeling plan, the corresponding feature extraction tasks and model training tasks are automatically configured. In step (D), the rapid modeling plan is automatically started.

10. The method of claim 9, wherein: In step (C), the feature extraction task and the model training task are automatically configured using the preset feature extraction configuration items and model training parameters.

11. The method of claim 1, wherein: The page for setting feature extraction configuration items is a graphical user interface, which includes a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

12. The method of claim 1, wherein: Step (D) further includes: downloading the saved results generated by the at least one modeling plan according to a predetermined percentage or a predetermined number of rows.

13. The method of claim 1 , further comprising: (E) Displaying the evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started in correspondence with the corresponding model training task and / or modeling plan.

14. The method of claim 1, wherein: In step (C), the model application task is configured as a manual application mode and / or an automatic application mode, wherein in the manual application mode, the model application is started according to the user's operation, and in the automatic application mode, the model application is started according to a preset time interval.

15. A system for managing data modeling, comprising: A project establishment module, used to establish a modeling project for managing data modeling; A plan establishment module is used to establish at least one modeling plan under an established modeling project, wherein the modeling plan is used to execute data modeling activities; a task configuration module is used to configure modeling tasks involved in the corresponding data modeling activities under each established modeling plan, wherein the modeling tasks include at least one of the following items: data input tasks, data splicing tasks, feature extraction tasks, model training tasks, model evaluation tasks, and model application tasks; a plan startup module is used to start the at least one modeling plan and save the results generated by the at least one modeling plan under the modeling project; After the user completes the specific configuration of the newly created modeling task in the modeling task configuration page, the task configuration module is used to display the interactive structural unit corresponding to the modeling task, or display the interactive structural unit corresponding to the newly created modeling task, and complete the specific configuration of the modeling task by performing operations on the interactive structural unit; When configuring a feature extraction task, the task configuration module generates feature extraction configuration items using preset feature extraction configuration items or according to input operations performed by a user on a page for setting feature extraction configuration items, wherein the feature extraction configuration items are used to define how to extract predetermined features from data records; Among them, the feature extraction configuration item of each predetermined feature includes a source field item and a processing method item, the source field item is used to limit the field of the data record involved in each predetermined feature as the source field, and the processing method item is used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function is used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item when the modeling plan is started to run the feature extraction task.

16. The system of claim 15, wherein: The project establishment module further specifies at least one user who participates in data modeling under the established modeling project, wherein the at least one user is set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

17. The system of claim 16, wherein: The at least one user includes a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

18. The system of claim 17, wherein: Modeling project participating users are configured to be able to share the system resources and data resources of the modeling project main user under the modeling project.

19. The system of claim 15, wherein: The plan establishment module establishes the at least one modeling plan by copying an already established modeling plan; Alternatively, the task configuration module configures the modeling tasks involved in the corresponding data modeling activity by copying the established modeling tasks.

20. The system of claim 15, wherein: The task configuration module displays a DAG diagram corresponding to the established modeling plan, wherein the DAG diagram includes the interactive structural units for respectively configuring the modeling tasks.

21. The system of claim 20, wherein: The interactive structural unit includes at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

22. The system of claim 21, wherein: The modeling task configuration entry and the modeling task progress indicator are displayed in the same area of the interactive structure unit in a reused manner.

23. The system of claim 15, wherein: The modeling project established by the project establishment module is a rapid modeling project; and the plan establishment module automatically establishes a rapid modeling plan under the rapid modeling project. After the task configuration module configures the input data records according to the user's input operations under the rapid modeling plan, it automatically configures the corresponding feature extraction tasks and model training tasks, and the plan startup module automatically starts the rapid modeling plan.

24. The system of claim 23, wherein: The task configuration module uses preset feature extraction configuration items and model training parameters to automatically configure feature extraction tasks and model training tasks.

25. The system of claim 15, wherein: The page for setting feature extraction configuration items is a graphical user interface, which includes a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

26. The system of claim 15, wherein: The plan starting module further downloads the saved results generated by the at least one modeling plan according to a predetermined percentage or a predetermined number of rows.

27. The system of claim 15, further comprising: A presentation module is used to display the evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started in correspondence with the corresponding model training task and / or modeling plan.

28. The system of claim 15, wherein: The task configuration module configures the model application task as a manual application mode and / or an automatic application mode, wherein in the manual application mode, the model application is started according to the user's operation, and in the automatic application mode, the model application is started according to a preset time interval.

29. A computing device for managing data modeling, comprising a storage component and a processor, wherein the storage component stores a set of computer-executable instructions, and when the computer-executable instructions are executed by the processor, the following steps are performed: (A) establishing a modeling project for managing data modeling; (B) establishing at least one modeling plan under the established modeling project, wherein: Modeling plans are used to perform data modeling activities; (C) configuring, under each established modeling plan, modeling tasks involved in the corresponding data modeling activities, wherein the modeling tasks include at least one of the following: a data input task, a data splicing task, a feature extraction task, a model training task, a model evaluation task, and a model application task; (D) starting the at least one modeling plan and saving the results generated by the at least one modeling plan under the modeling project; In step (C), after the user completes the specific configuration of the newly created modeling task on the modeling task configuration page, the interactive structural unit corresponding to the modeling task is displayed, or the interactive structural unit corresponding to the newly created modeling task is displayed, and the specific configuration of the modeling task is completed by performing operations on the interactive structural unit; Generate a feature extraction configuration item based on a preset feature extraction configuration item or an input operation performed by a user on a page for setting the feature extraction configuration item when configuring a feature extraction task, wherein the feature extraction configuration item is used to define how to extract predetermined features from data records; Among them, the feature extraction configuration item of each predetermined feature includes a source field item and a processing method item, the source field item is used to limit the field of the data record involved in each predetermined feature as the source field, and the processing method item is used to specify a reference to a data processing function pre-programmed as executable code, wherein the data processing function is used to perform data processing for extracting each predetermined feature for the field value of the source field defined by the source field item when the modeling plan is started to run the feature extraction task.

30. The computing device of claim 29, wherein: Step (A) also includes: specifying at least one user who participates in data modeling under the established modeling project, wherein the at least one user is set to have respective corresponding operation permissions for the modeling project, modeling plan and / or modeling task.

31. The computing device of claim 30, wherein: The at least one user includes a modeling project main user and a modeling project participating user, wherein the modeling project main user can perform all operations on the modeling project, modeling plan and / or modeling task, and the modeling project participating user can perform limited operations on the modeling project, modeling plan and / or modeling task.

32. The computing device of claim 31 , wherein: Modeling project participating users are configured to be able to share the system resources and data resources of the modeling project main user under the modeling project.

33. The computing device of claim 29, wherein: In step (B), the at least one modeling plan is established by copying an already established modeling plan; Alternatively, in step (C), the modeling tasks involved in the corresponding data modeling activity are configured by copying the established modeling tasks.

34. The computing device of claim 29, wherein: In step (C), a DAG graph corresponding to the established modeling plan is displayed, wherein the DAG graph includes the interactive structural units for respectively configuring the modeling tasks.

35. The computing device of claim 34, wherein: The interactive structural unit includes at least one of the following items: a modeling task name, a modeling task icon, a modeling task configuration entry, and a modeling task progress indicator.

36. The computing device of claim 35, wherein: The modeling task configuration entry and the modeling task progress indicator are displayed in the same area of the interactive structure unit in a reused manner.

37. The computing device of claim 29, wherein: The modeling project established in step (A) is a rapid modeling project; and, in step (B), a rapid modeling plan is automatically established under the rapid modeling project. In step (C), after the input data records are configured according to the user's input operations under the rapid modeling plan, the corresponding feature extraction tasks and model training tasks are automatically configured. In step (D), the rapid modeling plan is automatically started.

38. The computing device of claim 37, wherein: In step (C), the feature extraction task and the model training task are automatically configured using the preset feature extraction configuration items and model training parameters.

39. The computing device of claim 29, wherein: The page for setting feature extraction configuration items is a graphical user interface, which includes a text editing interface for manually editing feature extraction configuration items and / or a selection input interface for displaying content options of feature extraction configuration items for user selection.

40. The computing device of claim 29, wherein: Step (D) further includes: downloading the saved results generated by the at least one modeling plan according to a predetermined percentage or a predetermined number of rows.

41. The computing device of claim 29, wherein: When the computer executable instruction set is executed by the processor, the following steps are also performed: (E) the evaluation report of the data model generated when the model evaluation task under the at least one modeling plan is started is displayed corresponding to the corresponding model training task and / or modeling plan.

42. The computing device of claim 29, wherein: In step (C), the model application task is configured as a manual application mode and / or an automatic application mode, wherein in the manual application mode, the model application is started according to the user's operation, and in the automatic application mode, the model application is started according to a preset time interval.

Images