Time series feature development system and method
Through the application layer, feature design components and engine layer of the time series feature development system, an automated feature development process is realized, the problem of relying on professional manpower in traditional technology is solved, efficiency and consistency is improved, and the rapid growth feature demand is met. .
Patent Information
- Application Number
- CN202411431634.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-01-14
AI Technical Summary
The traditional time series feature design and development process relies on professional labor, which is costly and inefficient, making it difficult to meet the rapidly growing feature needs.
A time series feature development system was designed, including application layer, feature design component, feature processing component and engine layer. By standardizing feature design and feature processing caliber, an automated feature development process can be realized, and users can perform features on the interface with low threshold. Development.
Without relying on professional data developers, users can independently and efficiently complete the design and development of complex time series features, improving development efficiency and consistency.
Smart Images
Figure CN119311328A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of feature derivation technology, and in particular to a time series feature development system and method. Background Art
[0002] The rapid development of big data technology has enabled e-commerce platforms, banks and other companies to accumulate a large amount of time series data. These time series data have a lot of hidden value and play an important role in e-commerce user purchase recommendations and bank transaction anti-fraud. In the field of machine learning, the data features of time series data can effectively improve the effect of machine learning models. Therefore, the demand for high-quality time series features in front-line businesses continues to grow rapidly.
[0003] At present, the design and development of time series features are generally proposed by the business demand side, then implemented by professional data developers, and finally accepted by the business demand side. Obviously, the design and development of features in the above process are separated, highly dependent on data developers, the feature delivery cycle is uncontrollable, and the feature processing methods and processing calibers of different developers are inconsistent, and the features are scattered. As a result, a large amount of professional manpower is required in the feature design and development process. The development of features is not only costly, but also inefficient, making it difficult to meet the rapidly growing feature needs. Summary of the invention
[0004] The main purpose of this application is to provide a time series feature development system and method, which aims to solve the technical problem that the traditional feature design and development process requires a large amount of professional manpower, the feature development is not only costly, but also has low development efficiency and is difficult to meet the rapidly growing feature needs.
[0005] To achieve the above-mentioned purpose, the present application proposes a time series feature development system, the system comprising: an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service;
[0006] The application layer is used to receive feature design data initiated by the user based on the target application;
[0007] The feature design component is used to read the first feature data source according to the feature design data to perform configuration, generate standardized features, and send the standardized features to the feature processing component;
[0008] The feature processing component is used to convert the standardized feature into a feature processing caliber;
[0009] The feature processing component is further used to assemble the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service, and initiate task operation to the engine layer;
[0010] The engine layer is used to perform feature processing on the task startup parameters and output the time series feature results corresponding to the target application.
[0011] In one embodiment, the feature design component is further used to read a first feature data source from a preset relational database management system according to the feature design data;
[0012] The feature design component is also used to perform standardized configuration on the first feature data source according to a preset strategy, generate standardized features corresponding to the feature design data, and send the standardized features to the feature processing component.
[0013] In one embodiment, the engine layer further includes: a task scheduling engine service;
[0014] The task scheduling engine service is used to add the task startup parameters to the feature processing task queue;
[0015] The task scheduling engine service is also used to start the feature processing task after the task resources are ready;
[0016] The task scheduling engine service is further used to perform feature processing on the task startup parameters in the feature processing task queue, and output the time series feature results corresponding to the target application.
[0017] In one embodiment, the task scheduling engine service is further used to parse the task startup parameters in the feature processing task queue to obtain task input parameters;
[0018] The task scheduling engine service is further used to read a second feature data source from a Hive database according to the task input parameters;
[0019] The task scheduling engine service is further used to perform feature processing on the second feature data source and output a time series feature result corresponding to the target application.
[0020] In one embodiment, the engine layer further includes: a big data cluster service;
[0021] The task scheduling engine service is further used to send the time series feature results to the distributed file system of the big data cluster service for storage after the feature processing is completed;
[0022] The application layer is also used to generate a processing completion notification and push it to the user after the feature processing is completed.
[0023] In one embodiment, the component layer also includes a feature production component;
[0024] The application layer is also used to receive a feature production instruction initiated by a user based on a processing completion notification;
[0025] The characteristic production component is used to configure the time series characteristic result according to the characteristic production instruction to obtain production configuration data;
[0026] The feature production component is also used to initiate production launch to the engine layer according to the production configuration data.
[0027] In one embodiment, the component layer further includes a feature library component;
[0028] The engine layer is further used to perform deployment according to the production configuration data to obtain production deployment data;
[0029] The engine layer is further used to create a feature table corresponding to the time series feature result based on the production deployment data;
[0030] The engine layer is further used to register the feature tables in batches to the feature library component to maintain feature information;
[0031] The application layer is also used to generate a production completion notification and push it to the user after the registration is completed.
[0032] In addition, to achieve the above purpose, the present application also proposes a time series feature development method, which is applied to a time series feature development system, the system comprising an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service; the method comprising:
[0033] The application layer receives feature design data initiated by the user based on the target application;
[0034] The feature design component reads a first feature data source for configuration according to the feature design data, generates standardized features, and sends the standardized features to the feature processing component;
[0035] The feature processing component converts the standardized feature into a feature processing caliber;
[0036] The feature processing component assembles the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service, and initiates task operation to the engine layer;
[0037] The engine layer performs feature processing on the task startup parameters and outputs a time series feature result corresponding to the target application.
[0038] In one embodiment, the step of the feature design component reading the first feature data source for configuration according to the feature design data, generating standardized features, and sending the standardized features to the feature processing component includes:
[0039] The feature design component reads a first feature data source from a preset relational database management system according to the feature design data;
[0040] The feature design component performs standardized configuration on the first feature data source according to a preset strategy, generates standardized features corresponding to the feature design data, and sends the standardized features to the feature processing component.
[0041] In one embodiment, the engine layer further includes a task scheduling engine service; the engine layer performs feature processing on the task startup parameters and outputs the time series feature results corresponding to the target application, including:
[0042] The task scheduling engine service adds the task startup parameters into the feature processing task queue;
[0043] The task scheduling engine service starts the feature processing task after the task resources are ready;
[0044] The task scheduling engine service performs feature processing on the task startup parameters in the feature processing task queue, and outputs a time series feature result corresponding to the target application.
[0045] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes a time series feature development system, the system comprising: an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service; the application layer is used to receive feature design data initiated by a user based on a target application; the feature design component is used to read the first feature data source according to the feature design data for configuration, generate standardized features, and send the standardized features to the feature processing component; the feature processing component is used to convert the standardized features into feature processing calibers; the feature processing component is also used to assemble the feature processing calibers into task start parameters through the parameter template provided by the feature processing template service, and initiate task operation to the engine layer; the engine layer is used to perform feature processing on the task start parameters and output the time series feature results corresponding to the target application. Since this application uses the first feature data source built into the time series feature development system, it performs flexible feature processing according to the feature design component and the feature processing component, and can realize feature design and development of different data sources. Thereby, it can help feature demanders to independently and efficiently complete the design and development of complex time series features without relying on professional data developers, thereby improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0048] Figure 1 Functional module diagram provided for the first embodiment of the time series feature development system of this application;
[0049] Figure 2 A schematic diagram of the composition of the component modules provided in Example 1 of the time series feature development system of the present application;
[0050] Figure 3 A flow chart of the feature production provided in Example 3 of the present application;
[0051] Figure 4 A flowchart of the feature design and development provided for Example 3 of this application;
[0052] Figure 5A flowchart of the first embodiment of the time series feature development method of the present application is provided.
[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0056] The present application embodiment provides a time series feature development system, referring to Figure 1 , Figure 1 This is a functional module diagram provided for Example 1 of the time series feature development system of this application.
[0057] In this embodiment, the system includes: an application layer 10 , a component layer 20 including a feature design component 21 and a feature processing component 22 , and an engine layer 30 including a feature processing template service 31 .
[0058] The application layer 10 is used to receive feature design data initiated by the user based on the target application.
[0059] It should be noted that the application layer is a structural layer that integrates various applications in the system and can provide interface point-and-click services. The target application is the application that the feature demander needs to develop time series features.
[0060] It should be noted that feature design data refers to the features designed for the original time series data itself in the process of developing time series features, such as statistical features, trend features, periodic features, seasonal features, time interval features, etc.
[0061] Specifically, the user can select a target application for which time series feature development is required on the interface of the application layer, and then upload the feature design data to be developed for the target application.
[0062] In this implementation, through the point-and-click operation of the application layer interface, it is easy to use and does not require professional data development capabilities, allowing non-data developers to develop feature requirements in a low-threshold and process-based manner.
[0063] The feature design component 21 is used to read the first feature data source according to the feature design data to perform configuration, generate standardized features, and send the standardized features to the feature processing component 22.
[0064] It should be noted that the feature design component is a component in the system that provides feature design services, mainly to standardize the feature design process.
[0065] It should be noted that the first feature data source is the surface data corresponding to the original data source used to construct time series related features stored in the system, such as the name of the original data source, the location and other basic information.
[0066] In the subsequent feature processing process, the required original data source can be directly located through the first feature data source, reducing data redundancy of feature design components and improving development efficiency.
[0067] It should be noted that the standardized feature is a standardized feature obtained after the first feature data source is standardized.
[0068] Specifically, after reading the first feature data source, the feature design component can classify the first feature data source according to the name of the original data source and divide it into multiple feature types to be developed; then, according to the multiple feature types to be developed, match the corresponding processing methods (such as cross-type, ratio-type, composite-type, etc.), and finally obtain the standardized features to be developed.
[0069] In this implementation, the system's built-in scenario-based first feature data source can be flexibly arranged to reduce data redundancy of feature design components and improve development efficiency.
[0070] In a feasible implementation manner, the feature design component 21 described in this embodiment is also used to read the first feature data source from a preset relational database management system according to the feature design data; the feature design component 21 is also used to standardize the first feature data source according to a preset strategy, generate standardized features corresponding to the feature design data, and send the standardized features to the feature processing component.
[0071] It should be noted that the preset relational database management system (RDBMS) is a software system used to manage relational databases in the time series feature development system. The preset relational database management system organizes and stores the original data sources used to construct time series related features according to a specific relational model, and it presents the data in the form of tables. The preset relational database management system can facilitate the time series feature development system to effectively store, manage and retrieve data.
[0072] Exemplarily, the preset relational database management system may include MySQL, Oracle, and SQL Server, which is not limited in this embodiment.
[0073] It should be noted that the preset strategy is a pre-set strategy for standardizing the configuration of the first feature data source, such as Figure 2 As shown, standardization in aspects such as data source configuration, indicator type and processing method selection, and time window definition is shown.
[0074] Among them, the indicator type supports multiple data types such as numeric, discrete, date, character, etc. The processing method supports multiple calculation and processing methods such as cross type, ratio type, composite type, etc. The time window supports day, week, month, etc.
[0075] Specifically, after the user initiates feature design, the feature design component can read the first feature data source from MySQL and perform feature configuration, including: indicator type, processing method, time window; and finally obtain standardized features.
[0076] In this implementation, by presetting a relational database management system, the time series feature development system can be convenient for effectively storing, managing and retrieving the original data source of time series related features.
[0077] The feature processing component 22 is used to convert the standardized feature into a feature processing caliber.
[0078] It should be noted that if Figure 2 As shown, the feature processing component is a component that converts the designed standardized features into feature processing calibers for display. The component supports the functions of feature processing trial run, viewing trial run results, and evaluating feature effects.
[0079] It should be noted that the feature processing caliber is a unified standard followed when processing data features. The feature processing caliber can clearly define the specific requirements for converting the original data source into meaningful features, including the data selection range, calculation method, processing flow, etc.
[0080] In this embodiment, the feature processing aperture can ensure consistency and accuracy when performing feature processing in different links.
[0081] The feature processing component 22 is also used to assemble the feature processing aperture into task startup parameters through the parameter template provided by the feature processing template service 31, and initiate task operation to the engine layer 30.
[0082] It should be noted that the feature processing template service is a service module that provides parameter templates for feature design and processing modules. Through the feature processing template service, the feature processing caliber can be converted into the input parameters of the feature processing task.
[0083] Among them, the parameter template is a formal template corresponding to different operations on the feature processing aperture, such as a parameter template for operations such as summation and averaging.
[0084] It should be noted that the task startup parameters provide configuration information when starting the feature processing task, such as the file path, port number, configuration options, etc. specified by the feature processing.
[0085] Specifically, after generating the feature processing aperture, the feature processing component can also assemble the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service; then generate the task and start running.
[0086] In this implementation, the feature processing template service can be used to keep the feature processing method and caliber consistent, facilitating unified maintenance and management.
[0087] The engine layer 30 is used to perform feature processing on the task startup parameters and output a time series feature result corresponding to the target application.
[0088] It should be noted that the engine layer is a system layer that provides services such as execution, monitoring, and task stopping of feature processing tasks, and can provide functions such as processing task queuing and distributed task execution.
[0089] It should be noted that the time series feature results are data point features arranged in chronological order developed by the time series feature development system according to user needs, including trends (long-term rising or falling trends), seasonality (regular fluctuations within a cycle), periodicity (repeated patterns in longer cycles), volatility (the magnitude of data changes), and other features.
[0090] Specifically, after obtaining the task startup parameters, the engine layer can arrange the task startup parameters through a priority queue; then, according to the order of the priority queue, the data source corresponding to the task startup parameters is processed and the effect is evaluated. If the feature effect meets the user's needs, the time series feature results are output.
[0091] In this implementation, time series feature processing can be performed through the engine layer to meet feature processing requirements under different data sources, different calculation types, and different time window combinations.
[0092] In one possible implementation, please refer to Figure 2 , Figure 2A schematic diagram of the composition of the component modules provided for the first embodiment of the time series feature development system of this application. The engine layer 30 described in this example also includes: a task scheduling engine service 32; the task scheduling engine service 32 is used to add the task startup parameters to the feature processing task queue; the task scheduling engine service 32 is also used to start the feature processing task after the task resources are ready; the task scheduling engine service 32 is also used to perform feature processing on the task startup parameters in the feature processing task queue and output the time series feature results corresponding to the target application.
[0093] It should be noted that the task scheduling engine service is a service module that schedules and executes feature processing tasks, and also provides services such as monitoring of the task execution process and stopping tasks.
[0094] It should be noted that the feature processing task queue is an ordered list for storing tasks related to feature processing. Task startup parameters are added to this queue according to a certain order or rule (such as first-in-first-out, priority, etc.), waiting to be executed and processed in turn.
[0095] In this implementation, the feature processing task queue can ensure that the feature processing work is carried out in an orderly manner.
[0096] In a feasible implementation manner, the task scheduling engine service 32 described in this embodiment is also used to parse the task startup parameters in the feature processing task queue to obtain task input parameters; the task scheduling engine service 32 is also used to read the second feature data source from the Hive database according to the task input parameters; the task scheduling engine service 32 is also used to perform feature processing on the second feature data source and output the time series feature results corresponding to the target application.
[0097] It should be noted that the task input parameters are the data executed by the task scheduling engine service after parsing the task startup parameters.
[0098] It should be noted that the Hive database is a data warehouse tool based on Hadoop. The Hive database stores the original data sources used to construct time series related features.
[0099] It should be noted that the second feature data source is the original data source used to construct time series related features. It should be noted that unlike the first feature data source, the first feature data source is surface data, such as the name of the original data source, location and other basic information; while the second feature data source is the real original data source.
[0100] Specifically, after adding the task startup parameters to the feature processing task queue, the task scheduling engine service can first detect whether the task resources are ready. If the task resources are ready, the feature processing process is scheduled to be executed: first, the second feature data source is read from the Hive database, and then feature processing is performed according to the second feature data source. Then, the effect is evaluated according to the processing data of the feature processing. If the effect meets the user's needs, the target application (such as Figure 2 As shown, the time series feature results corresponding to various applications (such as application A, application B, application N, etc.)
[0101] In the technical solution provided in this embodiment, a time series feature development system is proposed, and the system includes: an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service; the user can select the target application that needs to be developed for time series features on the interface of the application layer, and then upload the feature design data to be developed for the target application. After the user initiates the feature design, the feature design component can read the first feature data source from MySQL and perform feature configuration, including: indicator type, processing method, time window; and finally obtain standardized features. After generating the feature processing caliber, the feature processing component can also assemble the feature processing caliber into task startup parameters through the parameter template provided by the feature processing template service; then generate the task and start running. Then the engine layer adds the task startup parameters to the feature processing task queue, and the task scheduling engine service can first detect whether the task resources are ready. If the task resources are ready, the scheduling execution feature processing process is started: first, the second feature data source is read from the Hive database, and then the feature processing is performed according to the second feature data source, and then the effect is evaluated according to the processing data of the feature processing. If the effect meets the user's needs, the time series feature results corresponding to the target application are output. Since this embodiment uses the first feature data source built into the time series feature development system to perform flexible feature processing according to the feature design component and the feature processing component, it can realize feature design and development of different data sources. This can help feature demanders to independently and efficiently complete the design and development of complex time series features without relying on professional data developers, thereby improving development efficiency.
[0102] Based on the above embodiments of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction and will not be repeated later.
[0103] like Figure 2As shown, the engine layer 30 of this example also includes: a big data cluster service 33; the task scheduling engine service 32 is also used to send the time series feature results to the distributed file system of the big data cluster service 33 for storage after the feature processing is completed; the application layer 10 is also used to generate a processing completion notification and push it to the user after the feature processing is completed.
[0104] It should be noted that the big data cluster service is a service module that stores time series feature results in a distributed manner.
[0105] It should be noted that a distributed file system is a file system that stores data related to time series feature results on multiple nodes, including but not limited to HDFS, Hadoop, Spark, Yarn, Hive, etc.
[0106] In this embodiment, after the feature processing is completed, the task scheduling engine service sends the time series feature results to the distributed file system of the big data cluster service for storage; and after the storage is completed, the generated processing completion notification is pushed to the application layer for display to the user.
[0107] In the technical solution provided in this embodiment, the data of the time series feature results can be divided and stored on different physical nodes through a distributed file system to achieve better storage expansion performance.
[0108] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 , Figure 3 A flow chart of feature production provided in Example 3 of the present application.
[0109] like Figure 3 As shown, the feature development process of the time series feature development system of the present application is divided into four processes. The first is feature design, which is implemented at the component layer. The design process may include data source configuration, selection of indicator type and processing method, and definition of time window. Then comes feature processing, which is implemented in the task scheduling engine service of the engine layer. The next step is feature production, which is implemented in the feature production component of the component layer. The production process may include selecting the person in charge of production and filling in the scheduling execution cycle. Finally, the production results are saved to the database. The specific implementation process is as follows:
[0110] like Figure 2As shown, the component layer 20 of this example also includes a feature production component 23; the application layer 10 is also used to receive a feature production instruction initiated by a user based on a processing completion notification; the feature production component 23 is used to configure the time series feature results according to the feature production instruction to obtain production configuration data; the feature production component 23 is also used to initiate production online to the engine layer 30 according to the production configuration data.
[0111] It should be noted that the feature production component is a component that can be put into production with one click after the feature design and trial run are completed. This component supports functions such as feature production application, feature production configuration, feature offline application, output result table creation, quick approval deployment, and batch scheduling execution.
[0112] It should be noted that the feature production instruction is an instruction for executing the actual application of the time series feature results.
[0113] It should be noted that the production configuration data is the relevant data configured during the actual operation of the time series feature results, such as selecting the production person in charge, selecting the scheduling cycle, and other configuration data.
[0114] Specifically, after receiving the pushed notification of feature processing completion, the user initiates a feature production instruction to the application layer. The feature production component performs production configuration according to the feature production instruction and initiates production launch to the engine layer.
[0115] In this implementation, the production configuration can ensure that the system can operate normally and effectively after production, so as to ensure that the system can operate and be put into production in an expected manner.
[0116] Furthermore, if Figure 2 As shown, the component layer 20 of this example also includes a feature library component 24; the engine layer 30 is also used to deploy according to the production configuration data to obtain production deployment data; the engine layer 30 is also used to create a feature table corresponding to the time series feature result based on the production deployment data; the engine layer 30 is also used to batch register the feature table to the feature library component 24 for feature information maintenance; the application layer 10 is also used to generate a production completion notification and push it to the user after the registration is completed.
[0117] It should be noted that the feature library component is a component that registers and saves the data after the feature is put into production. This component supports the maintenance of feature metadata, feature classification system maintenance, feature quality monitoring and alarm, etc.
[0118] It should be noted that the production deployment data refers to various types of data involved when the time series feature results are put into use and deployed.
[0119] It should be noted that the feature table is a table that records various feature information related to feature production deployment.
[0120] In this embodiment, after the production deployment is completed, the engine layer can also create a feature table corresponding to the time series feature results and register them in batches to the feature library component for feature information maintenance, so as to facilitate data management and maintenance of feature production.
[0121] In the technical solution provided in this embodiment, the system can be guaranteed to operate normally and effectively after production through production configuration, so as to ensure that the system can run and be put into production in the expected manner. After the production deployment is completed, the engine layer can also create a feature table corresponding to the time series feature results and register them in batches to the feature library component for feature information maintenance, so as to facilitate data management and maintenance of feature production.
[0122] For example, to help understand the implementation process of the time series feature development system obtained in each embodiment, please refer to Figure 4 , Figure 4 This is a flowchart of feature design and development provided in Example 3 of the present application. The feature processing workflow of the present application mainly includes the following steps:
[0123] (1) After the user initiates feature design to the application layer, the component layer reads the feature data source (i.e., the first feature data source) from MySQL and performs feature configuration, including indicator type, processing method, time window, etc.
[0124] (2) After the feature configuration is completed, the feature processing caliber is generated and stored in Spark SQL, and then the feature processing caliber is assembled into task startup parameters (task startup parameters are macro variables, a variable with a fixed value or expression).
[0125] (3) Then generate the task and initiate the task execution to the engine layer.
[0126] (4) In the engine layer, tasks enter the task scheduling engine service for queueing.
[0127] (5) If it is detected that the task resources are ready, the feature processing task (i.e., Spark task) is started and scheduled to execute feature processing: first, the task input parameters (i.e., macro variables) are parsed, and then the second feature data source is read from the Hive database. Then, the second feature data source is processed and the effect is evaluated; finally, the feature results are output and stored in HDFS.
[0128] (6) After the engine layer completes the feature processing event, the application layer pushes the feature processing completion notification to the user.
[0129] (7) After receiving the notification that feature processing is complete, the user initiates feature production. At this time, the component layer performs production configuration, including selecting the production manager and the scheduling cycle. After the production configuration is completed, the production launch is initiated to the engine layer.
[0130] (8) After receiving the notification of production launch, the engine layer performs production deployment and creates a feature table in the Hive database. At the same time, the component layer batch registers features to the feature library.
[0131] (9) After the feature is put into operation, the application layer pushes a notification of feature completion to the user.
[0132] The present application embodiment provides a time series feature development method, referring to Figure 5 , Figure 5 A flowchart of the first embodiment of the time series feature development method of the present application is provided.
[0133] The time series feature development method is applied to the above-mentioned time series feature development system, which includes an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service. The method described in this embodiment includes:
[0134] Step S10: The application layer receives feature design data initiated by the user based on the target application.
[0135] It should be noted that the application layer is a structural layer that integrates various applications in the system and can provide interface point-and-click services. The target application is the application that the feature demander needs to develop time series features.
[0136] It should be noted that feature design data refers to the features designed for the original time series data itself in the process of developing time series features, such as statistical features, trend features, periodic features, seasonal features, time interval features, etc.
[0137] Specifically, the user can select a target application for which time series feature development is required on the interface of the application layer, and then upload the feature design data to be developed for the target application.
[0138] In this implementation, through the point-and-click operation of the application layer interface, it is easy to use and does not require professional data development capabilities, allowing non-data developers to develop feature requirements in a low-threshold and process-based manner.
[0139] Step S20: the feature design component reads the first feature data source according to the feature design data for configuration, generates standardized features, and sends the standardized features to the feature processing component.
[0140] It should be noted that the feature design component is a component in the system that provides feature design services, mainly to standardize the feature design process.
[0141] It should be noted that the first feature data source is the surface data corresponding to the original data source used to construct time series related features stored in the system, such as the name of the original data source, the location and other basic information.
[0142] In the subsequent feature processing process, the required original data source can be directly located through the first feature data source, reducing data redundancy of feature design components and improving development efficiency.
[0143] It should be noted that the standardized feature is a standardized feature obtained after the first feature data source is standardized.
[0144] Specifically, after reading the first feature data source, the feature design component can classify the first feature data source according to the name of the original data source and divide it into multiple feature types to be developed; then, according to the multiple feature types to be developed, match the corresponding processing methods (such as cross-type, ratio-type, composite-type, etc.), and finally obtain the standardized features to be developed.
[0145] In this implementation, the system's built-in scenario-based first feature data source can be flexibly arranged to reduce data redundancy of feature design components and improve development efficiency.
[0146] In a feasible implementation, step S20 of this embodiment may include the steps of: the feature design component reads a first feature data source from a preset relational database management system according to the feature design data; the feature design component performs standardized configuration on the first feature data source according to a preset strategy, generates standardized features corresponding to the feature design data, and sends the standardized features to the feature processing component.
[0147] It should be noted that the preset relational database management system (RDBMS) is a software system used to manage relational databases in the time series feature development system. The preset relational database management system organizes and stores the original data sources used to construct time series related features according to a specific relational model, and it presents the data in the form of tables. The preset relational database management system can facilitate the time series feature development system to effectively store, manage and retrieve data.
[0148] Exemplarily, the preset relational database management system may include MySQL, Oracle, and SQL Server, which is not limited in this embodiment.
[0149] It should be noted that the preset strategy is a pre-set strategy for standardizing the configuration of the first feature data source, such as Figure 2 As shown, standardization in aspects such as data source configuration, indicator type and processing method selection, and time window definition is shown.
[0150] Among them, the indicator type supports multiple data types such as numeric, discrete, date, character, etc. The processing method supports multiple calculation and processing methods such as cross type, ratio type, composite type, etc. The time window supports day, week, month, etc.
[0151] Specifically, after the user initiates feature design, the feature design component can read the first feature data source from MySQL and perform feature configuration, including: indicator type, processing method, time window; and finally obtain standardized features.
[0152] In this implementation, by presetting a relational database management system, the time series feature development system can be convenient for effectively storing, managing and retrieving the original data source of time series related features.
[0153] Step S30: The feature processing component converts the standardized feature into a feature processing aperture.
[0154] It should be noted that if Figure 2 As shown, the feature processing component is a component that converts the designed standardized features into feature processing calibers for display. The component supports the functions of feature processing trial run, viewing trial run results, and evaluating feature effects.
[0155] It should be noted that the feature processing caliber is a unified standard followed when processing data features. The feature processing caliber can clearly define the specific requirements for converting the original data source into meaningful features, including the data selection range, calculation method, processing flow, etc.
[0156] In this embodiment, the feature processing aperture can ensure consistency and accuracy when performing feature processing in different links.
[0157] Step S40: the feature processing component assembles the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service, and initiates task execution to the engine layer.
[0158] It should be noted that the feature processing template service is a service module that provides parameter templates for feature design and processing modules. Through the feature processing template service, the feature processing caliber can be converted into the input parameters of the feature processing task.
[0159] Among them, the parameter template is a formal template corresponding to different operations on the feature processing aperture, such as a parameter template for operations such as summation and averaging.
[0160] It should be noted that the task startup parameters provide configuration information when starting the feature processing task, such as the file path, port number, configuration options, etc. specified by the feature processing.
[0161] Specifically, after generating the feature processing aperture, the feature processing component can also assemble the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service; then generate the task and start running.
[0162] In this implementation, the feature processing template service can be used to keep the feature processing method and caliber consistent, facilitating unified maintenance and management.
[0163] Step S50: the engine layer performs feature processing on the task startup parameters and outputs a time series feature result corresponding to the target application.
[0164] It should be noted that the engine layer is a system layer that provides services such as execution, monitoring, and task stopping of feature processing tasks, and can provide functions such as processing task queuing and distributed task execution.
[0165] It should be noted that the time series feature results are data point features arranged in chronological order developed by the time series feature development system according to user needs, including trends (long-term rising or falling trends), seasonality (regular fluctuations within a cycle), periodicity (repeated patterns in longer cycles), volatility (the magnitude of data changes), and other features.
[0166] Specifically, after obtaining the task startup parameters, the engine layer can arrange the task startup parameters through a priority queue; then, according to the order of the priority queue, the data source corresponding to the task startup parameters is processed and the effect is evaluated. If the feature effect meets the user's needs, the time series feature results are output.
[0167] In this implementation, time series feature processing can be performed through the engine layer to meet feature processing requirements under different data sources, different calculation types, and different time window combinations.
[0168] In a feasible implementation, the engine layer also includes a task scheduling engine service; step S50 of this embodiment may include the steps of: the task scheduling engine service adds the task startup parameters to a feature processing task queue; the task scheduling engine service starts the feature processing task after the task resources are ready; the task scheduling engine service performs feature processing on the task startup parameters in the feature processing task queue, and outputs the time series feature results corresponding to the target application.
[0169] It should be noted that the task scheduling engine service is a service module that schedules and executes feature processing tasks, and also provides services such as monitoring of the task execution process and stopping tasks.
[0170] It should be noted that the feature processing task queue is an ordered list for storing tasks related to feature processing. Task startup parameters are added to this queue according to a certain order or rule (such as first-in-first-out, priority, etc.), waiting to be executed and processed in turn.
[0171] In this implementation, the feature processing task queue can ensure that the feature processing work is carried out in an orderly manner.
[0172] In the technical solution provided in this embodiment, the user can select the target application that needs to be developed for time series features on the interface of the application layer, and then upload the feature design data to be developed for the target application. After the user initiates the feature design, the feature design component can read the first feature data source from MySQL and perform feature configuration, including: indicator type, processing method, time window; and finally obtain standardized features. After generating the feature processing caliber, the feature processing component can also assemble the feature processing caliber into task startup parameters through the parameter template provided by the feature processing template service; then generate the task and start running. Then the engine layer adds the task startup parameters to the feature processing task queue, and the task scheduling engine service can first detect whether the task resources are ready. If the task resources are ready, the scheduling execution feature processing process begins: first read the second feature data source from the Hive database, then perform feature processing according to the second feature data source, and then evaluate the effect according to the processing data of the feature processing. If the effect meets the user's needs, the time series feature results corresponding to the target application are output. Since this embodiment uses the first feature data source built into the time series feature development system, flexible feature processing is performed according to the feature design component and the feature processing component, and feature design and development of different data sources can be realized. This can help feature demanders to independently and efficiently complete the design and development of complex time series features without relying on professional data developers, thereby improving development efficiency.
[0173] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the time series feature development method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0174] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A time series feature development system, characterized in that: The system comprises: an application layer, a component layer including a feature design component and a feature processing component, and an engine layer including a feature processing template service; The application layer is used to receive feature design data initiated by the user based on the target application; The feature design component is used to read the first feature data source according to the feature design data to perform configuration, generate standardized features, and send the standardized features to the feature processing component; The feature processing component is used to convert the standardized feature into a feature processing caliber; The feature processing component is further used to assemble the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service, and initiate task operation to the engine layer; The engine layer is used to perform feature processing on the task startup parameters and output the time series feature results corresponding to the target application.
2. The system according to claim 1, characterized in that The feature design component is further used to read a first feature data source from a preset relational database management system according to the feature design data; The feature design component is also used to perform standardized configuration on the first feature data source according to a preset strategy, generate standardized features corresponding to the feature design data, and send the standardized features to the feature processing component.
3. The system according to claim 1, characterized in that The engine layer also includes: task scheduling engine services; The task scheduling engine service is used to add the task startup parameters to the feature processing task queue; The task scheduling engine service is also used to start the feature processing task after the task resources are ready; The task scheduling engine service is further used to perform feature processing on the task startup parameters in the feature processing task queue, and output the time series feature results corresponding to the target application.
4. The system according to claim 3, characterized in that The task scheduling engine service is also used to parse the task startup parameters in the feature processing task queue to obtain task input parameters; The task scheduling engine service is further used to read a second feature data source from a Hive database according to the task input parameters; The task scheduling engine service is further used to perform feature processing on the second feature data source and output a time series feature result corresponding to the target application.
5. The system according to claim 4, characterized in that The engine layer also includes: big data cluster services; The task scheduling engine service is further used to send the time series feature results to the distributed file system of the big data cluster service for storage after the feature processing is completed; The application layer is also used to generate a processing completion notification and push it to the user after the feature processing is completed.
6. The system according to any one of claims 1 to 5, characterized in that The component layer also includes feature production components; The application layer is also used to receive a feature production instruction initiated by a user based on a processing completion notification; The characteristic production component is used to configure the time series characteristic result according to the characteristic production instruction to obtain production configuration data; The feature production component is also used to initiate production launch to the engine layer according to the production configuration data.
7. The system according to claim 6, characterized in that The component layer also includes a feature library component; The engine layer is further used to perform deployment according to the production configuration data to obtain production deployment data; The engine layer is further used to create a feature table corresponding to the time series feature result based on the production deployment data; The engine layer is further used to register the feature tables in batches to the feature library component to maintain feature information; The application layer is also used to generate a production completion notification and push it to the user after the registration is completed.
8. A time series feature development method, characterized in that: The time series feature development method is applied to a time series feature development system, the system comprising an application layer, a component layer comprising a feature design component and a feature processing component, and an engine layer comprising a feature processing template service; the method comprising: The application layer receives feature design data initiated by the user based on the target application; The feature design component reads a first feature data source for configuration according to the feature design data, generates standardized features, and sends the standardized features to the feature processing component; The feature processing component converts the standardized feature into a feature processing caliber; The feature processing component assembles the feature processing aperture into task start parameters through the parameter template provided by the feature processing template service, and initiates task operation to the engine layer; The engine layer performs feature processing on the task startup parameters and outputs a time series feature result corresponding to the target application.
9. The method according to claim 8, characterized in that The step of the feature design component reading a first feature data source for configuration according to the feature design data, generating standardized features, and sending the standardized features to the feature processing component comprises: The feature design component reads a first feature data source from a preset relational database management system according to the feature design data; The feature design component performs standardized configuration on the first feature data source according to a preset strategy, generates standardized features corresponding to the feature design data, and sends the standardized features to the feature processing component.
10. The method according to claim 8, characterized in that The engine layer also includes a task scheduling engine service; the engine layer performs feature processing on the task startup parameters and outputs the time series feature results corresponding to the target application, including: The task scheduling engine service adds the task startup parameters into the feature processing task queue; The task scheduling engine service starts the feature processing task after the task resources are ready; The task scheduling engine service performs feature processing on the task startup parameters in the feature processing task queue, and outputs a time series feature result corresponding to the target application.