Data processing method and system for feature extraction

Abstract

A data processing method, system and storage medium for feature extraction are disclosed. A base layer includes one or more data processing functions; a calculation layer includes one or more feature calculation modules, the feature calculation modules obtain data and feature calculation element definition information, the feature calculation element definition information includes at least one category of feature calculation elements, each category of feature calculation elements is used to indicate one element of feature extraction processing, the feature calculation modules select one feature calculation element from each category of feature calculation elements to obtain a feature extraction processing strategy composed of feature calculation elements from different categories, the feature calculation modules extract data objects subjected to feature extraction processing from the data based on the feature extraction processing strategy, and determine data processing functions for performing feature extraction processing on the data objects. Thus, the feature calculation efficiency can be improved by automatically combining feature calculation elements using the feature calculation modules.

Description

[0001] This application is a divisional application of patent application No. 201910877370.6, filed on September 17, 2019, entitled "Data Processing Method and System for Feature Extraction". Technical Field

[0002] This invention generally relates to the field of data processing, and more specifically, to a data processing method, system, and storage medium for extracting features. Background Technology

[0003] In the field of data processing, feature extraction is often required to extract features from data that can represent specific meanings. For example, for data records consisting of multiple fields, specific calculation operations are often performed on specific fields or combinations of specific fields to obtain features that meet the requirements.

[0004] However, existing solutions only support generating feature calculation results in a one-to-one correspondence manner, meaning that a set of feature calculation element definitions must be provided for each feature to be calculated. When multiple features need to be calculated, duplicate feature calculation element definitions cannot be reused; instead, multiple sets of feature calculation element definitions need to be provided.

[0005] For example, suppose we want to calculate the feature set "minimum, maximum, average, median, and standard deviation of monthly user income in the past year" and "minimum, maximum, average, median, and standard deviation of monthly merchant income in the past year". For each feature, we need to repeatedly define and assign values ​​to the aggregate function, which is the feature calculation element. This means that the aggregate function, such as "minimum", is repeatedly used for "minimum of monthly user income in the past year" and "minimum of monthly merchant income in the past year".

[0006] It is evident that when a large number of features need to be generated and these features share the same feature calculation elements, existing solutions require a lot of mechanical and repetitive work, which is prone to human error and inconvenient for adding, deleting, or modifying features.

[0007] Therefore, a data processing solution that can improve efficiency is needed. Summary of the Invention

[0008] The exemplary embodiments of the present invention aim to overcome the inefficiency of existing feature extraction methods.

[0009] According to a first aspect of the present invention, a data processing system for feature extraction is proposed, comprising: a base layer including one or more data processing functions; and a computation layer including one or more feature computation modules. The feature computation modules acquire data and feature computation element definition information, the feature computation element definition information including at least one category of feature computation elements, each category of feature computation elements indicating an element for feature extraction processing. The feature computation modules select one feature computation element from each category of feature computation elements to obtain a feature extraction processing strategy composed of feature computation elements from different categories. Based on the feature extraction processing strategy, the feature computation modules extract data objects from the data to be subjected to feature extraction processing and determine the data processing function to perform feature extraction processing on the data objects.

[0010] Optionally, the feature calculation module selects one feature calculation element from the feature calculation elements of each category in a Cartesian product manner to obtain one or more feature extraction processing strategies.

[0011] Optionally, the feature calculation element definition information includes multiple feature calculation element sets, each feature calculation element set includes at least one set of feature calculation elements, and the feature calculation elements in different sets correspond to different categories. For each feature calculation element set, the feature calculation module selects one feature calculation element from each set of feature calculation elements in a Cartesian product manner to obtain multiple feature extraction processing strategies.

[0012] Optionally, the feature calculation module also receives a data processing function provided by the user and adds the data processing function to the base layer.

[0013] Optionally, the computation layer is divided into an implementation layer and an application layer. The implementation layer includes one or more first-type feature computation modules. The first-type feature computation modules are used to extract features corresponding to general business scenarios. The first-type feature computation modules obtain feature extraction processing strategies based solely on feature computation element definition information obtained from external sources. The application layer includes one or more second-type feature computation modules. The second-type feature computation modules are used to extract features corresponding to specific business scenarios. The second-type feature computation modules define at least some feature computation elements that constitute the feature extraction processing strategy. At least some feature computation elements are used to indicate one or more elements of the feature extraction processing corresponding to specific business scenarios.

[0014] Optionally, the first type of feature calculation module in the implementation layer can call the data processing function in the base layer, and / or the first type of feature calculation module can call other first type of feature calculation modules in the same layer; the second type of feature calculation module in the application layer can call the data processing function in the base layer, and / or the second type of feature calculation module can call other second type of feature calculation modules in the same layer, and / or the second type of feature calculation module can call the first type of feature calculation module in the implementation layer.

[0015] Optionally, the base layer may also include: a checking function for checking whether the feature calculation element definition information conforms to the specification; and / or a normalization function for converting the feature calculation element definition information into a data structure conforming to a predetermined form; and / or a constant library for providing one or more constants.

[0016] Optionally, the data includes multiple data records, each data record including one or more fields, and at least one category of feature calculation elements includes: a first category of feature calculation elements, including one or more target fields or combinations of target fields, the target fields or combinations of target fields being used to indicate the field names corresponding to the field values ​​for which feature extraction processing is performed; and / or a second category of feature calculation elements, including one or more windows or combinations of windows, the windows or combinations of windows being used to indicate the range corresponding to the field values ​​for which feature extraction processing is performed; and / or a third category of feature calculation elements, including one or more grouping fields or combinations of grouping fields, the grouping fields or combinations of grouping fields being used to indicate the field names on which the data is grouped; and / or a fourth category of feature calculation elements, including one or more algorithm names, the algorithm names being used to indicate the data processing logic for the field values ​​for which feature extraction processing is performed; and / or a fifth category of feature calculation elements, including one or more sorting fields, the sorting fields being used to indicate the field names on which the data is sorted.

[0017] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, and an algorithm name. The feature calculation module groups the data according to the grouping field or the combination of grouping fields. Data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group. The feature calculation module selects the field value corresponding to the field name indicated by the target field or the combination of target fields from each group of data to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0018] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, a window or a combination of windows, a sorting field, and an algorithm name. The feature calculation module sorts the data according to the field name indicated by the sorting field. For each data record, the feature calculation module extracts data records within the corresponding window range according to the range indicated by the window or window combination and the field name indicated by the sorting field to obtain first data that includes or excludes the data record. The feature calculation module selects data records from the first data that have the same field value as the field name indicated by the grouping field or grouping field combination in the data record to obtain second data that includes the data record. The feature calculation module selects field values ​​from the second data that correspond to the field name indicated by the target field or target field combination to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0019] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, a window or a combination of windows, a sorting field, and an algorithm name. The feature calculation module sorts the data according to the field name indicated by the sorting field. For each data record, the feature calculation module selects data records from the data that have the same field value as the field name indicated by the grouping field or the combination of grouping fields, to obtain third data including that data record. The feature calculation module extracts data records corresponding to the window range from the third data according to the range indicated by the window or the combination of windows and the field name indicated by the sorting field, to obtain fourth data including or excluding that data record. The feature calculation module selects field values ​​corresponding to the field name indicated by the target field or the combination of target fields from the fourth data to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0020] Optionally, one or more computing nodes in the distributed system can be used to call the corresponding data processing functions in the base layer to process the data object and obtain one or more features.

[0021] Optionally, the computation layer includes a time-series feature computation module and a summary feature computation module. The time-series feature computation module acquires time-series data and first feature computation element definition information. The time-series data includes multiple data records recorded in chronological order. Each data record includes one or more fields. The first feature computation element definition information includes one or more grouping fields or combinations of grouping fields. The grouping fields or combinations of grouping fields are used to indicate the field names on which the time-series data is grouped. The time-series feature computation module calls the summary feature computation module, which counts the frequency of each field value corresponding to the field name in the time-series data. The system also includes an optimization computation module, which is used to split the data records corresponding to each field value whose frequency is greater than or equal to a first predetermined threshold into a predetermined number of sub-time-series data, so that the data objects extracted by the time-series feature computation module from the predetermined number of sub-time-series data can be allocated to multiple computation nodes.

[0022] Optionally, the time series feature calculation module sends the time series data and the definition information of the second feature calculation element, which consists of a grouping field or a combination of grouping fields and an algorithm name used to characterize the frequency calculation, to the summary feature calculation module.

[0023] Optionally, in response to receiving time-series data and second feature calculation element definition information, the summarizing feature calculation module obtains one or more feature extraction processing strategies consisting of algorithm names and different grouping fields or combinations of grouping fields. Based on the grouping fields or combinations of grouping fields in the feature extraction processing strategy, the summarizing feature calculation module groups the time-series data. Data records with the same field value corresponding to the field name indicated by the grouping field or combination of grouping fields are grouped into the same group. Data in different groups are allocated to one or more computing nodes in the distributed system. The computing nodes call the data processing function corresponding to the algorithm name to process the data allocated to them in order to obtain the frequency of each field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the time-series data.

[0024] Optionally, the optimization calculation module traverses each data record in the sub-time series data, and determines whether the sub-time series data contains all the data records required to process the data record based on the data record's processing result dependency. If the sub-time series data does not contain all the data required to process the data record, the missing data record is obtained and added to the sub-time series data.

[0025] Optionally, the optimization calculation module also marks missing data records as redundant data.

[0026] Optionally, the time series feature calculation module ignores the processing of redundant data, or the time series feature calculation module processes redundant data and removes the processing results of redundant data after obtaining the processing results of sub-time series data.

[0027] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields. If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is greater than or equal to a first predetermined threshold, the time series feature calculation module extracts the data object to be processed from the sub-time series data corresponding to the field value. And / or if the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is less than the first predetermined threshold, the time series feature calculation module extracts the data object to be processed from the data record corresponding to the field value in the time series data.

[0028] Optionally, the data objects extracted by the time series feature calculation module from a predetermined number of sub-time series data are allocated to multiple computing nodes. The computing nodes process the allocated data objects by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

[0029] Optionally, the data objects extracted by the time series feature calculation module from the data records corresponding to field values ​​with frequencies less than a first predetermined threshold are assigned to a single computing node. The computing node processes the assigned data objects by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

[0030] Optionally, the optimization calculation module is also used to merge the processing results of the data objects corresponding to each sub-time series data and the processing results of the data objects corresponding to the data records whose frequencies are less than the first predetermined threshold, so as to obtain the feature extraction results of the time series data.

[0031] Optionally, the optimization calculation module is used to add the feature extraction results of the time series data as new fields to the time series data.

[0032] Optionally, the optimization calculation module is also used to obtain the optimization calculation strategy provided by the user. When the optimization calculation strategy is the first scheme, the optimization calculation module sets the feature processing result of the data record corresponding to the field value whose frequency is greater than or equal to the first predetermined threshold as a predetermined value. When the optimization calculation strategy is the second scheme, the optimization calculation module splits the data record corresponding to each field value whose frequency is greater than or equal to the first predetermined threshold into a predetermined number of sub-time series data.

[0033] Optionally, the optimization calculation module splits the data records corresponding to each field value whose frequency is greater than or equal to a first predetermined threshold and less than a second predetermined threshold into a predetermined number of sub-time series data. The optimization calculation module also sets the processing result of the data records corresponding to each field value or combination of field values ​​whose frequency is greater than or equal to the second predetermined threshold as a predetermined value, wherein the second predetermined threshold is greater than the first predetermined threshold.

[0034] Optionally, the size of the first predetermined threshold is positively correlated with computing resources, and / or the number of predetermined quantities is positively correlated with computing resources.

[0035] According to a second aspect of the present invention, a data processing method for feature extraction is also proposed, comprising: setting up a base layer and a computation layer, the base layer including one or more data processing functions, and the computation layer including one or more feature computation modules; providing one or more feature computation modules to a user; the feature computation module selected by the user acquiring the data provided by the user and feature computation element definition information, the feature computation element definition information including at least one category of feature computation elements, each category of feature computation elements being used to indicate an element for feature extraction processing; the feature computation module selected by the user selecting one feature computation element from each category of feature computation elements to obtain a feature extraction processing strategy composed of feature computation elements from different categories; the feature computation module extracting the data object to be subjected to feature extraction processing from the data based on the feature extraction processing strategy, and determining the data processing function to perform feature extraction processing on the data object.

[0036] Optionally, the feature calculation module selects one feature calculation element from the feature calculation elements of each category in a Cartesian product manner to obtain one or more feature extraction processing strategies.

[0037] Optionally, the feature calculation element definition information includes multiple feature calculation element sets, each feature calculation element set includes at least one set of feature calculation elements, and the feature calculation elements in different sets correspond to different categories. For each feature calculation element set, the feature calculation module selects one feature calculation element from each set of feature calculation elements in a Cartesian product manner to obtain multiple feature extraction processing strategies.

[0038] Optionally, the method further includes: the feature calculation module selected by the user obtains the data processing function provided by the user and adds the data processing function to the base layer.

[0039] Optionally, the computation layer is divided into an implementation layer and an application layer. The implementation layer includes one or more first-type feature computation modules. The first-type feature computation modules are used to extract features corresponding to general business scenarios. The first-type feature computation modules obtain feature extraction processing strategies based solely on feature computation element definition information obtained from external sources. The application layer includes one or more second-type feature computation modules. The second-type feature computation modules are used to extract features corresponding to specific business scenarios. The second-type feature computation modules define at least some feature computation elements that constitute the feature extraction processing strategy. At least some feature computation elements are used to indicate one or more elements of the feature extraction processing corresponding to specific business scenarios.

[0040] Optionally, the first type of feature calculation module in the implementation layer can call the data processing function in the base layer, and / or the first type of feature calculation module can call other first type of feature calculation modules in the same layer; the second type of feature calculation module in the application layer can call the data processing function in the base layer, and / or the second type of feature calculation module can call other second type of feature calculation modules in the same layer, and / or the second type of feature calculation module can call the first type of feature calculation module in the implementation layer.

[0041] Optionally, the base layer may also include: a checking function for checking whether the feature calculation element definition information conforms to the specification; and / or a normalization function for converting the feature calculation element definition information into a data structure conforming to a predetermined form; and / or a constant library for providing one or more constants.

[0042] Optionally, the data includes multiple data records, each data record including one or more fields, and at least one category of feature calculation elements includes: a first category of feature calculation elements, including one or more target fields or combinations of target fields, the target fields or combinations of target fields being used to indicate the field names corresponding to the field values ​​for which feature extraction processing is performed; and / or a second category of feature calculation elements, including one or more windows or combinations of windows, the windows or combinations of windows being used to indicate the range corresponding to the field values ​​for which feature extraction processing is performed; and / or a third category of feature calculation elements, including one or more grouping fields or combinations of grouping fields, the grouping fields or combinations of grouping fields being used to indicate the field names on which the data is grouped; and / or a fourth category of feature calculation elements, including one or more algorithm names, the algorithm names being used to indicate the data processing logic for the field values ​​for which feature extraction processing is performed; and / or a fifth category of feature calculation elements, including one or more sorting fields, the sorting fields being used to indicate the field names on which the data is sorted.

[0043] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, and an algorithm name. The steps of the feature calculation module to extract the data object to be feature extracted from the data based on the feature extraction processing strategy and to determine the data processing function to perform feature extraction processing on the data object include: the feature calculation module groups the data according to the grouping field or the combination of grouping fields, and data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group; the feature calculation module selects the field value corresponding to the field name indicated by the target field or the combination of target fields from each group of data to obtain the data object to be feature extracted; the feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0044] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, a window or a combination of windows, a sorting field, and an algorithm name. The steps of the feature calculation module extracting the data object to be feature extracted from the data based on the feature extraction processing strategy and determining the data processing function to perform feature extraction processing on the data object include: the feature calculation module sorts the data according to the field name indicated by the sorting field; for each data record, the feature calculation module extracts the data record corresponding to the window range according to the range indicated by the window or window combination and the field name indicated by the sorting field to obtain first data including or excluding the data record; the feature calculation module selects the data record from the first data whose field value is the same as the field name indicated by the grouping field or grouping field combination in the data record to obtain second data including the data record; the feature calculation module selects the field value corresponding to the field name indicated by the target field or target field combination from the second data to obtain the data object to be feature extracted; the feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0045] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, a window or a combination of windows, a sorting field, and an algorithm name. The steps of the feature calculation module extracting the data object to be subject to feature extraction processing from the data based on the feature extraction processing strategy and determining the data processing function to perform feature extraction processing on the data object include: the feature calculation module sorts the data according to the field name indicated by the sorting field; for each data record, the feature calculation module selects data records from the data that have the same field value as the field name indicated by the grouping field or the combination of grouping fields in that data record, to obtain third data including that data record; the feature calculation module extracts data records corresponding to the window range from the third data according to the range indicated by the window or the combination of windows and the field name indicated by the sorting field, to obtain fourth data including or excluding that data record; the feature calculation module selects the field value corresponding to the field name indicated by the target field or the combination of target fields from the fourth data, to obtain the data object to be subject to feature extraction processing; the feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0046] Optionally, the method further includes: using one or more computing nodes in the distributed system to call the corresponding data processing function in the base layer to process the data object to obtain one or more features.

[0047] Optionally, the computation layer includes a time-series feature computation module and a summary feature computation module. The time-series feature computation module acquires time-series data and first feature computation element definition information. The time-series data includes multiple data records recorded in chronological order. Each data record includes one or more fields. The first feature computation element definition information includes one or more grouping fields or combinations of grouping fields. The grouping fields or combinations of grouping fields are used to indicate the field names on which the time-series data is grouped. The time-series feature computation module calls the summary feature computation module, which counts the frequency of each field value corresponding to the field name in the time-series data. The method further includes: splitting the data records corresponding to each field value whose frequency is greater than or equal to a first predetermined threshold into a predetermined number of sub-time-series data, so that the data objects extracted by the time-series feature computation module from the predetermined number of sub-time-series data can be allocated to multiple computation nodes.

[0048] Optionally, the time series feature calculation module sends the time series data and the definition information of the second feature calculation element, which consists of a grouping field or a combination of grouping fields and an algorithm name used to characterize the frequency calculation, to the summary feature calculation module.

[0049] Optionally, in response to receiving time-series data and second feature calculation element definition information, the summarizing feature calculation module obtains one or more feature extraction processing strategies consisting of algorithm names and different grouping fields or combinations of grouping fields. Based on the grouping fields or combinations of grouping fields in the feature extraction processing strategy, the summarizing feature calculation module groups the time-series data. Data records with the same field value corresponding to the field name indicated by the grouping field or combination of grouping fields are grouped into the same group. Data in different groups are allocated to one or more computing nodes in the distributed system. The computing nodes call the data processing function corresponding to the algorithm name to process the data allocated to them in order to obtain the frequency of each field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the time-series data.

[0050] Optionally, the data processing method further includes: traversing each data record in the sub-time series data, determining whether the sub-time series data contains all the data records required to process the data record based on the data record's processing result dependency on the data, and if the sub-time series data does not contain all the data required to process the data record, obtaining the missing data records and adding the obtained data records to the sub-time series data.

[0051] Optionally, the data processing method further includes marking missing data records as redundant data.

[0052] Optionally, the time series feature calculation module ignores the processing of redundant data, or the time series feature calculation module processes redundant data and removes the processing results of redundant data after obtaining the processing results of sub-time series data.

[0053] Optionally, the feature extraction processing strategy includes a grouping field or a combination of grouping fields. If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is greater than or equal to a first predetermined threshold, the time series feature calculation module extracts the data object to be processed from the sub-time series data corresponding to the field value. And / or if the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is less than the first predetermined threshold, the time series feature calculation module extracts the data object to be processed from the data record corresponding to the field value in the time series data.

[0054] Optionally, the data processing method further includes: allocating data objects extracted from a predetermined number of sub-time series data by the time series feature calculation module to multiple computing nodes, and the computing nodes processing the allocated data objects by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

[0055] Optionally, the data processing method further includes: allocating the data objects extracted by the time series feature calculation module from the data records corresponding to field values ​​with frequencies less than a first predetermined threshold to a single computing node, and the computing node processing the allocated data records by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

[0056] Optionally, the data processing method further includes: merging the processing results of the data objects corresponding to each sub-time series data and the processing results of the data objects corresponding to each field value or combination of field values ​​with a frequency less than a first predetermined threshold, so as to obtain the feature extraction results of the time series data.

[0057] Optionally, the data processing method further includes adding the feature extraction results of the time series data as new fields to the time series data.

[0058] Optionally, the data processing method further includes: obtaining the user-provided optimization calculation strategy; if the optimization calculation strategy is a first scheme, setting the feature processing result of the data record corresponding to the field value whose frequency is greater than or equal to a first predetermined threshold as a predetermined value; if the optimization calculation strategy is a second scheme, splitting the data record corresponding to each field value whose frequency is greater than or equal to the first predetermined threshold into a predetermined number of sub-time series data.

[0059] Optionally, the step of splitting the data records corresponding to each field value whose frequency is greater than or equal to a first predetermined threshold into a predetermined number of sub-time series data includes: splitting the data records corresponding to each field value whose frequency is greater than or equal to the first predetermined threshold and less than a second predetermined threshold into a predetermined number of sub-time series data. The method further includes: setting the processing result of the data records corresponding to each field value or combination of field values ​​whose frequency is greater than or equal to the second predetermined threshold as a predetermined value, wherein the second predetermined threshold is greater than the first predetermined threshold.

[0060] Optionally, the size of the first predetermined threshold is positively correlated with computing resources, and / or the number of predetermined quantities is positively correlated with computing resources.

[0061] According to a third aspect of the invention, a system comprising at least one computing device and at least one storage device for storing instructions is also provided, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform the method described in the second aspect of the invention.

[0062] According to a fourth aspect of the invention, a computer-readable storage medium for storing instructions is also provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform the method described in the second aspect of the invention.

[0063] According to exemplary embodiments of the present invention, the data processing method, system and storage medium for feature extraction can automatically combine feature calculation elements using a feature calculation module to generate corresponding feature extraction processing strategies. This can reduce manual operations and reduce the workload of users in writing feature calculation element definition information. In other words, users do not need to input repetitive feature calculation elements when writing feature calculation element definition information, thereby improving feature calculation efficiency. Attached Figure Description

[0064] These and / or other aspects and advantages of the present invention will become clearer and more readily understood from the following detailed description of embodiments of the invention taken in conjunction with the accompanying drawings, wherein:

[0065] Figure 1 A structural block diagram of a data processing system for feature extraction according to an exemplary embodiment of the present invention is shown;

[0066] Figure 2 A structural block diagram of a data processing system for feature extraction according to another exemplary embodiment of the present invention is shown;

[0067] Figure 3 A structural block diagram of a data processing system for feature extraction according to another exemplary embodiment of the present invention is shown;

[0068] Figure 4 A schematic flowchart of a data processing method for extracting features according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0069] To enable those skilled in the art to better understand the present invention, exemplary embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] Figure 1 A structural block diagram of a data processing system for feature extraction according to an exemplary embodiment of the present invention is shown.

[0071] The feature extraction described in this invention refers to processing data to extract features that can represent specific meanings. This invention can be applied to data processing in specific business scenarios to obtain features relevant to those scenarios. For example, this invention can be applied to data processing in the field of risk monitoring, such as processing account transaction data to obtain features that reflect the level of account risk. Furthermore, this invention can also be applied to data processing in other business areas, such as processing users' shopping records on shopping platforms to obtain features that reflect users' shopping interests or spending levels.

[0072] See Figure 1The data processing system 100 includes a basic layer 110 and a computing layer 120.

[0073] The base layer 110 includes one or more data processing functions. Each data processing function corresponds to a data processing logic. A data processing function is a function used to implement its corresponding data processing logic. Data processing functions may include, but are not limited to, aggregation functions such as summation, maximization, minimization, averaging, and variance calculation, and bucketing functions that transform continuous values ​​into discrete values.

[0074] Optionally, the base layer 110 may also include other types of function functions, such as, but not limited to, one or more of the following: check functions, normalization functions, and a constant library. Check functions are used to check whether the feature calculation element definition information conforms to the specification; normalization functions are used to convert the feature calculation element definition information into a data structure conforming to a predetermined form; and the constant library is used to provide one or more constants. The feature calculation element definition information will be described in detail below and will not be repeated here.

[0075] The computation layer 120 includes one or more feature computation modules. These feature computation modules are used to perform feature computation on the data. In this invention, each feature computation module can be configured to perform computation on a predetermined type of feature, and different feature computation modules can be configured to perform computation on different types of features; that is, the execution logic of each feature computation module can be pre-defined.

[0076] The feature calculation module in computation layer 120 can call the base layer 110, such as calling the data processing functions in base layer 110, as well as the checking functions, standardization functions, and other functionalities in base layer 110. Furthermore, different feature calculation modules in computation layer 120 can also call each other.

[0077] The feature calculation module in the computing layer 120 can be called by users or other feature calculation modules. The called feature calculation module can obtain the data and feature calculation element definition information passed in by the caller, and perform feature calculation (i.e., feature extraction) on the data based on the feature calculation element definition information. Here, "user" refers to the party that wants to extract features from the data, which can be, but is not limited to, developers and business personnel.

[0078] The data passed in by the caller can be online-generated data, pre-generated and stored data, or data received from external sources through input devices or transmission media. This data may involve attribute information of individuals, businesses, or organizations, such as identity, education, occupation, assets, contact information, liabilities, income, profits, and tax payments. Alternatively, this data may also involve attribute information related to business projects, such as the transaction amount, parties involved, subject matter, and transaction location of a sales contract. It should be noted that the attribute information mentioned in the exemplary embodiments of this invention can relate to the performance or nature of any object or matter in a certain aspect, and is not limited to limiting or describing individuals, objects, organizations, units, institutions, projects, events, etc.

[0079] The feature calculation module can acquire structured or unstructured data from various sources, such as text data or numerical data. This data can originate from within the entity from which feature extraction results are desired, such as banks, businesses, or schools; it can also originate from outside these entities, such as data providers, the internet (e.g., social networking sites), mobile operators, app operators, courier companies, and credit agencies. Optionally, the aforementioned internal and external data can be combined to form data carrying more information.

[0080] Preferably, the data acquired by the feature calculation module can be structured data. For example, the data acquired by the feature calculation module may include multiple data records, each of which includes one or more fields.

[0081] The feature calculation element definition information includes at least one category of feature calculation elements, each category of feature calculation elements used to indicate one element in the feature extraction process. Generally, feature extraction process includes data objects and processing logic for the data objects; therefore, feature calculation elements can be elements used to indicate the data objects to which feature extraction processing is performed, or elements used to indicate the processing logic for the data objects.

[0082] Taking a dataset consisting of multiple data records, each containing one or more fields, as an example, the feature calculation element definition information may include, but is not limited to, one or more of the various categories of feature calculation elements mentioned below. For ease of distinction, these various categories of feature calculation elements may be referred to as Category 1 Feature Calculation Elements, Category 2 Feature Calculation Elements, Category 3 Feature Calculation Elements, Category 4 Feature Calculation Elements, and Category 5 Feature Calculation Elements, respectively.

[0083] The first type of feature calculation element includes one or more target fields or a combination of target fields. The target field or combination of target fields is used to indicate the field name corresponding to the field value to which feature extraction processing is performed. The field value to which feature extraction processing is performed is also the data object to which feature extraction processing is performed in the data.

[0084] The second type of feature calculation elements includes one or more windows or window combinations, which indicate the range corresponding to the field values ​​being processed for feature extraction. The range indicated by the window or window combination can be a time range indicated based on time units such as seconds, minutes, hours, days of the week, and months, or it can be a range indicated in other ways. For example, a window or window combination can be a time range such as "the past week" or "the past three months", or it can be a dimension range of "the previous N rows of data records to the current row of data records", where N is an integer greater than or equal to 1.

[0085] The third type of feature calculation element includes one or more grouping fields or combinations of grouping fields, which are used to indicate the field names on which the data is grouped.

[0086] The fourth type of feature calculation element includes one or more algorithm names or combinations of algorithm names, which are used to indicate the data processing logic of the field values ​​to be subjected to feature extraction processing.

[0087] The fifth category of feature calculation elements includes one or more sorting fields. These sorting fields indicate the field names on which the data is sorted, including but not limited to numerical fields with ordering significance such as time and height, as well as non-numerical fields with ordering significance (ordered categorical), such as clothing sizes (01, 02, etc.). It should be noted that, given a window or combination of windows, the sorting fields are required to filter the data indicated by the window or combination of windows, and / or to assign meaning and context to the window.

[0088] For example, if the sort field is year and the window is [-2, +2], then when calculating the feature results for each row of data, the data corresponding to the past two years to the next two years relative to the year value of that row will be filtered out. The sort field gives the window [-2, +2] meaning and context (i.e., the past two years to the next two years), and only through the sort field can data within the specified window range be filtered. As another example, if the sort field is height (cm) and the window is [-20, 0], then when calculating the feature results for each row of data, data corresponding to height values ​​less than or equal to the height value of that row but greater than or equal to 20 cm shorter than the height value of that row will be filtered out. Similarly, the sort field gives meaning to the window and is used to filter data within the specified window range.

[0089] For example, a window or window combination can contain meaning itself, such as [-10d,0] representing the past 10 days to the present. In this case, the sort field is required to represent the Unix timestamp (https: / / baike.baidu.com / item / unixtimestamp / 2078227), and the window does not need the sort field to give meaning and context, but the sort field is still needed to indicate the time corresponding to each row of data and to filter the data within the window range.

[0090] The data processing system 100 may have a predetermined interface specification. This specification defines the form and content that the feature calculation element definition information must meet. When calling the feature calculation module, the caller can upload feature calculation element definition information that conforms to the interface specification. For example, the interface specification can define the data structure that the feature calculation element definition information must conform to in form, and can specify the value range that must conform to in content. When generating feature calculation element definition information, the caller (such as a user) can generate feature calculation element definition information that conforms to the specification, so that the feature calculation module can recognize it. For example, the data structure of the feature calculation element definition information specified by the interface specification can be one or more sets of key-value pairs, where the key can be used to indicate the category of the feature calculation element, and the value can refer to one or more feature calculation elements under the category corresponding to the key. Optionally, the interface specification can also specify that the name of the data processing function in the feature calculation element definition information must conform to the name of the data processing function defined and implemented by the base layer 110.

[0091] As an example, the feature calculation module can also receive user-provided data processing functions and add them to the base layer 110. That is, users can add custom functions by calling the feature calculation module to extend the base layer 110. For instance, users can define custom functions according to the syntax specifications for data processing functions defined in the base layer 110, and then pass these custom functions to the feature calculation module, thereby registering the custom functions in the base layer 110.

[0092] The following is an exemplary description of the feature calculation process implemented by the feature calculation module.

[0093] After obtaining the data and feature calculation element definition information, the feature calculation module can select one feature calculation element from the feature calculation elements of each category to obtain a feature extraction processing strategy composed of feature calculation elements from different categories.

[0094] The process by which the feature calculation module generates a feature extraction strategy is essentially the process of combining the feature calculation elements in the feature calculation element definition information. By automatically combining feature calculation elements and generating corresponding feature extraction strategies using the feature calculation module, manual operations can be reduced, and the workload for users in writing feature calculation element definition information can be decreased. In other words, users do not need to input repetitive feature calculation elements when writing feature calculation element definition information.

[0095] For example, suppose we need to calculate the features: "Minimum, maximum, average, median, and standard deviation of monthly user income over the past year" and "Minimum, maximum, average, median, and standard deviation of monthly merchant income over the past year". Users can provide the feature calculation element definition information in key-value pairs as follows: {'agg_func':['max','min','avg','median','stdev.s'], 'window':[(-12,0)], 'target_col':['user_income', 'merchant_income'], 'sort_col':['month']}. Here, 'agg_func':['max','min','avg','median','stdev.s'] corresponds to the fourth type of feature calculation element mentioned above. 'agg_func' indicates the type of feature calculation element, i.e., the data processing function. 'max', 'min', 'avg', 'median', and 'stdev.s' represent the values ​​of the feature calculation element under this type, i.e., the names of the data processing functions. 'max' represents the function for finding the maximum, 'min' represents the function for finding the minimum, 'avg' represents the function for finding the average, and 'median' represents... The median function is 'stdev.s', which represents the standard deviation function. 'window':[(-12,0)] corresponds to the second type of feature calculation element mentioned above. 'window' represents the category of the feature calculation element, i.e., the window. (-12,0) represents the value of the feature calculation element under this category, which is a numerical range in this embodiment. 'sort_col':['month'] corresponds to the fifth type of feature calculation element mentioned above. 'sort_col' represents the type of feature calculation element, i.e., the sorting field. 'month' represents the value of the feature calculation element under this category, i.e., the month. (-12,0) and 'month' constitute the complete time window, i.e., the past 12 months; 'target_col':['user_income', 'merchant_income'] corresponds to the first type of feature calculation element mentioned above. 'target_col' represents the category of the feature calculation element, i.e., the target field, and 'user_income' and 'merchant_income' represent the values ​​of the feature calculation elements under this category, i.e., the field names of the target fields, where 'user_income' is the user's monthly income field and 'merchant_income' is the merchant's monthly income field. It can be seen that when multiple features need to be calculated, for repeated feature calculation elements, the user only needs to input them once, which can greatly reduce the user's workload and reduce the possibility of errors.

[0096] After obtaining the feature calculation element definition information, the feature calculation module can select one feature calculation element from the feature calculation elements of each category using a Cartesian product to obtain one or more feature extraction processing strategies. That is, the feature calculation module can traverse the feature calculation elements of each category in a "global Cartesian product" manner to obtain multiple feature extraction processing strategies composed of feature calculation elements from different categories.

[0097] In addition, the feature calculation module can also combine feature calculation elements in the form of "local Cartesian product" to obtain one or more feature extraction processing strategies.

[0098] As an example, the feature calculation element definition information may include multiple feature calculation element sets, each feature calculation element set including at least one set of feature calculation elements, and the feature calculation elements in different sets correspond to different categories. For each feature calculation element set, the feature calculation module can select one feature calculation element from each set of feature calculation elements in a Cartesian product manner to obtain multiple feature extraction processing strategies.

[0099] For example, suppose the feature calculation element definition information is [{'agg_func':['max','min'],'target_col':['income']},{'agg_func':['sum','cnt'],'target_col':['trxamt','payment']}], where the part within each {} can be regarded as a set of feature calculation elements. 'agg_func' represents the data processing function, 'max', 'min', 'sum', and 'cnt' represent the names of the data processing functions. 'max' represents the function to find the maximum, 'min' represents the function to find the minimum, 'sum' represents the summation function, 'cnt' represents the counting function, 'target_col' represents the target field, and 'income', 'trxamt', and 'payment' represent the target fields with three different field names. The feature calculation module performs a Cartesian product combination on the first set of feature calculation elements to obtain two feature extraction strategies: "max for the income field" and "min for the income field". It also performs a Cartesian product combination on the second set of feature calculation elements to obtain four feature extraction strategies: "sum for the trxamt field", "sum for the payment field", "cnt for the trxamt field", and "cnt for the payment field". However, the feature calculation module does not select feature calculation elements from different feature calculation element sets for combination.

[0100] In this invention, the feature calculation module can determine whether to use a global Cartesian product or a local Cartesian product to combine feature calculation elements based on the form of the feature calculation element definition information. For example, if the feature calculation element definition information includes multiple feature calculation element sets, the feature calculation module can use the aforementioned local Cartesian product method to combine feature calculation elements for each feature calculation element set; otherwise, it can use a global Cartesian product method. Furthermore, the feature calculation module can also determine whether to use a global Cartesian product or a local Cartesian product method based on other specified methods adopted by the user; these will not be elaborated further in this invention.

[0101] After combining the feature calculation elements in the feature calculation element definition information to obtain the feature extraction processing strategy, the feature calculation module can extract the data objects to be processed by feature extraction from the data based on the feature extraction processing strategy, and determine the data processing function to perform feature extraction processing on the data objects.

[0102] Taking the feature extraction processing strategy obtained by the feature calculation module through combination, which includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, and an algorithm name, as an example, for each feature extraction processing strategy, the feature calculation module can group the data according to the grouping field or the combination of grouping fields. Data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group. After grouping, the feature calculation module can select the field value corresponding to the field name indicated by the target field or the combination of target fields from each group of data to obtain the data object for which feature extraction processing has been performed.

[0103] The feature calculation module can call the base layer 110 to determine the data processing function used to execute the data processing logic represented by the algorithm name. For example, the feature calculation module can compare the algorithm name in the feature extraction processing strategy with the name of the data processing function in the base layer 110, and use the data processing function in the base layer 110 that matches the algorithm name as the data processing function used to execute the data processing logic represented by the algorithm name.

[0104] Taking a feature extraction processing strategy that includes a grouping field or a combination of grouping fields, a target field or a combination of target fields, a window or a combination of windows, a sorting field, and an algorithm name as an example, the feature calculation module can first sort the data according to the field name indicated by the sorting field. The range indicated by the window or combination of windows and the field name indicated by the sorting field can constitute a complete window.

[0105] For each sorted data record, the feature calculation module can extract data records within the corresponding window range based on the range indicated by the window or window combination and the field name indicated by the sorting field, to obtain first data that includes or excludes that data record. The feature calculation module then selects data records from the first data that have the same field value as the field name indicated by the grouping field or grouping field combination in that data record, to obtain second data that includes that data record. Finally, the feature calculation module selects field values ​​from the second data that correspond to the field name indicated by the target field or target field combination, to obtain the data object to be processed by feature extraction. The feature calculation module can also call the base layer 110 to determine the data processing function used to execute the data processing logic represented by the algorithm name. For example, the feature calculation module can compare the algorithm name in the feature extraction processing strategy with the names of the data processing functions in the base layer 110, and use the data processing function in the base layer 110 that matches the algorithm name as the data processing function used to execute the data processing logic represented by the algorithm name.

[0106] Furthermore, for each sorted data record, the feature calculation module can first select data records from the data that have the same field value as the field name indicated by the grouping field or grouping field combination in that data record, to obtain third data including that data record. Then, the feature calculation module can extract data records corresponding to the window range from the third data according to the range indicated by the window or window combination and the field name indicated by the sorting field, to obtain fourth data including or excluding that data record. The feature calculation module selects field values ​​from the fourth data corresponding to the field name indicated by the target field or target field combination, to obtain the data object to be subjected to feature extraction processing. The feature calculation module can also determine the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer 110. For example, the feature calculation module can compare the algorithm name in the feature extraction processing strategy with the name of the data processing function in the base layer 110, and use the data processing function in the base layer 110 that matches the algorithm name as the data processing function used to execute the data processing logic represented by the algorithm name.

[0107] After the feature calculation module determines the data object to be processed for feature extraction and the data processing function to perform feature extraction on the data object based on the feature extraction processing strategy, one or more computing nodes in the distributed system can call the corresponding data processing function in the base layer to process the data object and obtain one or more features. In other words, the final feature calculation operation can be executed by the underlying computing nodes.

[0108] like Figure 2As shown, the data processing system 100 can be connected to a computing engine (e.g., the Spark engine), and under the action of the computing engine, computing tasks can be distributed to one or more computing nodes in the distributed system, and the computing nodes can perform feature calculations.

[0109] Specifically, the feature calculation module combines the feature calculation elements in the feature calculation element definition information to obtain a feature extraction processing strategy. Based on the feature extraction processing strategy, it determines the data object to be subject to feature extraction processing and the data processing function to perform feature extraction processing on the data object. The computing engine can then generate a corresponding computing task based on the processing result of the feature calculation module. This computing task can instruct the data object to be subject to feature extraction and the data processing function to perform feature extraction processing on the data object, and distribute the computing task to one or more computing nodes in the distributed system. The computing nodes can call the corresponding data processing function in the basic layer 110 to process the data object to obtain one or more features.

[0110] For example, the computing engine can package the data objects and data processing functions determined by the feature calculation module into multiple subtasks, each subtask corresponding to a portion of the data objects to be calculated. The computing engine can distribute these multiple subtasks to one or more computing nodes in the distributed system. The computing nodes can obtain the data objects assigned to them based on the received subtasks, and can perform feature calculations on the assigned data objects by calling the data processing functions in the base layer 110.

[0111] In summary, the data processing system 100 of the present invention can be regarded as "middleware" that assists in feature calculation. On the one hand, the data processing system 100 can provide users with an interface to call the feature calculation module. Users can call the appropriate feature calculation module according to the features they want to extract from the data, and upload the data and feature calculation element definition information. Thus, when a large number of features need to be calculated, users do not need to provide a set of feature calculation element definitions for each feature, but can provide feature calculation element definition information in a set manner. The feature calculation module will automatically perform Cartesian combination, thereby eliminating the need for users to repeatedly input the same feature calculation elements, reducing manual operation and the possibility of errors.

[0112] On the other hand, the data processing system 100 can connect to the computing engine. After the feature calculation module determines the data object to be feature extracted and the data processing function to perform feature extraction processing on the data object, the data object can be allocated to one or more computing nodes in the distributed system under the scheduling of the computing engine. The computing nodes then perform feature calculation on the data object allocated to them by calling the corresponding data processing function in the base layer.

[0113] Figure 3 A structural block diagram of a data processing system for feature extraction according to another exemplary embodiment of the present invention is shown.

[0114] like Figure 3 As shown, in this embodiment, the computing layer 120 can be further divided into an implementation layer 121 and an application layer 123.

[0115] The implementation layer 121 includes one or more first-type feature calculation modules. The first-type feature calculation modules are used to extract features corresponding to general business scenarios. The first-type feature calculation modules obtain feature extraction processing strategies based solely on feature calculation element definition information obtained from external sources.

[0116] Application layer 123 includes one or more second-type feature calculation modules. These second-type feature calculation modules are used to extract features corresponding to a specific business scenario. Each second-type feature calculation module defines at least some feature calculation elements constituting a feature extraction processing strategy. These at least some feature calculation elements are used to indicate one or more elements of the feature extraction processing corresponding to a specific business scenario.

[0117] The first type of feature calculation module in implementation layer 121 provides users with general feature calculation functions, while the second type of feature calculation module in application layer 123 provides users with feature calculation functions specific to business scenarios. Users can choose which feature calculation module to use based on their needs. When selecting the second type of feature calculation module, the values ​​of some feature calculation elements are already defined by the second type of feature calculation module. Therefore, users only need to provide feature calculation element definition information for the parts not defined by the second type of feature calculation module. Taking the second type of feature calculation module as an example, which is used to provide clustered feature calculations (such as the number of associated transactions per device) for risk monitoring business, the data processing logic in the clustered feature calculation module can be defined as a counting function (i.e., the count function). In this way, when users call the clustered feature calculation module, the uploaded feature calculation element definition information does not need to include the algorithm name used to indicate the data processing logic.

[0118] The first type of feature calculation module in implementation layer 121 can call the base layer 110, such as calling the data processing functions in the base layer 110, as well as the checking functions, standardization functions, and other functional functions in the base layer 110. Furthermore, different first type of feature calculation modules in implementation layer 121 can also call each other; that is, a first type of feature calculation module can also call other first type of feature calculation modules in the same layer.

[0119] For example, the first type of feature calculation module in implementation layer 121 may include a summary feature calculation module and a time-series feature calculation module. The summary feature calculation module can be configured to handle calculations of summary features, such as, but not limited to, summation, averaging, standard deviation, and count values. The time-series feature calculation module can be configured to handle calculations of time-series features. For example, the time-series feature calculation module can be configured to process time-series data to obtain time-related features. Time-series data can consist of multiple data records recorded in chronological order, each data record including one or more fields. During operation, the time-series feature calculation module can call the summary feature calculation module to count the existence of hot data. If hot data is determined to exist, it can also perform hotspot removal operations. The workflows of the summary feature calculation module and the time-series feature calculation module will be described in detail below and will not be repeated here.

[0120] The second-type feature calculation module in application layer 123 can also call the base layer, such as the data processing functions in base layer 110, as well as the checking functions, standardization functions, and other functional functions in base layer 110. Furthermore, different second-type feature calculation modules in application layer 123 can also call each other; that is, a second-type feature calculation module can also call other second-type feature calculation modules in the same layer. In addition, the second-type feature calculation module can also call the first-type feature calculation module in implementation layer 121. Taking the clustering feature calculation module as an example, after obtaining the data and feature calculation element definition information, the clustering feature calculation module can call the summary feature calculation module in implementation layer 121. For example, the clustering feature calculation module can combine its defined counting function with the feature calculation element definition information received from the outside to form new feature calculation element definition information, and then send the data and the new feature calculation element definition information to the summary feature calculation module for processing.

[0121] In summary, the base layer, implementation layer, and application layer in this invention are designed in descending order of generality. The application layer, with the lowest generality, can call the base layer or implementation layer, which has higher generality, and the implementation layer, with the next lowest generality, can also call the base layer, which has higher generality. This allows for efficient reuse of lower-level code, avoiding the need to reinvent the wheel, as in the process of building a car. This enables the rapid construction of the data processing system 100 and allows developers to focus more on business exploration, such as designing more types of feature calculation modules.

[0122] The following examples illustrate the process of feature calculation in the summary feature calculation module and the time series feature calculation module.

[0123] 1. Summary Feature Calculation Module

[0124] When a user selects to use the summary feature calculation module, the uploaded feature calculation element definition information may include the first type of feature calculation elements and the fourth type of feature calculation elements mentioned above, and optionally may also include the third type of feature calculation elements mentioned above.

[0125] Taking the feature calculation element definition information as including the first type of feature calculation element, the third type of feature calculation element, and the fourth type of feature calculation element as an example, the summary feature calculation module combines the feature calculation elements in the feature calculation element definition information to obtain feature extraction processing strategies including grouping fields or combinations of grouping fields, target fields or combinations of target fields, and algorithm names.

[0126] For each feature extraction processing strategy, the summary feature calculation module can group the data according to the grouping field or grouping field combination. Data records with the same field value corresponding to the field name indicated by the grouping field or grouping field combination are grouped into the same group. The module can select the field value corresponding to the field name indicated by the target field or target field combination from each group of data to obtain the data object to be processed by feature extraction. The summary feature calculation module can determine the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

[0127] For example, suppose the data obtained by the summary feature calculation module is the bank transaction details table shown below, which includes, but is not limited to, fields such as "monthly income amount", "monthly expenditure amount", "account number", and "region code".

[0128] 001 S01 5000 3200 002 S03 2500 1600 002 S01 4300 3000 001 S03 3200 2400

[0129] Assume that the feature calculation element definition information obtained by the summary feature calculation module includes: aggregation function and target field ((mean, sum, standard deviation), (monthly income amount, monthly expenditure amount)), and grouping field (account number, region number).

[0130] The summary feature calculation module can first call the check function and standardization function in the base layer 110 to check whether the feature calculation element definition information is complete and accurate, and standardize the feature calculation element definition information into the structure required for subsequent calculations. For example, the above feature calculation element definition information can be converted into: target field: (monthly income amount, monthly expenditure amount); aggregation function: (average, sum, standard deviation); grouping field: (account number, region number).

[0131] The summary feature calculation module iterates through each defined value of the three types of feature calculation elements—target field, aggregation function, and grouping field—to obtain one or more feature extraction processing strategies.

[0132] For each feature extraction strategy, the aggregation feature calculation module extracts the data object from the data to be processed. Taking the feature extraction strategies "Account Number", "Monthly Income Amount", and "Average" as an example, the aggregation feature calculation module can group the input data according to the current value of the grouping field, with each group containing data of the same account number, i.e., each account corresponds to one group; then, based on the current value of the target field, "Monthly Income Amount", the target field value is selected for each group to obtain data D, which is the data object to be processed. By calling the average function of the base layer based on the current value of the aggregation function, the data processing function corresponding to data D can be determined.

[0133] In this way, by traversing each feature extraction and processing strategy, the feature calculation module can determine the data objects and their data processing functions corresponding to multiple features to be calculated.

[0134] After the feature calculation module determines the data object and its data processing function, one or more computing nodes in the distributed system can call the data processing function to process the data object to obtain one or more features.

[0135] Finally, through calculation, the following 12 features can be obtained: the average, sum, and standard deviation of monthly income for each account (3 features); the average, sum, and standard deviation of monthly expenditure for each account (3 features); the average, sum, and standard deviation of monthly income for each region (3 features); and the average, sum, and standard deviation of monthly expenditure for each region (3 features).

[0136] 2. Time Series Feature Calculation Module

[0137] When a user selects to use the time-series feature calculation module, the uploaded feature calculation element definition information may include the first type of feature calculation element, the second type of feature calculation element, and the fourth type of feature calculation element mentioned above, and optionally may also include the third type of feature calculation element and / or the fifth type of feature calculation element mentioned above.

[0138] Taking the feature calculation element definition information, which includes the first type of feature calculation element, the second type of feature calculation element, the third type of feature calculation element, the fourth type of feature calculation element, and the fifth type of feature calculation element, as an example, the time series feature calculation module combines the feature calculation elements in the feature calculation element definition information to obtain feature extraction processing strategies, including grouping fields or combinations of grouping fields, target fields or combinations of target fields, windows or combinations of windows, sorting fields, and algorithm names.

[0139] For example, suppose the data obtained by the time series feature calculation module is the bank transaction details table shown below, which includes, but is not limited to, fields such as "monthly income amount", "monthly expenditure amount", "account number", "region code" and "month".

[0140] 001 S01 5000 3200 1 002 S03 2500 1600 1 002 S01 4300 3000 2 001 S03 3200 2400 2

[0141] Suppose the user-provided feature calculation element definition information is as follows: Aggregate function and target field: ((average, sum), (monthly income, monthly expenditure)); grouping field: (account number, region code); time window: ([-3,0], [-6,0]); sorting field: (month). The time window defines two windows, each consisting of left and right intervals. The sorting field is used to sort the data and, together with the time window feature calculation element, defines the window range. For example, combining the sorting field and the first window, the time window is defined as the past three months (including the current month), and the second window represents the past six months (including the current month).

[0142] The time-series feature calculation module can first call the checking and standardization functions in the base layer 110 to check whether the feature calculation element definition information is complete and accurate, and standardize the feature calculation element definition information into the structure required for subsequent calculations. For example, the above feature calculation element definition information can be converted into: target field: (monthly income amount, monthly expenditure amount); aggregation function: (average, sum); grouping field: (account number, region number); time window: ([-3,0], [-6,0]); sorting field: (month).

[0143] The time-series feature calculation module iterates through each defined value of the four types of feature calculation elements—target field, aggregation function, grouping field, and time window—to obtain one or more feature extraction and processing strategies.

[0144] For each feature extraction strategy, the time-series feature calculation module extracts the data object to be processed from the data. Taking the feature extraction strategies "Account Number", "Monthly Income Amount", "Average", and "[-3,0]" as an example, the time-series feature calculation module extracts data within the corresponding window range (data from the past three months including the current row) for each row of input data based on the current value of the time window and the current value of the sorting field. It then groups the input data based on the current value of the grouping field, extracting only data with the same account number as the current row. Based on the current value of the target field (monthly income amount), it selects the target field value for the data in the group to obtain data D, which is the data object to be processed. Finally, by calling the average function of the base layer based on the current value of the aggregation function, the data processing function corresponding to data D can be determined.

[0145] Thus, by traversing each feature extraction and processing strategy, the time-series feature calculation module can determine the data objects and their data processing functions corresponding to multiple features to be calculated.

[0146] After the feature calculation module determines the data object and its data processing function, one or more computing nodes in the distributed system can call the data processing function to process the data object to obtain one or more features.

[0147] Finally, through calculation, a total of 16 features can be obtained: the average and sum of monthly income for each account over the past three months and the past six months (4 features); the average and sum of monthly expenditure for each account over the past three months and the past six months (4 features); the average and sum of monthly income for each region over the past three months and the past six months (4 features); and the average and sum of monthly expenditure for each region over the past three months and the past six months (4 features).

[0148] During operation, the time-series feature calculation module can also call the summary feature calculation module to count the presence of hot data. If hot data is identified, it can perform optimization calculations on the hot data to address potential data skew issues in time-series data processing. Data skew refers to a situation where a large amount of data is allocated to a single computing node for computation, causing the computation speed of this data to be much lower than the average computation speed, resulting in a slow overall computation process.

[0149] The following provides an example of a solution to the data skew problem in time series data processing.

[0150] The time-series feature calculation module can obtain time-series data and feature calculation element definition information (for ease of distinction, this can be referred to as the first feature calculation element definition information). The first feature calculation element definition information includes one or more grouping fields or combinations of grouping fields, which indicate the field names used to group the time-series data. Additionally, the first feature calculation element definition information may also include other categories of feature calculation elements, which will not be elaborated upon here.

[0151] In this embodiment, the time-series feature calculation module can still combine the feature calculation elements in the first feature calculation element definition information according to the processing mechanism of the feature calculation module described above, to obtain a feature extraction processing strategy. Based on the feature extraction processing strategy, the data object to be subjected to feature extraction processing is extracted from the time-series data, and the processing function of the data object is determined. The difference is that, with the help of the summary feature calculation module and the optimization calculation module mentioned below, the time-series feature calculation module can achieve optimized calculation for hot data to solve the data skew problem that may exist in the time-series data processing.

[0152] Specifically, the time-series feature calculation module can call the summary feature calculation module, which then counts the frequency of each field value in the time-series data corresponding to the field name indicated by the grouping field or combination of grouping fields. For example, the time-series feature calculation module can send the time-series data and the definition information of the second feature calculation element, which consists of the grouping field or combination of grouping fields and the algorithm name used to characterize the frequency calculation, to the summary feature calculation module, which then counts the frequencies.

[0153] In response to receiving time-series data and second feature calculation element definition information, the summary feature calculation module can combine the feature calculation elements in the second feature calculation element definition information to obtain one or more feature extraction processing strategies consisting of algorithm names for representing frequency calculation and different grouping fields or combinations of grouping fields. Based on the grouping fields or combinations of grouping fields in the feature extraction processing strategy, the summary feature calculation module groups the time-series data. Data records with the same field value corresponding to the field name indicated by the grouping field or combination of grouping fields are divided into the same group. Data in different groups are allocated to one or more computing nodes in the distributed system. The computing nodes call the data processing function (such as a counting function) corresponding to the algorithm name to process the data allocated to them to obtain the frequency of each field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the time-series data.

[0154] The data processing system 100 also includes an optimization calculation module. This module compares each frequency with a first predetermined threshold. If the frequency is less than the first predetermined threshold, it indicates that the data record corresponding to that frequency is non-hotspot data and will not cause data skew. If the frequency is greater than or equal to the first predetermined threshold, it indicates that the data record corresponding to that frequency is hotspot data and will cause data skew. The first predetermined threshold can be determined based on the size of computing resources, where computing resources can refer to resources such as CPU and / or memory of a single computing node. The size of the first predetermined threshold is positively correlated with computing resources; that is, the more computing resources, the larger the first predetermined threshold, and vice versa.

[0155] For data records corresponding to field values ​​or combinations of field values ​​whose frequency is less than a first predetermined threshold, they can be processed according to the normal processing flow. For example, if the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is less than the first predetermined threshold, the time series feature calculation module can extract the data object to be processed from the data record corresponding to the field value in the time series data. Since this data object will not produce data skew, it can be assigned to a single computing node for processing.

[0156] Data records corresponding to field values ​​or combinations of field values ​​with a frequency greater than or equal to a first predetermined threshold are considered hot data that can cause data skew. In this embodiment, the optimization calculation module can provide various optimization calculation schemes for hot data to solve the data skew problem caused by hot data in time series data.

[0157] For example, for hot data, the optimization calculation module can split the data records corresponding to field values ​​whose frequencies are greater than or equal to a first predetermined threshold into a predetermined number of sub-time series data. When the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is greater than or equal to the first predetermined threshold, the time series feature calculation module can extract the data object to be processed from the sub-time series data corresponding to the field value. If the data object extracted by the time series feature calculation module from the predetermined number of sub-time series data is assigned to a single computing node, it may cause data skew. Therefore, the data object can be assigned to multiple computing nodes for processing.

[0158] In addition, the optimization calculation module can also directly set the processing results of the data records corresponding to the field values ​​whose frequencies are greater than or equal to the first predetermined threshold as predetermined values ​​to improve calculation efficiency.

[0159] Optionally, the optimization calculation strategy for hot data can be determined by the user. That is, the optimization calculation module can also obtain the optimization calculation strategy provided by the user. When the optimization calculation strategy is the first scheme, the optimization calculation module sets the feature processing result of the data record corresponding to the field value with a frequency greater than or equal to the first predetermined threshold as a predetermined value. When the optimization calculation strategy is the second scheme, the optimization calculation module splits the data record corresponding to each field value with a frequency greater than or equal to the first predetermined threshold into a predetermined number of sub-time series data.

[0160] In addition, the optimization calculation module can compare the frequency with a first predetermined threshold and a second predetermined threshold, where the second predetermined threshold is greater than the first predetermined threshold. For data records corresponding to field values ​​or combinations of field values ​​that are greater than or equal to the first predetermined threshold and less than the second predetermined threshold, the optimization calculation module can perform the calculation by splitting the data. For data records corresponding to field values ​​or combinations of field values ​​whose frequency is greater than the second predetermined threshold, the optimization calculation module can directly set the processing result as a predetermined value.

[0161] In summary, this invention does not pre-determine the number of hot data points, but instead counts the frequency of each field value corresponding to a predetermined field name or combination of predetermined field names. By comparing the frequency with a predetermined threshold, it identifies potential hot data points in the time-series data. Furthermore, this invention splits the hot data into multiple sub-time-series data points according to chronological order, rather than randomly grouping them. This ensures that the chronological order of data records within the split sub-time-series data is not disrupted, thus enabling the split sub-time-series data to support the original data processing logic. Further, the predetermined field names or combinations of predetermined field names mentioned in this invention represent grouping fields in the time-series data processing process, which can be determined based on the time-series data processing logic. Therefore, this invention fully considers the data processing logic of time-series data in practical applications, providing a data skew solution adapted to the data processing logic of time-series data to solve the data skew problem in the time-series data processing process.

[0162] Furthermore, considering that even after splitting hot data into a predetermined number of sub-time series data according to chronological order, a single sub-time series data may still contain missing data records that the calculation depends on. Therefore, this invention proposes that the optimization calculation module can add missing data records needed for processing the data records but not in the sub-time series data, based on the data dependency relationship of the processing results of the data records in the sub-time series data. For example, the optimization calculation module can traverse each data record in the sub-time series data, determine whether the sub-time series data contains all the data records needed to process that data record based on the data dependency relationship of the processing results of that data record, and if it determines that the sub-time series data does not contain all the data required to process that data record, obtain the missing data records and add them to the sub-time series data. In this way, all the data required by the sub-time series data during the processing can be completed.

[0163] In the process of adding missing data records to sub-time series data, this invention can also mark the missing data records as redundant data. The time series feature calculation module can either ignore or process redundant data during processing. Redundant data are added data records used to provide data support for the calculation of other data records. The data upon which the processing result of the redundant data itself depends may not all be in the sub-time series data, and adding missing data to redundant data is unnecessary. Therefore, when the time series feature calculation module processes redundant data, if some data records from all the data records required to process the redundant data are missing in the sub-time series data, the time series feature calculation module can use a preset value (such as a null value) to replace the missing data. In the case of processing redundant data, after obtaining the processing result of the sub-time series data, the processing result of the redundant data can also be removed.

[0164] This invention can also merge the processing results of each sub-time series data and the processing results of data records corresponding to field values ​​or combinations of field values ​​with frequencies less than a first predetermined threshold to obtain the processing result of the time series data. Where it is necessary to return the original field, the optimization calculation module can add the feature extraction result of the time series data as a new field to the time series data.

[0165] In this invention, other types of feature calculation modules can also optimize calculations in response to data skew problems by referring to the above method. The specific implementation process will not be elaborated in this invention.

[0166] Figure 4 A schematic flowchart of a data processing method for extracting features according to an exemplary embodiment of the present invention is shown.

[0167] See Figure 4 In step S110, the base layer and the computation layer are set. For details regarding the base layer and the computation layer, please refer to the relevant descriptions above; they will not be repeated here.

[0168] In step S120, one or more feature calculation modules are provided to the user.

[0169] This section provides users with APIs for calling feature calculation modules in the computation layer, allowing them to invoke appropriate modules. Users can call these modules via code or visual methods (such as clicking or dragging). For example, feature calculation modules can be visually presented to users, showcasing the modules provided by the computation layer and the feature types they are suitable for processing. This allows users to make informed choices based on a thorough understanding of the module's functionality.

[0170] In step S130, the feature calculation module selected by the user obtains the data and feature calculation element definition information provided by the user.

[0171] Here, users can provide data and feature calculation element definition information to the feature calculation module via code, or they can use other methods that can carry data information, such as configuration files, Excel files, text files, and web pages. For details regarding data and feature calculation element definition information, please refer to the relevant descriptions above; they will not be repeated here.

[0172] In step S140, the feature calculation module selected by the user selects one feature calculation element from the feature calculation elements of each category to obtain a feature extraction processing strategy composed of feature calculation elements from different categories. Based on the feature extraction processing strategy, the feature calculation module extracts the data object to be subjected to feature extraction processing from the data and determines the data processing function to perform feature extraction processing on the data object. The specific processing procedure of the feature calculation module can be found in the relevant description above, and will not be repeated here.

[0173] It should be understood that the specific implementation of the data processing method for feature extraction according to the exemplary embodiments of the present invention can be referred to in conjunction with the above description. Figures 1 to 3 The specific implementation methods described will not be elaborated here.

[0174] Reference above Figures 1 to 4 A data processing method and system for extracting features according to exemplary embodiments of the present invention are described. It should be understood that the above method can be implemented by a program recorded on a computer-readable medium. For example, according to exemplary embodiments of the present invention, a computer-readable storage medium storing instructions can be provided, wherein instructions for execution are recorded on the computer-readable medium. Figure 4 A computer program is shown for a data processing method used to extract features.

[0175] The computer program in the aforementioned computer-readable medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the computer program can be used to perform actions other than... Figure 4 In addition to the steps shown, additional steps may be performed besides those described above, or more specific processing may be performed while executing the steps described above. The details of these additional steps and further processing have been referenced. Figure 4 The above has already been described, and will not be repeated here to avoid repetition.

[0176] It should be noted that the data processing system for feature extraction according to an exemplary embodiment of the present invention can rely entirely on the operation of a computer program to realize the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to realize the corresponding functions.

[0177] on the other hand, Figures 1 to 3The various functional modules shown can also be implemented using hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0178] For example, an exemplary embodiment of the present invention can also be implemented as a computing device, which includes a storage component and a processor, wherein the storage component stores a set of computer-executable instructions, and when the set of computer-executable instructions is executed by the processor, a data processing method for extracting features is performed.

[0179] Specifically, the computing device can be deployed on a server or client, or on a node device in a distributed network environment. Furthermore, the computing device can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions.

[0180] Here, the computing device is not necessarily a single computing device, but can be any collection of devices or circuits capable of executing the above instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0181] In the computing device, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0182] Some operations described in the data processing method for feature extraction according to an exemplary embodiment of the present invention can be implemented in software, some operations can be implemented in hardware, and some operations can be implemented in a combination of software and hardware.

[0183] The processor can execute instructions or code stored in one of its storage components, which can also store data. The instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0184] Storage components can be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, storage components can include separate devices, such as external disk drives, storage arrays, or other storage devices that can be used by any database system. Storage components and the processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the storage component.

[0185] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be interconnected via a bus and / or network.

[0186] The operations involved in the data processing method for feature extraction according to an exemplary embodiment of the present invention can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logic device or operate according to non-precise boundaries.

[0187] For example, as described above, a data processing system for extracting features according to an exemplary embodiment of the present invention may include a storage component and a processor, wherein the storage component stores a set of computer-executable instructions, and when the set of computer-executable instructions is executed by the processor, the data processing method for extracting features described above is performed.

[0188] The foregoing has described various exemplary embodiments of the present invention. It should be understood that the above description is merely exemplary and not exhaustive, and the present invention is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing system for extracting features, comprising: The base layer includes one or more data processing functions; The computation layer includes one or more feature computation modules. Different feature computation modules perform computations on different types of features. Each feature computation module acquires data and feature computation element definition information. The feature computation element definition information includes at least one category of feature computation elements. Each category of feature computation elements is used to indicate an element for feature extraction processing. Each feature computation module selects one feature computation element from each category of feature computation elements to obtain a feature extraction processing strategy composed of feature computation elements from different categories. Based on the feature extraction processing strategy, the feature computation module extracts the data object to be subjected to feature extraction processing from the data and determines the data processing function to perform feature extraction processing on the data object. in, The feature calculation module selects a feature calculation element from the feature calculation elements of each category in the form of a Cartesian product to obtain one or more of the feature extraction processing strategies; the feature calculation module determines the Cartesian product method according to the form of the feature element definition information, wherein the Cartesian product method includes global Cartesian product and local Cartesian product.

2. The data processing system according to claim 1, wherein, The feature calculation element definition information includes multiple feature calculation element sets. Each feature calculation element set includes at least one set of feature calculation elements. The feature calculation elements in different sets correspond to different categories. For each feature calculation element set, the feature calculation module selects one feature calculation element from each set of feature calculation elements in a Cartesian product manner to obtain multiple feature extraction processing strategies.

3. The data processing system according to claim 1, wherein, The feature calculation module also receives data processing functions provided by the user and adds these functions to the base layer.

4. The data processing system according to claim 1, wherein, The computing layer is divided into an implementation layer and an application layer. The implementation layer includes one or more first-type feature calculation modules. These first-type feature calculation modules are used to extract features corresponding to general business scenarios. The first-type feature calculation modules obtain the feature extraction processing strategy based solely on feature calculation element definition information obtained from external sources. The application layer includes one or more second-type feature calculation modules. The second-type feature calculation modules are used to extract features corresponding to a specific business scenario. The second-type feature calculation modules define at least some feature calculation elements that constitute the feature extraction processing strategy. The at least some feature calculation elements are used to indicate one or more elements of the feature extraction processing corresponding to the specific business scenario.

5. The data processing system according to claim 4, wherein, The first type of feature calculation module in the implementation layer can call the data processing function in the base layer, and / or the first type of feature calculation module can call other first type of feature calculation modules in the same layer. The second type of feature calculation module in the application layer can call the data processing function in the base layer, and / or the second type of feature calculation module can call other second type of feature calculation modules in the same layer, and / or the second type of feature calculation module can call the first type of feature calculation module in the implementation layer.

6. The data processing system according to claim 1, wherein, The base layer also includes: A check function is used to check whether the feature calculation element definition information conforms to the specification; and / or A standardization function is used to convert the feature calculation element definition information into a data structure conforming to a predetermined form; and / or A constant library for providing one or more constants.

7. The data processing system according to claim 1, wherein, The data includes multiple data records, each data record includes one or more fields, and the feature calculation elements of the at least one category include: The first type of feature calculation element includes one or more target fields or combinations of target fields, wherein the target fields or combinations of target fields are used to indicate the field names corresponding to the field values ​​to which feature extraction processing is performed; and / or The second type of feature calculation element includes one or more windows or window combinations, which are used to indicate the range corresponding to the field values ​​to which feature extraction processing is performed; and / or The third type of feature calculation element includes one or more grouping fields or combinations of grouping fields, wherein the grouping fields or combinations of grouping fields are used to indicate the field names on which the data is grouped; and / or The fourth category of feature calculation elements includes one or more algorithm names, which indicate the data processing logic for the field values ​​to be subjected to feature extraction processing; and / or The fifth type of feature calculation element includes one or more sorting fields, which are used to indicate the field names on which the data is sorted.

8. The data processing system according to claim 7, wherein, The feature extraction processing strategy includes the grouping field or combination of the grouping fields, the target field or combination of the target fields, and the algorithm name. The feature calculation module groups the data according to the grouping field or the combination of grouping fields. Data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group. The feature calculation module selects field values ​​from each set of data that correspond to the field names indicated by the target field or the combination of target fields, in order to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

9. The data processing system according to claim 7, wherein, The feature extraction processing strategy includes the grouping field or a combination of the grouping fields, the target field or a combination of target fields, the window or a combination of windows, the sorting field, and the algorithm name. The feature calculation module sorts the data according to the field name indicated by the sorting field. For each data record, the feature calculation module extracts the data records within the corresponding window range based on the range indicated by the window or window combination and the field name indicated by the sorting field, to obtain the first data. The feature calculation module selects data records from the first data that have the same field value as the field name indicated by the grouping field or the combination of the grouping fields, to obtain second data including that data record. The feature calculation module selects field values ​​from the second data that correspond to the field names indicated by the target field or the combination of target fields, to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

10. The data processing system according to claim 7, wherein, The feature extraction processing strategy includes the grouping field or a combination of the grouping fields, the target field or a combination of target fields, the window or a combination of windows, the sorting field, and the algorithm name. The feature calculation module sorts the data according to the field name indicated by the sorting field. For each data record, the feature calculation module selects data records from the data whose field values ​​are the same as the field names indicated by the grouping field or the combination of grouping fields, to obtain third data including that data record. The feature calculation module extracts data records corresponding to the window range from the third data based on the range indicated by the window or window combination and the field name indicated by the sorting field to obtain the fourth data. The feature calculation module selects field values ​​from the fourth data that correspond to the field names indicated by the target field or the combination of target fields, to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

11. The data processing system according to claim 1, wherein, One or more computing nodes in the distributed system call the corresponding data processing functions in the basic layer to process the data object and obtain one or more features.

12. The data processing system according to claim 1, wherein, The computation layer includes a time-series feature calculation module and a summary feature calculation module. The time-series feature calculation module acquires time-series data and first feature calculation element definition information. The time-series data includes multiple data records recorded in chronological order, and each data record includes one or more fields. The first feature calculation element definition information includes one or more grouping fields or combinations of grouping fields. The grouping fields or combinations of grouping fields are used to indicate the field names on which the time-series data is grouped. The time-series feature calculation module calls the summary feature calculation module, which then counts the frequency of the segment values ​​corresponding to the field names in the time-series data. The system also includes an optimization calculation module, which is used to split the data records corresponding to each field value with a frequency greater than or equal to a first predetermined threshold into a predetermined number of sub-time series data, so as to allocate the data objects extracted by the time series feature calculation module from the predetermined number of sub-time series data to multiple calculation nodes.

13. The data processing system according to claim 12, wherein, The time-series feature calculation module sends the time-series data and the definition information of the second feature calculation element, which consists of the grouping field or a combination of grouping fields and the algorithm name used to characterize frequency calculation, to the summary feature calculation module.

14. The data processing system according to claim 13, wherein, In response to receiving the time-series data and the second feature calculation element definition information, the summary feature calculation module obtains one or more feature extraction processing strategies composed of the algorithm name and different grouping fields or combinations of grouping fields. Based on the grouping fields or combinations of grouping fields in the feature extraction processing strategy, the summary feature calculation module groups the time-series data. Data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group. Data from different groups is allocated to one or more computing nodes in a distributed system. The computing nodes call a data processing function corresponding to the algorithm name to process the data allocated to them in order to obtain the frequency of each field value in the time series data corresponding to the field name indicated by the grouping field or the combination of the grouping fields.

15. The data processing system according to claim 12, wherein, The optimization calculation module traverses each data record in the sub-time series data. Based on the data dependency relationship of the processing result of the data record, it determines whether the sub-time series data contains all the data records required to process the data record. If the sub-time series data does not contain all the data required to process the data record, it obtains the missing data records and adds the obtained data records to the sub-time series data.

16. The data processing system according to claim 15, wherein, The optimization calculation module also marks the missing data records as redundant data.

17. The data processing system according to claim 16, wherein, The time-series feature calculation module ignores the processing of redundant data, or The time-series feature calculation module processes the redundant data and, after obtaining the processing result of the sub-time-series data, removes the processing result of the redundant data.

18. The data processing system according to claim 12, wherein, The feature extraction processing strategy includes the grouping field or a combination of grouping fields. If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is greater than or equal to the first predetermined threshold, the time series feature calculation module extracts the data object from the sub-time series data corresponding to that field value that has undergone feature extraction processing, and / or If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is less than the first predetermined threshold, the time series feature calculation module extracts the data object to be subjected to feature extraction processing from the data record corresponding to the field value in the time series data.

19. The data processing system according to claim 18, wherein, The data objects extracted from the predetermined number of sub-time series data by the time series feature calculation module are distributed to multiple computing nodes. The computing node processes the data object assigned to it by calling the corresponding data processing function in the base layer to obtain one or more time-series features.

20. The data processing system according to claim 19, wherein, The data objects extracted by the time series feature calculation module from the data records corresponding to field values ​​with frequencies less than a first predetermined threshold are assigned to a single computing node. The computing node processes the assigned data objects by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

21. The data processing system according to claim 20, wherein, The optimization calculation module is further used to merge the processing results of the data objects corresponding to each sub-time series data and the processing results of the data objects corresponding to the data records whose frequencies are less than the first predetermined threshold, so as to obtain the feature extraction results of the time series data.

22. The data processing system according to claim 21, wherein, The optimization calculation module is used to add the feature extraction results of the time series data as new fields to the time series data.

23. The data processing system according to claim 12, wherein, The optimization calculation module is also used to obtain optimization calculation strategies provided by the user. When the optimization calculation strategy is the first scheme, the optimization calculation module sets the feature processing result of the data record corresponding to the field value whose frequency is greater than or equal to the first predetermined threshold as the predetermined value. When the optimization calculation strategy is the second scheme, the optimization calculation module splits the data records corresponding to each field value with a frequency greater than or equal to the first predetermined threshold into a predetermined number of sub-time series data.

24. The data processing system according to claim 12, wherein, The optimization calculation module splits the data records corresponding to field values ​​whose frequencies are greater than or equal to a first predetermined threshold and less than a second predetermined threshold into a predetermined number of sub-time series data. The optimization calculation module also sets the processing result of the data record corresponding to each field value or combination of field values ​​whose frequency is greater than or equal to the second predetermined threshold as a predetermined value, wherein the second predetermined threshold is greater than the first predetermined threshold.

25. The data processing system according to claim 12, wherein, The size of the first predetermined threshold is positively correlated with computing resources, and / or the quantity of the predetermined number is positively correlated with computing resources, which include the CPU and / or memory size of a single computing node.

26. A data processing method for extracting features, comprising: The system is configured with a base layer and a computation layer. The base layer includes one or more data processing functions, and the computation layer includes one or more feature calculation modules. The system provides users with one or more feature calculation modules, with different feature calculation modules performing calculations for different types of features; The feature calculation module selected by the user obtains the data and feature calculation element definition information provided by the user. The feature calculation element definition information includes at least one category of feature calculation elements. Each category of feature calculation elements is used to indicate an element for feature extraction processing. The feature calculation module selected by the user selects one feature calculation element from the feature calculation elements of each category to obtain a feature extraction processing strategy composed of feature calculation elements from different categories. Based on the feature extraction processing strategy, the feature calculation module extracts the data object to be subjected to feature extraction processing from the data and determines the data processing function to perform feature extraction processing on the data object. in, The feature calculation module selects a feature calculation element from the feature calculation elements of each category in the form of a Cartesian product to obtain one or more of the feature extraction processing strategies; the feature calculation module determines the Cartesian product method according to the form of the feature element definition information, wherein the Cartesian product method includes global Cartesian product and local Cartesian product.

27. The data processing method for feature extraction according to claim 26, wherein, The feature calculation element definition information includes multiple feature calculation element sets. Each feature calculation element set includes at least one set of feature calculation elements. The feature calculation elements in different sets correspond to different categories. For each feature calculation element set, the feature calculation module selects one feature calculation element from each set of feature calculation elements in a Cartesian product manner to obtain multiple feature extraction processing strategies.

28. The data processing method according to claim 26, further comprising: The feature calculation module selected by the user obtains the data processing function provided by the user and adds the data processing function to the base layer.

29. The data processing method according to claim 26, wherein, The computing layer is divided into an implementation layer and an application layer. The implementation layer includes one or more first-type feature calculation modules. These first-type feature calculation modules are used to extract features corresponding to general business scenarios. The first-type feature calculation modules obtain the feature extraction processing strategy based solely on feature calculation element definition information obtained from external sources. The application layer includes one or more second-type feature calculation modules. The second-type feature calculation modules are used to extract features corresponding to a specific business scenario. The second-type feature calculation modules define at least some feature calculation elements that constitute the feature extraction processing strategy. The at least some feature calculation elements are used to indicate one or more elements of the feature extraction processing corresponding to the specific business scenario.

30. The data processing method according to claim 29, wherein, The first type of feature calculation module in the implementation layer can call the data processing function in the base layer, and / or the first type of feature calculation module can call other first type of feature calculation modules in the same layer. The second type of feature calculation module in the application layer can call the data processing function in the base layer, and / or the second type of feature calculation module can call other second type of feature calculation modules in the same layer, and / or the second type of feature calculation module can call the first type of feature calculation module in the implementation layer.

31. The data processing method according to claim 26, wherein, The base layer also includes: A check function is used to check whether the feature calculation element definition information conforms to the specification; and / or A standardization function is used to convert the feature calculation element definition information into a data structure conforming to a predetermined form; and / or A constant library for providing one or more constants.

32. The data processing method according to claim 26, wherein, The data includes multiple data records, each data record includes one or more fields, and the feature calculation elements of the at least one category include: The first type of feature calculation element includes one or more target fields or combinations of target fields, wherein the target fields or combinations of target fields are used to indicate the field names corresponding to the field values ​​to which feature extraction processing is performed; and / or The second type of feature calculation element includes one or more windows or window combinations, which are used for... Indicates the range of field values ​​corresponding to the feature extraction process being performed; and / or The third type of feature calculation element includes one or more grouping fields or combinations of grouping fields, wherein the grouping fields or combinations of grouping fields are used to indicate the field names on which the data is grouped; and / or The fourth category of feature calculation elements includes one or more algorithm names, which indicate the data processing logic for the field values ​​to be subjected to feature extraction processing; and / or The fifth type of feature calculation element includes one or more sorting fields, which are used to indicate the field names on which the data is sorted.

33. The data processing method according to claim 32, wherein, The feature extraction processing strategy includes the grouping field or a combination of the grouping fields, the target field or a combination of the target fields, and the algorithm name. The steps of the feature calculation module extracting the data object to be feature extracted from the data based on the feature extraction processing strategy and determining the data processing function to perform feature extraction processing on the data object include: The feature calculation module groups the data according to the grouping field or the combination of grouping fields, and data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group; The feature calculation module selects field values ​​from each group of data that correspond to the field names indicated by the target field or the combination of target fields, so as to obtain the data object to be subjected to feature extraction processing. The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

34. The data processing method according to claim 32, wherein, The feature extraction processing strategy includes the grouping field or the grouping field combination, the target field or the target field combination, the window or window combination, the sorting field, and the algorithm name. The steps of the feature calculation module extracting the data object to be feature extracted from the data based on the feature extraction processing strategy and determining the data processing function to perform feature extraction processing on the data object include: The feature calculation module sorts the data according to the field name indicated by the sorting field; For each data record, the feature calculation module extracts data records corresponding to the window range based on the range indicated by the window or window combination and the field name indicated by the sorting field, to obtain first data that includes or excludes that data record: The feature calculation module selects a data record from the first data that has the same field value as the field name indicated by the grouping field or the combination of the grouping fields in that data record, so as to obtain the second data including that data record; The feature calculation module selects field values ​​from the second data that correspond to the field names indicated by the target field or the combination of target fields, so as to obtain the data object to be subjected to feature extraction processing; The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

35. The data processing method according to claim 32, wherein, The feature extraction processing strategy includes the grouping field or a combination of the grouping fields, the target field or a combination of target fields, the window or a combination of windows, the sorting field, and the algorithm name. The steps of the feature calculation module extracting the data object to be feature extracted from the data based on the feature extraction processing strategy and determining the data processing function to perform feature extraction processing on the data object include: The feature calculation module sorts the data according to the field name indicated by the sorting field; For each data record, the feature calculation module selects data records from the data that have the same field value as the field name indicated by the grouping field or the combination of the grouping fields in that data record, so as to obtain third data including that data record; The feature calculation module extracts data records corresponding to the window range from the third data according to the range indicated by the window or window combination and the field name indicated by the sorting field, so as to obtain fourth data including or excluding the data record. The feature calculation module selects field values ​​from the fourth data that correspond to the field names indicated by the target field or the combination of target fields, so as to obtain the data object to be subjected to feature extraction processing; The feature calculation module determines the data processing function used to execute the data processing logic represented by the algorithm name by calling the base layer.

36. The data processing method according to claim 26, further comprising: One or more computing nodes in the distributed system call the corresponding data processing functions in the basic layer to process the data object and obtain one or more features.

37. The data processing method according to claim 26, wherein, The computation layer includes a time-series feature computation module and a summary feature computation module. The time-series feature computation module acquires time-series data and first feature computation element definition information. The time-series data includes multiple data records recorded in chronological order, and each data record includes one or more fields. The first feature computation element definition information includes one or more grouping fields or combinations of grouping fields. The grouping fields or combinations of grouping fields are used to indicate the field names on which the time-series data is grouped. The time-series feature computation module calls the summary feature computation module, and the summary feature computation module performs statistical analysis on the time-series data. The summary feature computation module also includes: The data records corresponding to each field value with a frequency greater than or equal to a first predetermined threshold are split into a predetermined number of sub-time series data, so that the data objects extracted by the time series feature calculation module from the predetermined number of sub-time series data can be distributed to multiple computing nodes.

38. The data processing method according to claim 37, wherein, The time-series feature calculation module sends the time-series data and the definition information of the second feature calculation element, which consists of the grouping field or a combination of grouping fields and the algorithm name used to characterize frequency calculation, to the summary feature calculation module.

39. The data processing method according to claim 38, wherein, In response to receiving the time-series data and the second feature calculation element definition information, the summary feature calculation module obtains one or more feature extraction processing strategies composed of the algorithm name and different grouping fields or combinations of grouping fields. Based on the grouping fields or combinations of grouping fields in the feature extraction processing strategy, the summary feature calculation module groups the time-series data. Data records with the same field value corresponding to the field name indicated by the grouping field or the combination of grouping fields are grouped into the same group. Data from different groups is allocated to one or more computing nodes in a distributed system. The computing nodes call a data processing function corresponding to the algorithm name to process the data allocated to them in order to obtain the frequency of each field value in the time series data corresponding to the field name indicated by the grouping field or the combination of the grouping fields.

40. The data processing method according to claim 37, further comprising: Iterate through each data record in the sub-time series data, and determine whether the sub-time series data contains all the data records required to process the data record based on the data record's processing result dependency. If the sub-time series data does not contain all the data required to process the data record, obtain the missing data records and add them to the sub-time series data.

41. The data processing method according to claim 40, further comprising: The missing data records are marked as redundant data.

42. The data processing method according to claim 41, wherein, The time-series feature calculation module ignores the processing of redundant data, or The time-series feature calculation module processes the redundant data and, after obtaining the processing result of the sub-time-series data, removes the processing result of the redundant data.

43. The data processing method according to claim 37, wherein, The feature extraction processing strategy includes the grouping field or a combination of grouping fields. If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is greater than or equal to the first predetermined threshold, the time series feature calculation module extracts the data object from the sub-time series data corresponding to that field value that has undergone feature extraction processing, and / or If the frequency of the field value corresponding to the field name indicated by the grouping field or combination of grouping fields in the feature extraction processing strategy is less than the first predetermined threshold, the time series feature calculation module extracts the data object to be subjected to feature extraction processing from the data record corresponding to the field value in the time series data.

44. The data processing method according to claim 37 further includes: The data objects extracted from the predetermined number of sub-time series data by the time series feature calculation module are allocated to multiple computing nodes. The computing nodes process the allocated data objects by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

45. The data processing method according to claim 44, further comprising: The data objects extracted by the time series feature calculation module from the data records corresponding to field values ​​with frequencies less than a first predetermined threshold are assigned to a single computing node. The computing node processes the assigned data records by calling the corresponding data processing functions in the base layer to obtain one or more time series features.

46. ​​The data processing method according to claim 45, further comprising: The processing results of the data objects corresponding to each sub-time series data and the processing results of the data objects corresponding to each field value or combination of field values ​​with a frequency less than a first predetermined threshold are merged to obtain the feature extraction results of the time series data.

47. The data processing method according to claim 46, further comprising: The feature extraction results of the time series data are used as new fields and added to the time series data.

48. The data processing method according to claim 37, further comprising: Obtain the optimization calculation strategy provided by the user. When the optimization calculation strategy is the first scheme, the feature processing result of the data record corresponding to the field value whose frequency is greater than or equal to the first predetermined threshold is set to the predetermined value. When the optimization calculation strategy is the second scheme, the data records corresponding to the field values ​​whose frequencies are greater than or equal to the first predetermined threshold are split into a predetermined number of sub-time series data.

49. The data processing method according to claim 37, wherein, The step of splitting the data records corresponding to field values ​​whose frequencies are greater than or equal to a first predetermined threshold into a predetermined number of sub-time series data includes: splitting the data records corresponding to field values ​​whose frequencies are greater than or equal to the first predetermined threshold and less than a second predetermined threshold into a predetermined number of sub-time series data. The method further includes: The processing result of the data record corresponding to each field value or combination of field values ​​whose frequency is greater than or equal to the second predetermined threshold is set as a predetermined value, wherein the second predetermined threshold is greater than the first predetermined threshold.

50. The data processing method according to claim 37, wherein, The size of the first predetermined threshold is positively correlated with computing resources, and / or the quantity of the predetermined number is positively correlated with computing resources, which include the CPU and / or memory size of a single computing node.

51. A system comprising at least one computing device and at least one storage device for storing instructions, wherein, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 26 to 50.

52. A computer-readable storage medium for storing instructions, wherein, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 26 to 50.

Images