Feature processing method, device, medium and computing device
By defining the feature engineering strategy and performing calculations, the problems of poor readability and low versatility of feature engineering in feature processing are solved, and the efficiency of feature processing is improved.
Patent Information
- Application Number
- CN202010753536.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-07-30
AI Technical Summary
In the existing feature processing methods, feature engineering strategies have poor readability, low versatility, and long iteration cycles, resulting in low feature processing efficiency.
Use predefined computing units to describe feature engineering strategies and perform calculations through common computing units to generate final target features, simplifying feature configuration and adjustment process.
The efficiency of feature processing is improved, the feature adjustment and iteration process is simplified, and the problems of poor readability and low versatility of feature engineering are solved.
Smart Images

Figure CN111859928B_ABST
Abstract
Description
Background Art
[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.
[0003] In the related art, the feature processing method mainly uses the feature processing function directly in the program code to process the features one by one, or extracts fields from the data table for feature processing. However, the above processing methods have problems such as poor readability of feature engineering strategies, low universality, long iteration cycle, and difficulty in reuse, which will affect the efficiency of feature processing. Summary of the invention
[0004] The present invention is intended to provide a feature processing method, apparatus, medium and computing device to at least solve the above-mentioned technical problems.
[0005] In a first aspect of the embodiments of the present invention, a feature processing method is provided, comprising:
[0006] Based on the feature engineering configuration item, a list of operation units is generated; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute;
[0007] Based on the operation unit list, an output feature list and a calculation path of the output feature are generated; wherein the operation units included in all the calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list;
[0008] Based on the target feature and its corresponding calculation path, determine the attribute data record to be processed and the target operation unit; wherein the target feature is the output feature in the output feature list;
[0009] An output result is obtained by performing calculation based on the attribute data record to be processed and the target calculation unit, the output result is assigned to the target feature, and the target feature is output based on a target format.
[0010] In one embodiment of the present invention, the predefined operation unit includes at least one of the following:
[0011] Operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
[0012] In one embodiment of the present invention, the method further comprises:
[0013] Performing semantic verification and / or grammatical verification on the feature engineering configuration item;
[0014] If there is a semantic verification error and / or a syntactic verification error in the feature engineering configuration item, a first prompt message is generated; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
[0015] In one embodiment of the present invention, generating an output feature list and a calculation path of the output features based on the computing unit list includes:
[0016] Generate a computing unit feature dependency graph according to the input-output dependency relationship of at least one computing unit included in the computing unit list; wherein the computing unit feature dependency graph is a directed acyclic graph;
[0017] Delete N operation units in the operation unit feature dependency graph to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units that are directly or indirectly dependent on non-final output features; and N is an integer;
[0018] Based on the adjusted operation unit feature dependency graph, an output feature list and a calculation path of the output features are generated.
[0019] In one embodiment of the present invention, the step of generating an output feature list and a calculation path of the output features based on the adjusted computing unit feature dependency graph further includes:
[0020] According to different objects to which the attributes of each operation unit depend, the adjusted operation unit feature dependency graph is divided into at least one subgraph; wherein the at least one subgraph includes a first type of subgraph and a second type of subgraph; outputs of at least some operation units in the first type of subgraph serve as inputs of at least some operation units in the second type of subgraph;
[0021] Based on the at least one subgraph, an output feature list and a calculation path of the output features included in the output feature list are generated.
[0022] In one embodiment of the present invention, the method further comprises:
[0023] Verifying the attribute data record to be processed to obtain a verification result; wherein the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type;
[0024] When the inspection result indicates that an error exists in the attribute data record to be processed, a second prompt information is generated; wherein the second prompt information is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error and the original data.
[0025] In one embodiment of the present invention, the method further comprises:
[0026] When the acquired attribute data record to be processed is empty, the attribute data to be processed is set to a default value.
[0027] In one embodiment of the present invention, the method further comprises:
[0028] When there are computing units without input-output dependency in the target computing units, the computing units without input-output dependency are controlled to perform parallel calculations.
[0029] In one embodiment of the present invention, when performing calculation based on the attribute data record to be processed and the target calculation unit to obtain an output result, the method further includes:
[0030] Performing operation check on the execution process of the target operation unit;
[0031] When an error occurs during the operation of the target operation unit, a third prompt message is generated; the third prompt message is used to prompt at least one of the operation unit information, error information, input, and output where the error occurs.
[0032] In one embodiment of the present invention, the method of determining the target format includes:
[0033] Determining the target format based on the feature engineering configuration item, or determining the target format according to an incoming parameter request; the incoming parameter request includes the specified target format;
[0034] The target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
[0035] In one embodiment of the present invention, the method further comprises:
[0036] If an error occurs in the process of converting or outputting the target feature based on the target format, fourth prompt information is generated; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format.
[0037] In one embodiment of the present invention, the method further comprises:
[0038] For the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature, corresponding statistical data are obtained.
[0039] In one embodiment of the present invention, the method further comprises:
[0040] The attribute data record to be processed is obtained from a local file, a big data cluster or a network service.
[0041] A second aspect of the embodiments of the present invention provides a feature processing device, including:
[0042] A parsing module, configured to generate a list of operation units based on a feature engineering configuration item; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute;
[0043] A compiling module, configured to generate an output feature list and a calculation path of the output features based on the operation unit list; wherein the operation units included in all calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list;
[0044] A data acquisition module, used to determine the attribute data record to be processed and the target operation unit based on the target feature and its corresponding calculation path; wherein the target feature is the output feature in the output feature list;
[0045] The feature processing module is used to perform calculations based on the attribute data record to be processed and the target calculation unit to obtain an output result, assign the output result to the target feature, and output the target feature based on a target format.
[0046] In one embodiment of the present invention, the predefined operation unit includes at least one of the following:
[0047] Operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
[0048] In one embodiment of the present invention, the device further comprises:
[0049] The first verification module is used to perform semantic verification and / or grammatical verification on the feature engineering configuration item; if the feature engineering configuration item has a semantic verification error and / or a grammatical verification error, a first prompt message is generated; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
[0050] In one embodiment of the present invention, the compilation module is used to generate an operation unit feature dependency graph according to the input-output dependency of at least one operation unit included in the operation unit list; wherein the operation unit feature dependency graph is a directed acyclic graph; N operation units in the operation unit feature dependency graph are deleted to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units on which non-final output features are directly or indirectly dependent; N is an integer; based on the adjusted operation unit feature dependency graph, an output feature list and a calculation path of the output feature are generated.
[0051] In one embodiment of the present invention, the compilation module is used to divide the adjusted operation unit feature dependency graph into at least one sub-graph according to different objects to which the attributes of each operation unit depend belong; wherein at least one sub-graph includes a first type of sub-graph and a second type of sub-graph; the outputs of at least some of the operation units in the first type of sub-graph serve as inputs of at least some of the operation units in the second type of sub-graph; based on the at least one sub-graph, an output feature list and a calculation path of the output features contained in the output feature list are generated.
[0052] In one embodiment of the present invention, the device further comprises:
[0053] The second verification module is used to verify the attribute data record to be processed and obtain a verification result; wherein the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type; when the verification result indicates that there is an error in the attribute data record to be processed, a second prompt information is generated; wherein the second prompt information is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error and the original data.
[0054] In one embodiment of the present invention, the data acquisition module is used to set the attribute data to be processed as a default value when the acquired attribute data to be processed record is empty.
[0055] In one embodiment of the present invention, the feature processing module is used to control the operation units without input-output dependency to perform parallel calculations when there are operation units without input-output dependency in the target operation units.
[0056] In one embodiment of the present invention, the device further comprises:
[0057] The third verification module is used to perform operation checks on the execution process of the target operation unit; when an error occurs in the operation process of the target operation unit, a third prompt message is generated; the third prompt message is used to prompt at least one of the operation unit information, error information, input, and output where the error occurs.
[0058] In one embodiment of the present invention, the feature processing module is used to determine the target format based on the feature engineering configuration item, or determine the target format according to an incoming parameter request; the incoming parameter request includes the specified target format;
[0059] The target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
[0060] In one embodiment of the present invention, the device further comprises:
[0061] The fourth verification module is used to generate fourth prompt information if an error occurs in the process of converting or outputting the target feature based on the target format; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format.
[0062] In one embodiment of the present invention, the data acquisition module is used to acquire corresponding statistical data for the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature.
[0063] In one embodiment of the present invention, the data acquisition module is used to acquire the attribute data record to be processed from a local file, a big data cluster or a network service.
[0064] In a third aspect of the embodiments of the present invention, a computer readable medium is provided, on which a computer program is stored, and the program is executed by a processor to implement the steps of the above method.
[0065] In a fourth aspect of the embodiments of the present invention, a computing device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0066] According to the embodiment of the present invention, the feature engineering strategy can be described using predefined operation units, and then the final target feature can be calculated using the general operation units. In this way, since the predefined operation units are highly universal, when it is necessary to adjust the strategy, adjust the target feature to be obtained, and update the strategy, it is only necessary to implement it again based on the predefined operation unit feature engineering configuration items, thereby solving the problems of poor readability, poor universality, and long iteration cycle of feature adjustment in feature engineering, making the adjustment of feature processing simple and fast, and improving the efficiency of feature processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:
[0068] Figure 1 The schematic diagram shows a feature processing method according to an embodiment of the present invention. Figure 1 ;
[0069] Figure 2 A schematic diagram schematically shows a feature dependency graph of a computing unit generated according to a feature engineering configuration item according to an embodiment of the present invention;
[0070] Figure 3 A schematic diagram schematically shows the processing flow of offline testing and online prediction of the model;
[0071] Figure 4 The following schematically shows a flow chart of a feature processing method according to an embodiment of the present invention. Figure 2 ;
[0072] Figure 5 A schematic diagram of a medium structure according to an embodiment of the present invention is schematically shown;
[0073] Figure 6 A schematic diagram of the structure of a feature processing device according to an embodiment of the present invention is shown;
[0074] Figure 7 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown schematically.
[0075] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0076] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0077] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0078] According to an embodiment of the present invention, a feature processing method, apparatus, medium and computing device are proposed.
[0079] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.
[0080] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION
[0082] The inventors have found that the existing feature processing methods include: directly using feature processing functions in the program code to process features one by one; or configuring the feature engineering process through static text, extracting fields from the data table for feature engineering. However, the above processing methods have problems such as poor readability of feature engineering strategies, low universality, few supported data types, and difficulty in reusing feature engineering configurations, which affect the efficiency of feature processing.
[0083] In view of this, the present invention provides a feature processing method, device, medium and computing device, which use predefined operation units to describe feature engineering strategies, and then can use universal operation units to perform calculations to obtain the final target features. In this way, since the predefined operation units are highly universal, when it is necessary to adjust the strategy, adjust the target features to be obtained, and update the strategy, it is only necessary to adjust the feature engineering configuration items based on the predefined operation units and available attributes again, thereby solving the problems of poor readability, poor universality, and long iteration cycle of feature adjustment in feature engineering, making the adjustment of feature processing simple and fast, and improving the efficiency of feature processing.
[0084] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.
[0085] Exemplary Methods
[0086] Reference below Figure 1 The feature processing method according to an exemplary embodiment of the present invention is described as follows, comprising:
[0087] S101: Generate a list of operation units based on a feature engineering configuration item; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute;
[0088] S102: Based on the operation unit list, generate an output feature list and a calculation path of the output feature; wherein the operation units included in all the calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list;
[0089] S103: Determine the attribute data record to be processed and the operation unit based on the target feature and its corresponding calculation path; wherein the target feature is the output feature in the output feature list;
[0090] S104: performing calculations based on the attribute data record to be processed and the calculation unit to obtain output results, assigning the output results to target features, and outputting the target features based on a target format.
[0091] In S101, the feature engineering configuration item may be configured according to a preset operation unit list and an available attribute list in a specific format, wherein the preset operation unit list includes at least one predefined operation unit, and the available attribute list includes at least one predefined available attribute.
[0092] Specifically, the feature engineering configuration item can be selected from a preset list of operation units, and one or more available attributes are selected from a list of available attributes, and the user configures the selected predefined operation units and available attributes in a specific format. When the user configures the feature engineering configuration item, it can be configured according to the current actual needs. For example, the user needs to model a certain model, and then can determine one or more feature information required for this model, and then can configure the feature engineering configuration item according to such needs. In other words, the feature engineering configuration items configured by the user in different situations may be different, but the processing method for obtaining the feature configuration engineering items is the same.
[0093] The predefined operation unit includes at least one of the following contents: operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
[0094] The above-mentioned input fields and output fields may be one or more input fields (that is, one or more inputs), and one or more output fields (that is, one or more outputs).
[0095] Thus, the predefined operation unit of this embodiment has clear functions and high universality. By setting the predefined operation unit, the difficulty of feature engineering configuration can be reduced, thereby improving the configuration efficiency of user feature engineering configuration items.
[0096] Furthermore, the feature engineering configuration item may include a feature engineering strategy; the feature engineering strategy includes a feature engineering operation for one or more features. Each feature engineering operation includes a predefined operation unit and its corresponding input and output.
[0097] In this embodiment, the feature engineering configuration item can be configured in a text description manner. For example, a json file can be used for description. For example, an example of describing a feature engineering configuration item based on a json file can be as follows:
[0098]
[0099] Among them, "features" is used to indicate that its corresponding content is a feature engineering strategy of a feature engineering configuration item; in a feature engineering strategy, the "feat" keyword is used to indicate a feature engineering operation. A feature engineering operation can include the name of the operation unit, input, and output. For example, there are a total of 6 operation units in the feature engineering configuration item, and 5 of them are directly written in the above example, namely: map_top_k, map_norm, bucketize, map_get_weight, and cross.
[0100] In particular, in the above example, the name of the operation unit is not directly written in the item {"feat":"i_cate"}. In this case, the default operation unit directly_output can be used, which represents the direct use of input as output; in other words, {"feat":"i_cate"} and {"feat":"i_cate=directly_output(i_cate)"} have the same meaning.
[0101] In addition, the feature engineering configuration item can also include the definition of the target format. For example, in the above example, the target format is set to "HASH_SPARSE" through the content corresponding to the "feature_format" keyword.
[0102] When executing S101, the method also includes: performing semantic verification and / or grammatical verification on the feature engineering configuration item; if there is a semantic verification error and / or a grammatical verification error in the feature engineering configuration item, generating a first prompt message; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
[0103] Specifically, syntax checking may include spelling checking, parameter checking, etc. The semantic checking may include operation unit name checking, input dependency checking, etc.
[0104] That is, when a spelling error or a format error occurs in the configuration content of the feature engineering configuration item during the semantic verification and / or syntax verification of the feature engineering configuration item, it is determined that there is a syntax error; and / or,
[0105] When the attribute that the configuration content in the feature engineering configuration item depends on cannot be found in the available attributes, it is determined that there is a syntax error; here, the configuration content can include an operation unit in the feature engineering configuration item and its input and output. If the input of the operation unit should be an attribute, but it cannot be found in the list of available attributes, it is considered that there is a syntax error.
[0106] and / or,
[0107] When the input that the configuration content in the feature engineering configuration item depends on cannot be found in the output of the configuration content before the current configuration line, it is determined that there is a semantic error; here, the configuration content can include a computing unit in a feature engineering configuration item and its input and output. When the input of a computing unit is neither an available attribute in the available attribute list nor the output of other computing units, it can be determined that there is a semantic error.
[0108] and / or,
[0109] When at least one of the number of inputs, the input format, and the number of outputs of the operation unit does not match the preset definition, it is determined that there is a semantic error.
[0110] Correspondingly, when it is determined that a configuration item has a syntax error or a semantic error, at least one of the configuration error location, the error cause and the error type is prompted through the first prompt information.
[0111] By executing S101, the pre-generated (or pre-configured) feature engineering configuration items can be parsed to obtain a list of all operation units described by the feature engineering configuration items, wherein the list of operation units may include at least one operation unit, and multiple inputs and / or multiple outputs of each operation unit in the at least one operation unit. Here, the output of the operation unit may be the final output, or it may be an intermediate output (or intermediate feature); the intermediate output (or intermediate feature) may be a feature that is not finally output, but needs to be used as an input of other operation units for use by other operation units. If the output of the operation unit is the final feature, it can be called an output feature.
[0112] After obtaining the list of operation units through S101, the final features can be obtained by directly executing them one by one in order according to the list of operation units. However, there is usually a lot of computational redundancy in the feature engineering configuration defined by the user, and the operation efficiency is very low. Therefore, the solution provided in this embodiment compiles and optimizes the feature engineering configuration list through the subsequent step S102.
[0113] Specifically, in the above S102, generating the output feature list and the calculation path of the output features based on the operation unit list includes:
[0114] Generate a computing unit feature dependency graph according to the input-output dependency relationship of at least one computing unit included in the computing unit list; wherein the computing unit feature dependency graph is a directed acyclic graph;
[0115] Delete N operation units in the operation unit feature dependency graph to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units that are directly or indirectly dependent on non-final output features; and N is an integer;
[0116] Based on the adjusted operation unit feature dependency graph, an output feature list and a calculation path of the output features are generated.
[0117] The aforementioned operation unit dependency graph may be a directed acyclic graph. Then, the operation units that are not depended on by the final output feature are deleted from the operation unit dependency graph.
[0118] It should be noted that N is an integer including N can be 0 or an integer greater than or equal to 1. In other words, if the feature engineering configuration items set by the user may be relatively simple or non-redundant, then there may be no operation units that need to be deleted in the final feature dependency graph, and N can be 0. In other cases, as long as the operation unit feature dependency graph contains operation units that are not directly or indirectly dependent on the output feature, these operation units can be deleted. In this case, N can be greater than or equal to 1.
[0119] The step of generating an output feature list and a calculation path of the output features based on the adjusted computing unit feature dependency graph further includes:
[0120] According to different objects to which the attributes of each operation unit depend belong, the adjusted operation unit feature dependency graph is divided into at least one subgraph; wherein the at least one subgraph includes a first type of subgraph and a second type of subgraph; the outputs of at least some operation units in the first type of subgraph can be used as inputs of at least some operation units in the second type of subgraph;
[0121] Based on the at least one subgraph, an output feature list and a calculation path of the output features included in the output feature list are generated.
[0122] That is, the objects to which the initial attributes that the operation units depend are divided into different subgraphs. For example, assuming that among multiple operation units, the input of the first operation unit depends on the output of the second operation unit, if the input of the second operation unit is the first attribute in the available attribute list, then it can be considered that the initial attribute that the first operation unit and the second operation unit depend on is the first attribute in the available attributes, and the first operation unit and the second operation unit can be divided into the same subgraph based on the object to which the first attribute belongs. The determination of the initial attributes of other operation units is similar, and will not be repeated here.
[0123] One of the subgraphs contains all the operation units that only depend on one same object, and the other operation units belong to another subgraph. The number of subgraphs obtained by the specific division is related to the number of attributes or types of the object. For example, if there are two types of objects, one is a user and the other is an item, then two subgraphs can be obtained. If there are three or more objects, such as objects 1, 2, and 3, then three subgraphs can be obtained based on these three objects. Of course, it can also be pre-specified that only two subgraphs are divided. For example, the object to which the initial attribute of the operation unit depends is the user, which is divided into one subgraph, and the object to which the initial attribute of the operation unit depends is other objects other than the user, which is divided into another subgraph.
[0124] After obtaining a list of operation units based on the example of the above-mentioned feature engineering configuration items, a corresponding operation unit feature dependency graph can be generated based on the operation unit list, and an adjusted operation unit feature dependency graph can be obtained after deleting some operation units from the feature dependency graph, and then the adjusted operation unit feature dependency graph can be divided to obtain at least one sub-graph.
[0125] by Figure 2 Take this as an example to illustrate: Figure 2The left subgraph to the left of the middle dotted line may be a first-type subgraph, which contains a subgraph of operation units that only depend on users (attributes), including all operation units that only depend on user attributes (attributes prefixed with u_, where u stands for user). Figure 2 In the figure, the right subgraph on the right side of the dotted line may be a subgraph of the second type, which contains other operation unit subgraphs, including all feature units that depend on items (attributes) (attributes prefixed with i_, i represents item, i.e., item).
[0126] from Figure 2 It can be seen that the original attributes that the left subgraph (i.e., the first type of subgraph) and the right subgraph (i.e., the second type of subgraph) depend on belong to the user and the item respectively; and some operation units of the right subgraph depend on the output of the left subgraph. For example, the input of the operation unit "map_get_value" in the right subgraph depends on the output "u_cate_top_norm" of the left subgraph.
[0127] For example, in a recommendation scenario, a to-be-processed attribute data record may contain 1 user and M items (M is an integer greater than or equal to 2). If the feature unit list is used directly for calculation, the operation unit corresponding to the feature that only depends on the user attribute will be calculated M times, which results in a large amount of computational redundancy. However, in the solution provided by this embodiment, since some operation units in the second-type subgraph depend on the output of the first-type subgraph, after a certain operation unit in the first-type subgraph obtains the corresponding output, no matter how many operation units in the second-type subgraph need to reference the output, there is no need to calculate again. Instead, the operation unit of the second-type subgraph only needs to directly obtain the corresponding output feature of a certain operation unit in the first-type subgraph.
[0128] In the above process, the original list of operation units is first converted into a feature operation dependency graph based on the input-output dependency, and then the operation units that are not dependent on the final output are deleted from the graph. At the same time, the dependency graph is divided into two subgraphs according to the different objects to which the original attributes of the dependency belong. This dependency graph calculation mode can effectively reduce unnecessary calculations, improve the parallelism of calculations, and speed up calculations.
[0129] An output feature list may be obtained based on at least one subgraph. The output feature list may include at least one output feature. Here, the output feature may be understood as the output of a part of the computing unit.
[0130] Whether the output of an operation unit is an output feature (that is, an output feature included in the output feature list) or an intermediate output (or intermediate feature) can be determined based on the configuration content for each operation unit (or feature engineering item) in the feature engineering configuration item.
[0131] For example, the preceding example contains the following:
[0132]
[0133] Among them, "visibility": "hidden" is used to indicate whether the output is visible. If it is configured as "hidden", it means hidden, that is, the output "u_cate_top" is not finally visible. In other words, it can be considered as an intermediate output or intermediate feature. u_cate_top is not the final output feature and will not appear in the output feature list.
[0134] For another example, in the aforementioned {"feat":"u_cate_top_norm=map_norm(u_cate_top)"}, in the definition of this feature engineering item, "visibility":"hidden" is not set, then the output u_cate_top_norm of the operation unit is finally visible, that is, the output u_cate_top_norm is one of the output features in the output feature list.
[0135] In addition, the complete calculation path of the output feature in the feature dependency graph can be listed according to at least one subgraph. A calculation path can include the initial attribute, one or more operation units, and the input and output of each operation unit.
[0136] by Figure 2 For example, the calculation path corresponding to the feature "i_cate_match_weight" is: attribute i_cate, input u_cate_top_norm, and operation unit map_get_value, and the output is the feature "i_cate_match_weight".
[0137] In addition, the at least one subgraph may also include the names and data types of all dependent (original or initial) attributes, and the objects to which all dependent (original or initial) attributes belong. Here, the data type may be a numeric type (e.g., an integer type int, a floating point type float), a text type (e.g., a string type string), a list type (e.g., an integer list type list <int>) or unstructured types (for example, map<string,double> ) etc. The object may be the aforementioned user or item etc. In this way, since the computing unit can perform feature engineering processing on various types of data including unstructured data, support for unstructured data is achieved.
[0138] The aforementioned processing of S102 may further include: acquiring corresponding statistical data for the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature.
[0139] That is, based on the obtained output feature list and the calculation path of the output features, all the operation units that may eventually be used can be determined. In this embodiment, the required statistical data are obtained in advance for loading for the operation units that rely on statistical data among all the operation units. In this way, reading a large amount of statistical data during the calculation process can be avoided to improve processing efficiency.
[0140] Whether the operation unit needs to rely on statistical data can also be determined according to the description or definition in the feature engineering configuration item. For example, the json file example above contains {"feat":"u_age_bucket=bucketize(u_age)","boundaries":[15,20,25,35,40,50]}, where boundaries means the statistical data to be obtained, and the following "[15,20,25,35,40,50]" indicates at which boundaries or within which thresholds the statistical data is obtained.
[0141] Based on at least one subgraph obtained by dividing the adjusted computing unit feature dependency graph, an output feature list and a calculation path of the output feature can be obtained, wherein the calculation path of the output feature can be a calculation path corresponding to one or more output features included in the output feature list.
[0142] After completing the above processing, the user can specify one or more features for calculation, wherein the one or more features specified by the user are one or more of the output features in the output feature list. Here, the one or more features specified by the user are the target features in the above S103.
[0143] Furthermore, when the target feature is determined, the calculation path of the target feature in S103 may be determined. The calculation path of the target feature may be determined by searching the calculation path corresponding to the target feature from the calculation paths of the aforementioned multiple output features.
[0144] Here, no matter how many target features are specified by the user, each target feature is processed in the same way. Therefore, this embodiment will not explain them one by one, but will only describe them in detail from the perspective of one target feature.
[0145] In S103, at least one operation unit included in the calculation path of the target feature can be used as the target operation unit, and the (original or original) attribute that the calculation path relies on can be used as the attribute to be processed; then the attribute data record to be processed is obtained from a local file, a big data cluster or a network service. In particular, the need to quickly add features with static attributes can be achieved by obtaining the attribute repository.
[0146] That is to say, based on the calculation path corresponding to the target feature, the initial attributes that each target operation unit in the calculation path depends on can be determined. Based on the initial attributes, the corresponding attribute record data can be obtained from local files, big data clusters or network services, and the obtained attribute record data can be used as the attribute record data to be processed.
[0147] The attribute data record to be processed may be one or more, and the target operation unit may be one or more, which is not limited here.
[0148] When executing S103, the method further includes: verifying the attribute data record to be processed to obtain a verification result; wherein the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type;
[0149] When the inspection result indicates that an error exists in the attribute data record to be processed, a second prompt information is generated; wherein the second prompt information is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error and the original data.
[0150] For example, the attribute data type to be processed can be a numeric type (e.g., an integer type int, a floating point type float), a text type (e.g., a string type string), a list type (e.g., an integer list <int>) or unstructured types (for example, map<string,double> ). The attribute data range can be set according to the actual situation and is related to the type of attribute data to be processed. For example, if the attribute data to be processed is text type but a decimal appears, the verification result is an error. Then, the aforementioned second prompt information is generated according to the name of the attribute data to be processed, the cause of the error, etc.
[0151] Optionally, the method further includes: when the acquired attribute data record to be processed is empty, setting the attribute data to be processed as a default value.
[0152] It should be noted that the default value of the attribute data to be processed is related to its type, and different types of attribute data to be processed may correspond to different default values. For example, if the attribute data to be processed is of text type, then its default value is a default text data; for another example, if the attribute data to be processed is of floating point type, then the corresponding default value is a default floating point data. The default values of other types of attribute data to be processed may also have their own corresponding default values, which will not be enumerated in this embodiment.
[0153] By executing S103, the corresponding calculation path can be determined according to the target feature, and then the corresponding attribute data record to be processed and one or more target operation units to be used for the operation can be determined. Then, S104 is executed to perform the operation based on the attribute data record to be processed and the processing logic of one or more target operation units, and the obtained operation result is assigned to the target feature.
[0154] The input of each target operation unit may be the attribute data record to be processed or the output of a target operation unit that has completed the operation.
[0155] In S104, when the output result is obtained by performing calculation based on the attribute data record to be processed and the target calculation unit, there may be the following ways:
[0156] In one method, operations are performed based on the attribute data records to be processed and the corresponding target operation units in turn to obtain the output characteristics of each target operation unit until the final output result is obtained. Figure 2 Take this as an example to illustrate. Assuming that the target features are "u_age_bucket" and "u_age_bucket_i_cate", we can first calculate the attribute data record "u_age" and the target operation unit "bucketize" to be processed to obtain the corresponding output result and assign the target feature; then, based on "u-age" and "i-cate", we can calculate the corresponding output result through the target operation unit "cross" and assign the target feature.
[0157] In another embodiment, when there are computing units without input-output dependency in the target computing units, the computing units without input-output dependency are controlled to perform parallel computing.
[0158] The operation unit without input-output dependency refers to a plurality of target operation units where the input characteristics of the first target operation unit are irrelevant to the output characteristics of the second target operation unit, and the output characteristics of the first target operation unit are not used as the input attributes of the second target operation unit. Then, there is no input-output dependency between the first target operation unit and the second target operation unit. Figure 2 For example, if the target computing unit contains Figure 2 The bucketize and map_top_k operation units are Figure 2 It can be seen that there is no input-output dependency between the two computing units, so the two computing units can be calculated in parallel.
[0159] It should be understood that, in the process of a target feature calculation, there may be multiple target calculation units that have no dependencies on each other, not limited to the aforementioned two calculation units. For example, there may be 3 or more calculation units that have no input-output dependencies, and all of them can be calculated in parallel at the same time, but this embodiment does not exhaustively list them.
[0160] On the contrary, if one of the target computing units depends on the output of other target computing units, parallel computing cannot be performed, but the computing must be performed after the other target computing unit is completed. Figure 2 In the example, the output of "map_norm" "u_cate_top_norm" is the "corss" operation unit ( Figure 2 The rightmost unit in the figure is used to obtain the input of the computing unit of the feature "u_age_bucket_i_cate". These two computing units cannot be calculated in parallel. The calculation of "cross" needs to wait until "map_norm" completes the calculation and obtains the output of "u_cate_top_norm".
[0161] This method can determine the target computing units for parallel processing and then perform parallel computing, which can improve the degree of parallelism between computing units in the same group and improve computing efficiency to a certain extent.
[0162] It should be pointed out that no matter which of the above two methods is used to calculate the target feature unit, if the output of a target operation unit is used by one or more other target operation units, or is used multiple times by another target operation unit, the output only needs to be calculated once. This method avoids multiple calculations of some operation units and improves the efficiency of feature engineering calculations. For example, in a recommendation scenario, it is necessary to recommend L candidate items to a user (L is an integer greater than or equal to 2). Using the method of calculating the subgraph separately, the operation unit in the user subgraph only needs to be calculated once, while the user features need to be calculated L times in the direct calculation method. It can be seen from this that the aforementioned processing method of dividing the subgraph in this embodiment greatly reduces the computational redundancy for subsequent calculations.
[0163] The target format also needs to be determined before executing S104. Because different machine learning algorithms and frameworks may rely on different feature formats, even the same machine learning algorithm may have different effects on different input format features in different data scenarios. Taking the logistic regression classification algorithm as an example, in large-scale sparse data scenarios, it is suitable to use sparse format feature vectors indexed as hash values; when the data volume is small and there are many numerical features, it may be better to use dense vectors without names to train the model. Therefore, in actual processing, the user may determine which target format to use for the final target feature based on the current scenario or current needs.
[0164] In the solution provided by this embodiment, the target format can be determined by: determining the target format based on the feature engineering configuration item, or determining the target format according to an incoming parameter request; the incoming parameter request contains the specified target format; wherein the target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
[0165] Specifically, in one case, when configuring a feature engineering configuration item, the user sets the content or field "feature_format". In this way, when generating a list of operation units by parsing the feature engineering configuration item in the process of executing S101, the target format can be determined by parsing the content of the feature format in the feature engineering configuration item.
[0166] In another case, the user may not set the "feature_format" content or field when configuring the feature engineering configuration item. In this case, before executing S104, or before executing S103, or before executing S101, the user may issue an incoming parameter request, and the incoming parameter includes the target format specified by the user. In this way, the target format can be determined by receiving the incoming parameter request. Furthermore, the user can determine the format of the target feature currently required according to actual needs, and then use it as the target format and add the target format to the incoming parameter request. The incoming parameter can be an incoming parameter of a specified model (a model in training or a prediction model).
[0167] There is another case, that is, when the user configures the feature engineering configuration item, the "feature_format" content or field is set in it, and the user also initiates an incoming parameter request. In this case, the target format to be used can be determined according to the pre-set rules. For example, the pre-set rule is that the incoming parameter request initiated by the user has the highest priority, then regardless of whether the feature engineering configuration item contains the feature format, the target format is determined according to the incoming parameter request; or vice versa, that is, setting the priority of determining the target format according to the feature format contained in the feature engineering configuration item is the highest, then regardless of whether the user issues an incoming parameter request, the target format is determined according to the feature format contained in the feature engineering configuration item.
[0168] S104: based on the determined target format, convert the target features into features of a specified target format and output them.
[0169] Take the target feature as the user category preference feature (u_category) as an example. Assume that the user category preference data is {"entertainment":0.8,"sports":0.6}, which means that the user's preference coefficient for the entertainment category is 0.8, and the preference coefficient for sports is 0.6. When the feature name is a sparse format feature vector, the output result is {"u_category__Entertainment":0.8,"u_category__Sports":0.6}; when the output format is a sparse format feature vector with the index as a hash value, the output result is {1931643211:0.8,1355713286:0.6}, where 1931643211 and 13557132 86 is the result of hashing the two values "u_category__Entertainment" and "u_category__Sports"; the format is a dense vector with a name, and the output result is {"u_category_0":0.8,"u_category_3":0.6}, where the values 0 and 3 are the index values of "Entertainment" and "Sports" in the category statistics list; when the output format is a dense vector without a name, the output result is {12:0.8,15:0.6}, where 12 and 15 are the indexes of the "u_category_0" and "u_category_3" features in the list of all output feature names.
[0170] It should also be pointed out that for the same target feature, multiple processing can be performed to obtain the same target feature in multiple target formats. The multiple processing can be multiple processing of converting the target feature into the target format, and / or can be a process of multiple calculations using different attribute data records, which is not limited in this embodiment. The target format can be indicated or determined in different ways in the multiple processings, for example, one or more processings can determine the target format through the feature engineering configuration item, and another one or more processings can determine the target format based on the incoming parameter request.
[0171] In addition, the method also includes: recording the output generated after all target operation units perform calculations. The output here may include intermediate outputs (or may be called intermediate features other than the final target features) and final output results (that is, output results used to assign values to target features). Among them, the recorded output results or intermediate outputs may be data content that is unrelated to the target format. Based on this, when an inspection is required, the content of the intermediate output may be used to check whether there are any errors in the intermediate process of executing the calculation. In addition, by saving the calculation results of the target operation unit as data that is unrelated to the target format, it is easy to convert the output results into feature outputs of different target formats, meeting the different requirements of different machine learning algorithms and frameworks for feature input formats.
[0172] Furthermore, the process of executing S104 may also include: performing an operation check on the execution process of the target operation unit; generating a third prompt message when an error occurs in the operation process of the target operation unit; the third prompt message is used to prompt at least one of the operation unit information, error information, and input / output where the error occurs.
[0173] That is to say, each time the operation of the target feature is completed, an operation check can be performed. Here, the operation check can use the aforementioned record of the output generated after all target operation units perform the calculation. For example, it can be determined whether there is an error when the output of a certain target operation unit is used as the input of another target operation unit, such as an error caused by format mismatch, or an error in the result, etc. If it is determined that there is an error, then the error operation unit information, error information and the corresponding input location can be prompted. Of course, there may be other errors, and this embodiment does not list them all.
[0174] The solution provided by this embodiment can also provide other verification methods. It has been described above that for the same target feature, multiple processing can be performed to obtain the same target feature in multiple target formats, that is, the same target feature in multiple different target formats can be obtained. Correspondingly, this embodiment can also: use the same target feature output in multiple target formats to perform mutual verification to obtain a verification result. The verification result can be used to indicate that the target feature of a certain target format does not match the content of the target features of other formats. At this time, it can be considered that there is an error in the calculation, and the output recorded each time the target feature is calculated can be checked back to determine the location of the error, and then a prompt can be given to indicate at least one of the target format where the error occurred, the target operation unit where the error occurred, the error information, etc.
[0175] That is, target features of different target formats can be used for mutual verification and debugging comparison. For example, when the target format is a sparse format feature vector with a hash value as the index, it is not easy to determine the feature name corresponding to the hash value, and features in a sparse format feature vector format with a feature name as the target format can be used for comparison one by one.
[0176] In this way, the solution of this embodiment can output the result generated by the feature operation according to the specified target format. Since the target format can be specified according to actual needs, it supports multiple format feature outputs and can be directly adapted to different algorithms and frameworks. At the same time, different features can verify each other to ensure the correctness of the calculation process.
[0177] The solution provided in this embodiment can also provide other verification methods, such as: if an error occurs in the process of converting or outputting the target feature based on the target format, a fourth prompt information is generated; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format. This verification can determine whether the target feature finally output is in the target format, or in other words, whether the format of the target feature is the specified target format. If not, the feature conversion and feature output processing process is checked back to determine whether there is an error, and then determine the location of the error, etc.
[0178] The aforementioned multiple verification methods provided in this embodiment can be used in full each time the target feature is generated, or only part of them can be used according to the settings. These methods are all within the protection scope of this embodiment and will not be described one by one.
[0179] The feature processing method provided in this embodiment can be applied to electronic devices or computing devices, for example, it can be executed by a system (or server system) or computing device that performs feature engineering for machine learning.
[0180] The feature processing method provided in this embodiment can be applied in the field of machine learning. The machine learning focuses on automatically analyzing and obtaining rules from data, and using the rules to predict unknown data. Figure 3 The basic process of offline training and online prediction of machine learning is shown as follows: Figure 3 As shown in the figure, the offline training part first needs to prepare the training data, convert the original attribute data into target features through feature engineering steps, then input the target features into the machine learning algorithm to train the model, and then evaluate the training model offline. Figure 3 As shown, in the online prediction process, data needs to be prepared first, and then the attribute data is converted into target features through feature engineering, and then the target features are input into the model that has been loaded with offline training for prediction, and finally the online prediction results are evaluated. Machine learning methods include supervised learning, unsupervised learning and other methods, which all belong to the scope of sample learning. No matter which learning method is used, it depends on the analysis and understanding of the attribute data to be analyzed. Usually, each record in the attribute data to be analyzed contains one or more attribute information (such as the age and gender of the character portrait), among which some attribute information (such as the gender of the data type is a string) cannot be directly applied to the machine learning algorithm and needs to be converted into numerical features. At the same time, in order to more accurately characterize the data to be analyzed and train a better model, the feature itself also needs to be converted and / or combined into a new feature. This process is generally called feature engineering. It can be seen that it is very important to obtain the target feature through feature processing of feature engineering in the field of machine learning, and the quality of feature processing directly affects the quality of the machine learning training model. Based on this, this embodiment provides the above processing scheme for feature processing.
[0181] The feature processing method provided in the above embodiment is combined with Figure 4 An exemplary description may include: parsing feature engineering configuration items; compiling and optimizing feature engineering strategies; obtaining attribute data records to be processed; executing feature engineering strategy operations; and outputting specified format features (that is, outputting target features in the target format). Among them, the process of parsing feature engineering configuration items includes the processing content of the aforementioned S101; the process of compiling and optimizing feature engineering strategies includes the processing content of the aforementioned S102; the process of obtaining attribute data records to be processed may include the processing content of the aforementioned S103; the process of executing feature engineering strategy operations and outputting specified format features may include the processing content of the aforementioned S104, which will not be discussed here. Figure 4 The processing process is described in detail.
[0182] The present invention uses predefined operation units and available attributes to describe feature engineering strategies, and then can use universal operation units to perform calculations to obtain the final target features. In this way, since the predefined operation units are highly universal, when it is necessary to adjust the strategy, adjust the target features to be obtained, and update the strategy, it is only necessary to adjust the feature engineering configuration items based on the predefined operation units and available attributes again, and no longer rely on the upgrade deployment of the program, thereby solving the problems of poor readability, poor universality, non-reusability, long feature adjustment iteration cycle, high calculation redundancy, etc. of feature engineering in the prior art, making feature adjustment simple and fast, and improving the efficiency of feature processing.
[0183] In addition, since the operation units included in all calculation paths of the output features in the output feature list in the process of compiling the optimized feature operation strategy of the present invention are at least part of the operation units in the operation unit list, that is, some unnecessary operation units can be deleted, thereby reducing the computational redundancy in feature processing and improving processing efficiency.
[0184] Exemplary Media
[0185] After introducing the method of the exemplary embodiment of the present invention, next, refer to Figure 5 A medium according to an exemplary embodiment of the present invention is described.
[0186] In some possible embodiments, various aspects of the present invention may also be implemented as a computer-readable medium having a program stored thereon, which, when executed by a processor, is used to implement the steps of the feature processing method according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.
[0187] Specifically, the processor is used to implement the following steps when executing the program:
[0188] Based on the feature engineering configuration item, a list of operation units is generated; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute;
[0189] Based on the operation unit list, an output feature list and a calculation path of the output feature are generated; wherein the operation units included in all the calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list;
[0190] Based on the target feature and its corresponding calculation path, determine the attribute data record to be processed and the target operation unit; wherein the target feature is the output feature in the output feature list;
[0191] An output result is obtained by performing calculation based on the attribute data record to be processed and the target calculation unit, the output result is assigned to the target feature, and the target feature is output based on a target format.
[0192] It should be noted that the above-mentioned medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0193] like Figure 5 As shown, a medium 50 according to an embodiment of the present invention is described, which can be a portable compact disk read-only memory (CD-ROM) and include a program and can be run on a device. However, the present invention is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device.
[0194] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0195] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).
[0196] Exemplary Devices
[0197] After introducing the medium of the exemplary embodiment of the present invention, next, reference is made to Figure 6 A device according to an exemplary embodiment of the present invention is described.
[0198] The embodiment of the present invention provides a feature processing device, such as Figure 6 As shown, including:
[0199] A parsing module 61 is used to generate a list of operation units based on a feature engineering configuration item; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute;
[0200] A compiling module 62, configured to generate an output feature list and a calculation path of the output features based on the operation unit list; wherein the operation units included in all the calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list;
[0201] A data acquisition module 63 is used to determine the attribute data record to be processed and the target operation unit based on the target feature and its corresponding calculation path; wherein the target feature is the output feature in the output feature list;
[0202] The feature processing module 64 is used to perform calculations based on the attribute data record to be processed and the target calculation unit to obtain an output result, assign the output result to the target feature, and output the target feature based on a target format.
[0203] The predefined operation unit includes at least one of the following:
[0204] Operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
[0205] The device also includes:
[0206] The first verification module 65 is used to perform semantic verification and / or grammatical verification on the feature engineering configuration item; if the feature engineering configuration item has a semantic verification error and / or a grammatical verification error, a first prompt message is generated; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
[0207] The compilation module 62 is used to generate an operation unit feature dependency graph according to the input-output dependency relationship of at least one operation unit included in the operation unit list; wherein the operation unit feature dependency graph is a directed acyclic graph; delete N operation units in the operation unit feature dependency graph to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units that are directly or indirectly dependent on non-final output features; N is an integer; based on the adjusted operation unit feature dependency graph, generate an output feature list and a calculation path of the output features.
[0208] The compilation module 62 is used to divide the adjusted operation unit feature dependency graph into at least one sub-graph according to different objects to which the attributes of each operation unit depend belong; wherein at least one sub-graph includes a first type of sub-graph and a second type of sub-graph; the outputs of at least some of the operation units in the first type of sub-graph serve as inputs of at least some of the operation units in the second type of sub-graph; based on the at least one sub-graph, an output feature list and a calculation path of the output features contained in the output feature list are generated.
[0209] The device also includes:
[0210] The second verification module 66 is used to verify the attribute data record to be processed and obtain a verification result; wherein, the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type; when the verification result indicates that there is an error in the attribute data record to be processed, a second prompt message is generated; wherein, the second prompt message is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error and the original data.
[0211] The data acquisition module 63 is used to set the attribute data to be processed as a default value when the acquired attribute data to be processed is empty.
[0212] The feature processing module 64 is used to control the operation units without input-output dependency to perform parallel calculations when there are operation units without input-output dependency in the target operation units.
[0213] The feature processing module 64 is used to determine the target format based on the feature engineering configuration item, or determine the target format according to an incoming parameter request; the incoming parameter request includes the specified target format;
[0214] The target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
[0215] The device also includes:
[0216] The third verification module 67 is used to perform operation check on the execution process of the target operation unit; when an error occurs in the operation process of the target operation unit, a third prompt message is generated; the third prompt message is used to prompt at least one of the operation unit information, error information, input, and output where the error occurs.
[0217] The device also includes:
[0218] The fourth verification module 68 is used to generate fourth prompt information if an error occurs in the process of converting or outputting the target feature based on the target format; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format.
[0219] The data acquisition module 63 is used to acquire corresponding statistical data for the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature.
[0220] The data acquisition module 63 is used to acquire the attribute data record to be processed from a local file, a big data cluster or a network service.
[0221] The specific processing of each module in the feature processing device in this embodiment is the same as the processing in the feature processing method in the aforementioned embodiment, and will not be repeated here.
[0222] Exemplary Computing Devices
[0223] After introducing the method, medium and apparatus of the exemplary embodiments of the present invention, next, reference is made to Figure 7 A computing device according to an exemplary embodiment of the present invention is described.
[0224] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0225] In some possible implementations, a computing device according to an implementation of the present invention may include at least one processing unit and at least one storage unit. The storage unit stores program code, and when the program code is executed by the processing unit, the processing unit executes the steps of the feature processing method according to various exemplary implementations of the present invention described in the above "Exemplary Method" section of this specification.
[0226] Refer to the following Figure 7 1 and 2 to describe a computing device 90 according to this embodiment of the present invention. Figure 7 The computing device 90 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present invention.
[0227] like Figure 7 As shown, the computing device 90 is in the form of a general computing device. The components of the computing device 90 may include but are not limited to: at least one processing unit 901, at least one storage unit 902, and a bus 903 connecting different system components (including the processing unit 901 and the storage unit 902).
[0228] The bus 903 includes a data bus, a control bus, and an address bus.
[0229] The storage unit 902 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 9021 and / or a cache memory 9022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 9023 .
[0230] The storage unit 902 may also include a program / utility 9025 having a set (at least one) of program modules 9024, such program modules 9024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0231] The computing device 90 may also communicate with one or more external devices 904 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 905. Furthermore, the computing device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 906. Figure 7 As shown, the network adapter 906 communicates with other modules of the computing device 90 via the bus 903. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computing device 90, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0232] It should be noted that although several units / modules or sub-units / modules of the feature processing device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.
[0233] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0234] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.< / int> < / int>
Claims
1. A feature processing method, include: Based on the feature engineering configuration item, a list of operation units is generated; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute; Based on the operation unit list, an output feature list and a calculation path of the output feature are generated; wherein the operation units included in all the calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list; Based on the target feature and its corresponding calculation path, determine the attribute data record to be processed and the target operation unit; wherein the target feature is the output feature in the output feature list; Performing calculation based on the attribute data record to be processed and the target calculation unit to obtain an output result, assigning the output result to the target feature, and outputting the target feature based on a target format; Wherein, the generating of the output feature list and the calculation path of the output feature based on the operation unit list includes: generating an operation unit feature dependency graph according to the input-output dependency relationship of at least one operation unit included in the operation unit list; wherein the operation unit feature dependency graph is a directed acyclic graph; deleting N operation units in the operation unit feature dependency graph to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units directly or indirectly dependent on non-final output features; N is an integer; generating the output feature list and the calculation path of the output feature based on the adjusted operation unit feature dependency graph; The method of generating an output feature list and a calculation path of the output features based on the adjusted operation unit feature dependency graph includes: dividing the adjusted operation unit feature dependency graph into at least one subgraph according to different objects to which the attributes to which each operation unit depends belong; wherein at least one subgraph includes a first type of subgraph and a second type of subgraph; the outputs of at least some of the operation units in the first type of subgraph serve as inputs of at least some of the operation units in the second type of subgraph; and generating an output feature list and a calculation path of the output features included in the output feature list based on the at least one subgraph.
2. The method according to claim 1, It is characterized in that The predefined operation unit includes at least one of the following: Operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
3. The method according to claim 1, It is characterized in that The method further comprises: Performing semantic verification and / or grammatical verification on the feature engineering configuration item; If there is a semantic verification error and / or a syntactic verification error in the feature engineering configuration item, a first prompt message is generated; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: Verifying the attribute data record to be processed to obtain a verification result; wherein the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type; When the verification result indicates that there is an error in the attribute data record to be processed, a second prompt information is generated; wherein the second prompt information is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error, and the original data.
5. The method according to claim 4, It is characterized in that The method further comprises: When the acquired attribute data record to be processed is empty, the attribute data to be processed is set to a default value.
6. The method according to claim 1, It is characterized in that The method further comprises: When there are computing units without input-output dependency in the target computing units, the computing units without input-output dependency are controlled to perform parallel calculations.
7. The method according to claim 1, It is characterized in that When the output result is obtained by performing calculation based on the attribute data record to be processed and the target calculation unit, the method further includes: Performing operation check on the execution process of the target operation unit; When an error occurs during the operation of the target operation unit, a third prompt message is generated; the third prompt message is used to prompt at least one of the operation unit information, error information, input, and output where the error occurs.
8. The method according to claim 1, It is characterized in that Methods for determining the target format include: Determining the target format based on the feature engineering configuration item, or determining the target format according to an incoming parameter request; the incoming parameter request includes the specified target format; The target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
9. The method according to claim 1, It is characterized in that The method further comprises: If an error occurs in the process of converting or outputting the target feature based on the target format, fourth prompt information is generated; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format.
10. The method according to claim 1, It is characterized in that The method further comprises: For the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature, corresponding statistical data are obtained.
11. The method according to claim 1, It is characterized in that The method further comprises: The attribute data record to be processed is obtained from a local file, a big data cluster or a network service.
12. A feature processing device, include: A parsing module, configured to generate a list of operation units based on a feature engineering configuration item; wherein the list of operation units includes at least one operation unit and its corresponding input and output; wherein the feature engineering configuration item is generated based on a predefined operation unit and a predefined available attribute; A compiling module, configured to generate an output feature list and a calculation path of the output features based on the operation unit list; wherein the operation units included in all calculation paths of the output features in the output feature list are at least part of the operation units in the operation unit list; A data acquisition module, used to determine the attribute data record to be processed and the target operation unit based on the target feature and its corresponding calculation path; wherein the target feature is the output feature in the output feature list; A feature processing module, used for performing calculation based on the attribute data record to be processed and the target calculation unit to obtain an output result, assigning the output result to the target feature, and outputting the target feature based on a target format; The compiling module is further used to generate an operation unit feature dependency graph according to the input-output dependency relationship of at least one operation unit included in the operation unit list; wherein the operation unit feature dependency graph is a directed acyclic graph; delete N operation units in the operation unit feature dependency graph to obtain an adjusted operation unit feature dependency graph; wherein the N operation units are operation units that are directly or indirectly dependent on non-final output features; N is an integer; based on the adjusted operation unit feature dependency graph, generate an output feature list and a calculation path of the output feature; The compilation module is also used to divide the adjusted operation unit feature dependency graph into at least one sub-graph according to different objects to which the attributes of each operation unit depend belong; wherein at least one sub-graph includes a first type of sub-graph and a second type of sub-graph; the outputs of at least some of the operation units in the first type of sub-graph serve as inputs of at least some of the operation units in the second type of sub-graph; based on the at least one sub-graph, generate an output feature list and a calculation path of the output features contained in the output feature list.
13. The device according to claim 12, It is characterized in that The predefined operation unit includes at least one of the following: Operation unit name, function description, input field, input field data format, output field, output field data format, operation parameters and usage examples.
14. The device according to claim 12, It is characterized in that The device also includes: The first verification module is used to perform semantic verification and / or grammatical verification on the feature engineering configuration item; if the feature engineering configuration item has a semantic verification error and / or a grammatical verification error, a first prompt message is generated; wherein the first prompt message includes at least one of the following: the location of the incorrect configuration, the cause of the error, and the type of error.
15. The device according to any one of claims 12 to 14, It is characterized in that The device also includes: The second verification module is used to verify the attribute data record to be processed and obtain a verification result; wherein the verification result is used to indicate whether the attribute data record to be processed exceeds the attribute data range, or whether the type of the attribute data record to be processed matches the output feature type; when the verification result indicates that there is an error in the attribute data record to be processed, a second prompt information is generated; wherein the second prompt information is used to indicate that an error occurs in the attribute data record to be processed, and prompts at least one of the attribute name of the error, the cause of the error and the original data.
16. The device according to claim 15, It is characterized in that The data acquisition module is used to set the attribute data to be processed as a default value when the acquired attribute data to be processed is empty.
17. The device according to claim 12, It is characterized in that The feature processing module is used to control the operation units without input-output dependency to perform parallel calculations when there are operation units without input-output dependency in the target operation units.
18. The device according to claim 12, It is characterized in that The device also includes: The third verification module is used to perform operation checks on the execution process of the target operation unit; when an error occurs in the operation process of the target operation unit, a third prompt message is generated; the third prompt message is used to prompt at least one of the operation unit information, error information, input, and output where the error occurs.
19. The device according to claim 12, It is characterized in that The feature processing module is used to determine the target format based on the feature engineering configuration item, or determine the target format according to an incoming parameter request; the incoming parameter request includes the specified target format; The target format includes at least one of the following: original feature data, a sparse format feature vector indexed by a feature name, a sparse format feature vector indexed by a hash value, a dense vector with a name, and a dense vector without a name.
20. The device according to claim 12, It is characterized in that The device also includes: The fourth verification module is used to generate fourth prompt information if an error occurs in the process of converting or outputting the target feature based on the target format; the fourth prompt information is used to prompt at least one of the error location, feature name, feature data and target format.
21. The device according to claim 12, It is characterized in that The data acquisition module is used to acquire corresponding statistical data for the operation units that need to rely on statistical data among all the operation units included in the calculation path of the output feature.
22. The device according to claim 12, It is characterized in that The data acquisition module is used to acquire the attribute data record to be processed from a local file, a big data cluster or a network service.
23. A medium storing a computer program, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
24. A computing device, include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 11.
