Method and device for constructing feature engineering computing topology graph

Abstract

The embodiment of the present invention provides a method and device for constructing a feature engineering calculation topology map, which belongs to the field of computer technology. The method includes: traversing the operator functions in the configuration file of the feature engineering in reverse order according to the order of the dependencies between the operator functions in the configuration file of the feature engineering to determine the feature dependency list to construct the feature engineering calculation topology map, wherein one of the operator functions corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and the operator function, and the feature engineering calculation topology map includes a feature dependency list, and one of the feature dependency lists is for a group of the features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations. Thereby, the performance of online feature engineering is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for constructing a feature engineering computing topology graph. Background Art

[0002] Topological graphs in mathematics and computer science are often used in real-life applications, such as common traffic route maps, task assignment, duration calculation, and aviation network problems. A topological graph has several ordered vertices, and each directed edge points from the top vertex to the bottom vertex. Through topological graphs and topological sorting, we can find the shortest path or critical path in a task or plan to guide us to make reasonable decisions and arrangements, and ultimately complete the task efficiently.

[0003] At present, deep learning models are the main means of implementing artificial intelligence technology in the industry. A complete offline deep learning model training process includes sample construction, feature engineering, model training, model evaluation and model release; when reflected in online services, it mainly involves the steps of obtaining feature data, online feature engineering, and sending it to the sorting model service for calculation and sorting. Among them, online feature engineering is the process that consumes the most online service performance. The feature data infrastructure of different Internet companies is different. In feature engineering calculations, the industry uses configuration files to obtain the features of each material to be sorted, and then performs specific feature operator calculations on each material. However, in the use scenarios of search recommendation advertising, there may be more materials to be sorted each time, for example, the total number of sorted materials exceeds 100, which is more time-consuming.

[0004] Currently, online feature engineering calculates the features of multiple materials through multi-threaded concurrent calculations. As the features required by the model become more abundant, there are more operators in the configuration file, and there is a sequential dependency relationship between the operators. Therefore, when performing feature engineering for each material, only linear calculations can be performed, that is, the operators are calculated in sequence from the first to the last according to the dependency relationship. The performance of online feature engineering is poor. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method and device for constructing a feature engineering calculation topology map, which can solve or at least partially solve the above-mentioned problems.

[0006] In order to achieve the above-mentioned purpose, one aspect of an embodiment of the present invention provides a method for constructing a feature engineering calculation topology map, the method comprising: traversing the operator functions in the configuration file of the feature engineering in reverse order according to the forward and backward dependency order between the operator functions in the configuration file of the feature engineering to determine a feature dependency linked list to construct the feature engineering calculation topology map, wherein one of the operator functions corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and the operator function, and the feature engineering calculation topology map includes a feature dependency linked list, and one of the feature dependency linked lists is for a group of the features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations, and the feature dependency linked list corresponding to a group of the features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency linked list corresponding to a single feature without corresponding calculation dependencies includes the feature information of the features.

[0007] Optionally, when traversing any of the operator functions, the following operations are performed: for any of the features corresponding to the operator function, based on the feature information of the feature, a feature dependency list corresponding to the feature is determined, the feature information of the feature is matched to the determined feature dependency list, and when there are adjacent features in the determined feature dependency list, the computational dependency relationship with the adjacent features is described; when the feature information of all the features corresponding to the operator function is matched to the feature dependency list, it is determined whether all of the operator functions in the configuration file have been traversed; and when not all of the operator functions in the configuration file have been traversed, the next operator function of the operator function is determined according to the reverse order to traverse the next operator function.

[0008] Optionally, for any of the features corresponding to the operator function, a feature dependency linked list corresponding to the feature is determined based on the feature information of the feature, the feature information of the feature is matched to the determined feature dependency linked list, and when there are adjacent features in the determined feature dependency linked list, the computational dependency relationship with the adjacent features is described, including: determining whether the feature is a necessary feature that needs to be input into the deep learning model; and in response to the feature being the necessary feature, constructing a new feature dependency linked list for the feature and placing the feature information of the feature at the starting node of the constructed feature dependency linked list; and / or in response to the feature not being the necessary feature, finding the feature dependency linked list in which the feature information of the tail node in the existing feature dependency linked list includes the same dependency feature as the feature, and adding a node at the position following the tail node of the found feature dependency linked list, placing the feature information of the feature at the added node and describing the computational dependency relationship between the added node and the nodes adjacent to the added node.

[0009] Optionally, in response to the feature not being the necessary feature, the method further includes: updating the pointing relationship between the tail node of the found feature dependency linked list and the feature included in the preset necessary feature list to a pointing relationship to the added node.

[0010] Optionally, for any feature corresponding to the operator function, the determination of whether the feature is a necessary feature that needs to be input into the deep learning model includes: comparing the feature with the necessary features included in a preset necessary feature list to determine whether the feature is the necessary feature; and / or in response to the feature not being the necessary feature, finding the feature dependency list in which the dependent feature included in the feature information of the tail node of the existing feature dependency list is the same as the feature, including: obtaining the feature information at the tail of the existing feature dependency list based on the pointing relationship between the tail node of the existing feature dependency list and the preset necessary feature list; and comparing the dependent feature in the acquired feature information with the feature, wherein the feature dependency list corresponding to the dependent feature that is the same as the feature is the feature dependency list in which the dependent feature included in the feature information at the tail of the existing feature dependency list is the same as the feature.

[0011] Correspondingly, another aspect of an embodiment of the present invention provides a device for constructing a feature engineering calculation topology map, the device comprising: a traversal module, used to traverse the operator functions in the configuration file of the feature engineering in reverse order of the forward and backward dependency order between the operator functions in the configuration file to determine a feature dependency linked list to construct the feature engineering calculation topology map, wherein one of the operator functions corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and the operator function, and the feature engineering calculation topology map includes a feature dependency linked list, one of the feature dependency linked lists is for a group of the features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations, the feature dependency linked list corresponding to a group of the features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency linked list corresponding to a single feature without corresponding calculation dependencies includes the feature information of the features.

[0012] Optionally, when traversing any of the operator functions, the following operations are performed: for any of the features corresponding to the operator function, based on the feature information of the feature, a feature dependency list corresponding to the feature is determined, the feature information of the feature is matched to the determined feature dependency list, and when there are adjacent features in the determined feature dependency list, the computational dependency relationship with the adjacent features is described; when the feature information of all the features corresponding to the operator function is matched to the feature dependency list, it is determined whether all of the operator functions in the configuration file have been traversed; and when not all of the operator functions in the configuration file have been traversed, the next operator function of the operator function is determined according to the reverse order to traverse the next operator function.

[0013] Optionally, for any of the features corresponding to the operator function, a feature dependency linked list corresponding to the feature is determined based on the feature information of the feature, the feature information of the feature is matched to the determined feature dependency linked list, and when there are adjacent features in the determined feature dependency linked list, the computational dependency relationship with the adjacent features is described, including: determining whether the feature is a necessary feature that needs to be input into the deep learning model; and in response to the feature being the necessary feature, constructing a new feature dependency linked list for the feature and placing the feature information of the feature at the starting node of the constructed feature dependency linked list; and / or in response to the feature not being the necessary feature, finding the feature dependency linked list in which the feature information of the tail node in the existing feature dependency linked list includes the same dependency feature as the feature, and adding a node at the position following the tail node of the found feature dependency linked list, placing the feature information of the feature at the added node and describing the computational dependency relationship between the added node and the nodes adjacent to the added node.

[0014] Optionally, the device further includes: an updating module for updating the pointing relationship between the tail node of the found feature dependency linked list and the preset necessary feature list to a pointing relationship to the added node in response to the feature not being the necessary feature.

[0015] Optionally, for any feature corresponding to the operator function, the determination of whether the feature is a necessary feature that needs to be input into the deep learning model includes: comparing the feature with the necessary features included in a preset necessary feature list to determine whether the feature is the necessary feature; and / or in response to the feature not being the necessary feature, finding the feature dependency list in which the dependent feature included in the feature information of the tail node of the existing feature dependency list is the same as the feature, including: obtaining the feature information at the tail of the existing feature dependency list based on the pointing relationship between the tail node of the existing feature dependency list and the preset necessary feature list; and comparing the dependent feature in the acquired feature information with the feature, wherein the feature dependency list corresponding to the dependent feature that is the same as the feature is the feature dependency list in which the dependent feature included in the feature information at the tail of the existing feature dependency list is the same as the feature.

[0016] In addition, another aspect of an embodiment of the present invention further provides a feature engineering calculation method, which performs calculations according to a feature engineering calculation topology graph constructed according to the above method.

[0017] In addition, another aspect of an embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above method.

[0018] In addition, another aspect of an embodiment of the present invention further provides a processor for running a program, wherein the program is used to execute the above method when being run.

[0019] In addition, another aspect of an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the above method when executed by a processor.

[0020] Through the above technical scheme, a feature engineering calculation topology map is constructed, and the feature engineering calculation topology map includes a feature dependency list. A feature dependency list is for a group of features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations, and the feature dependency list corresponding to a group of features with calculation dependencies includes feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without corresponding calculation dependencies includes feature information of the features. When performing feature engineering calculations, calculations are performed using the feature dependency list as a unit. In this way, by constructing a feature engineering calculation topology map, each group of features with calculation dependencies or a single feature without dependent features is calculated separately when performing feature engineering calculations, and feature dependency lists can be calculated in parallel, which solves the problem in the prior art that the operator calculations are performed in sequence from first to last according to the dependency, thereby improving the online feature engineering performance.

[0021] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0023] Figure 1 It is a schematic diagram of constructing a feature engineering calculation topology map provided by an embodiment of the present invention;

[0024] Figure 2 is a logical schematic diagram of a method for constructing a feature engineering calculation topology graph provided by another embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of a preset necessary feature list and a feature engineering calculation topology map provided by another embodiment of the present invention; and

[0026] Figure 4 It is a logical schematic diagram of a method for constructing a feature engineering calculation topology map provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0027] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0028] One aspect of an embodiment of the present invention provides a method for constructing a feature engineering calculation topology map. In an embodiment of the present invention, by constructing a feature engineering calculation topology map, the situation of calculating by relying on the sequence of dependencies between operator functions during feature engineering is converted into the situation of calculating by relying on the calculation dependency relationship between features, such as Figure 1 As shown in the figure, the feature engineering calculation topology diagram in the dashed box on the right is constructed according to the situation in the dashed box on the left. Figure 1 As shown, the dotted box on the left is the feature engineering configuration in the prior art, OP represents the operator function, each operator function corresponds to at least one feature (that is, for any operator function, at least one feature is calculated using the operator function), and there is a front-to-back dependency order between these operator functions, for example, operator function OP5 depends on operator function OP4, operator function OP4 depends on operator function OP3, operator function OP3 depends on operator function OP2, and operator function OP2 depends on operator function OP1. In the prior art, in order not to destroy the dependency relationship between operator functions, linear sequential calculation is adopted, that is, the calculation is performed in the order of operator functions OP1, OP2, OP3, OP4, and OP5. In an embodiment of the present invention, the feature engineering configuration is improved using a topological graph-like structure, and the features corresponding to each operator function are separated out individually as units, and then the topological relationship graph is reconstructed according to the dependency relationship of the original left graph, that is, a feature engineering calculation topology graph is constructed, wherein the constructed feature engineering calculation topology graph can refer to the following. Figure 1 As shown in the dashed box on the right side of . The feature engineering calculation topology diagram includes at least one feature dependency list, and a feature dependency list is for a group of features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations. Figure 1 As shown, features F1, F7 and F11 have a dependency relationship, that is, feature F7 depends on feature F1, feature F11 depends on feature F7, and a feature dependency list is constructed for features F1, F7 and F11; features F8 and F13 have a dependency relationship, feature F13 depends on feature F8, and a feature dependency list is constructed for features F8 and F13; feature F6 has no corresponding computational dependency relationship, and a feature dependency list is constructed for feature F6, and so on. In addition, a feature dependency list for a group of features with computational dependencies includes feature information of corresponding features and describes the computational dependencies between features. The feature information of a feature includes the feature, dependent features and operator functions, wherein for a feature, the dependent features are combined with the operator function corresponding to the feature to obtain the feature. As shown in FIG. Figure 1 As shown, for the feature dependency linked lists corresponding to features F1, F7 and F11, each node includes the feature information of the feature corresponding to the node ( Figure 1The feature information is not fully displayed in the figure). In addition, the computational dependency describes the dependency and computation order between adjacent features when performing feature engineering. The computational dependency expresses the computation order between adjacent features when performing feature engineering, which is to first compute the dependent features and then the features that depend on other features. For example, Figure 1 The feature F1 and feature F7 shown in the figure use arrows to describe the computational dependency between adjacent features. The computational dependency expresses that feature F7 depends on feature F1 and that feature F7 is executed after feature F1 is executed during feature engineering. It should be noted that although for a set of features with computational dependencies, the feature that is first calculated does not depend on the features in the configuration file during feature engineering, it also has a computational dependency with the features in the configuration file because the calculation of other features depends on it. For example, Figure 1 As shown in the figure, for the feature dependency lists corresponding to features F1, F7, and F11, when performing feature engineering, feature F1 is calculated first. F1 does not depend on the features in the configuration file, and is obtained based on the dependent features and operator functions in its feature configuration information. However, there is a computational dependency relationship between feature F1 and feature F7, because feature F7 depends on feature F1. For feature F6, there is no computational dependency, because feature F6 does not depend on the features in the configuration file, and feature F6 is not the calculation of other features in the configuration file and does not depend on it. In addition, for example, Figure 1 The feature engineering calculation topology diagram shown in the dotted box on the right side, when performing feature engineering calculations, for a set of features with calculation dependencies corresponding to the feature dependency list, the feature starting with the arrow is the head of the feature dependency list when performing feature engineering calculations, and the feature the arrow finally points to is the tail of the feature dependency list when performing feature engineering calculations. For example, for the feature dependency list corresponding to features F1, F7 and F11, feature F1 is the head when performing feature engineering calculations, and feature F11 is the tail when performing feature engineering calculations.

[0029] In the feature dependency list, pointers can be used to describe the computational dependencies between adjacent features, for example Figure 1 As shown in , arrows are used to express the computational dependency relationship described by pointers between adjacent features. In addition, other forms can be used to express the computational dependency relationship between adjacent features in the feature dependency list. Figure 1 Therefore, when performing feature engineering calculations according to the feature engineering calculation topology diagram shown in the dotted box on the right, the seven feature dependency lists are calculated in parallel without affecting each other to complete the entire feature engineering calculation. In addition, it should be noted that Figure 1The dashed box on the right shows the constructed feature engineering calculation topology diagram. However, when constructing the feature engineering calculation topology diagram, Figure 1 For any feature dependency list shown in , when constructing the feature dependency list, the construction starts from the last feature determined by the arrow. For example, for the feature dependency list corresponding to features F1, F7, and F11, the construction starts from feature F11, but when performing feature engineering, it starts from feature F1.

[0030] Specifically, in an embodiment of the present invention, a method for constructing a feature engineering calculation topology map includes the following contents. According to the reverse order of the forward and backward dependency order between the operator functions in the feature engineering configuration file, the operator functions in the configuration file are traversed to determine the feature dependency list to construct the feature engineering calculation topology map, wherein one operator function corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and the operator function. The feature engineering calculation topology map includes a feature dependency list, and a feature dependency list is for a group of features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations. The feature dependency list corresponding to a group of features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without corresponding calculation dependencies includes the feature information of the features.

[0031] Optionally, in an embodiment of the present invention, when traversing any operator function, the following operations are performed: for any feature corresponding to the operator function, based on the feature information of the feature, a feature dependency list corresponding to the feature is determined, the feature information of the feature is matched to the determined feature dependency list, and when there are adjacent features in the determined feature dependency list, the computational dependency relationship with the adjacent features is described; when the feature information of all features corresponding to the operator function is matched to the feature dependency list, it is determined whether all operator functions in the configuration file are traversed; and when not all operator functions in the configuration file are traversed, the next operator function of the operator function is determined in reverse order to traverse the next operator function.

[0032] Figure 2 This is a logical diagram of a method for constructing a feature engineering calculation topology map provided by an embodiment of the present invention. Figure 2 The method for constructing a feature engineering calculation topology graph provided by an embodiment of the present invention is exemplarily introduced.

[0033] According to the reverse order of the front-to-back dependency order between operator functions in the configuration file of feature engineering, starting from the last operator function in the front-to-back dependency order, the following traversal operation is performed. Specifically, for an operator function, the feature information of all features corresponding to the operator function is matched to the feature dependency list. For example, for any feature corresponding to the operator function, the feature dependency list corresponding to the feature is determined according to the feature information of the feature, the feature information of the feature is matched to the determined feature dependency list, and the computational dependency relationship with the adjacent features is described when there are adjacent features in the determined feature dependency list. For example, if there are adjacent features adjacent to the feature in the determined feature dependency list, a pointer of the feature pointing to the adjacent feature is established to describe the computational dependency relationship between the adjacent feature and the feature. When the feature information of all features corresponding to the operator function is matched to the feature dependency list, it is determined whether all operator functions in the configuration file are traversed. If so, the construction of the feature engineering computational topology map is completed; if not, the next operator function is determined according to the reverse order of the front-to-back dependency order between the operator functions in the configuration file. The traversal operation is repeatedly performed for the next determined operator function until all operator functions in the configuration file are traversed, completing the construction of the feature engineering calculation topology graph.

[0034] Through the above technical scheme, a feature engineering calculation topology map is constructed, and the feature engineering calculation topology map includes a feature dependency list. A feature dependency list is for a group of features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations, and the feature dependency list corresponding to a group of features with calculation dependencies includes feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without corresponding calculation dependencies includes feature information of the features. When performing feature engineering calculations, calculations are performed using the feature dependency list as a unit. In this way, by constructing a feature engineering calculation topology map, each group of features with calculation dependencies or a single feature without dependent features is calculated separately when performing feature engineering calculations, and feature dependency lists can be calculated in parallel, which solves the problem in the prior art that the operator calculations are performed in sequence from first to last according to the dependency, thereby improving the online feature engineering performance.

[0035] Optionally, in an embodiment of the present invention, for any feature corresponding to an operator function, a feature dependency list corresponding to the feature is determined based on the feature information of the feature, the feature information of the feature is matched to the determined feature dependency list, and when there are adjacent features in the determined feature dependency list, the computational dependency relationship with the adjacent features is described, which may include the following content. Determine whether the feature is a necessary feature that needs to be input into the deep learning model, wherein the necessary feature is a dependent feature that is finally input into the deep learning model. Figure 1As shown, in the feature engineering calculation topology diagram shown in the dotted box on the right, feature F11, feature F13, feature F4, feature F5, feature F6, feature F14 and feature F12 are features that need to be input into the deep learning model and are necessary features; feature F1, feature F7, feature F8, feature F2, feature F3, feature F9 and feature F10 do not need to be input into the deep learning model and are non-essential features. If the feature is a necessary feature, then the feature dependency list for the feature has not been constructed at this time, and a new feature dependency list needs to be constructed for the feature, and the newly constructed feature dependency list is the feature dependency list corresponding to the feature. If the feature is not a necessary feature, then there is already a feature dependency list corresponding to the feature, wherein the feature dependency list corresponding to the feature satisfies the following conditions: the dependent features in the feature information placed at the end of the feature dependency list are the same as the feature. In the case where the feature is a necessary feature, a new feature dependency list is constructed for the feature and the feature information of the feature is placed at the starting node of the constructed feature dependency list; and / or in the case where the feature is not a necessary feature, a feature dependency list with the same dependency feature as the feature included in the feature information at the tail of the existing feature dependency list is found, and a node is added at the position following the tail node of the found feature dependency list, the feature information of the feature is placed at the added node and the computational dependency relationship between the added node and the nodes adjacent to the added node is described, that is, the computational dependency relationship between the feature and the adjacent features is described. Among them, describing the computational dependency relationship between the added node and the nodes adjacent to the added node can be to establish a pointer from the added node to the node adjacent to the added node. It should be noted that for an existing feature dependency list that only includes one necessary feature, the starting node is the tail node. In addition, the feature information at the tail of the feature dependency list with the new node added is updated to the feature information placed at the newly added node.

[0036] Optionally, in an embodiment of the present invention, in response to the feature not being a necessary feature, the method further includes: updating the pointing relationship between the tail node of the found feature dependency linked list and the preset necessary feature list to a pointing relationship to the added node.

[0037] Optionally, in an embodiment of the present invention, for any feature corresponding to an operator function, determining whether the feature is a necessary feature that needs to be input into the deep learning model includes: comparing the feature with the necessary features included in a preset necessary feature list to determine whether the feature is a necessary feature. The necessary features included in the configuration file of feature engineering are pre-filled in the preset necessary feature list to initialize the preset necessary feature list, and when determining whether a feature is a necessary feature, the feature is compared with the necessary features included in the preset necessary feature list, and whether the feature is a necessary feature is determined by determining whether there is a necessary feature in the preset necessary feature list that is the same as the feature. In addition, for any feature corresponding to an operator function, in response to the feature not being a necessary feature, finding a feature dependency list whose dependency features included in the feature information at the tail of an existing feature dependency list are the same as the feature may include the following content. Pre-filling the pointing relationship between the tail node of the existing feature dependency list and the tail node of the feature dependency list in the preset necessary feature list, wherein, for a feature dependency list, the tail node of the feature dependency list can be found according to the pointing relationship. According to the pointing relationship between the tail node of the existing feature dependency list and the tail node of the existing feature dependency list included in the preset necessary feature list, the feature information of the tail node is obtained. Specifically, according to a pointing relationship with the tail node, the tail node is found, and the feature information placed in the tail node is taken out. It should be noted that if the preset necessary feature list includes several pointing relationships, the feature information of several tail nodes is obtained. The dependent features in the acquired feature information are compared with the feature, wherein the feature dependency chain list corresponding to the dependent features identical to the feature is the feature dependency chain list in which the dependent features identical to the feature are included in the feature information of the tail node in the existing feature dependency chain list, wherein the feature dependency chain list in which the feature information identical to the feature is placed in the tail node is the feature dependency chain list corresponding to the dependent features identical to the feature. In addition, in the preset necessary feature list, the filled-in pointing relationship can be associated with the corresponding necessary feature. For a feature dependency chain list, the feature information placed at the starting node is the feature information of the corresponding necessary feature, that is, the necessary feature has a one-to-one correspondence with the feature dependency chain list, and one necessary feature corresponds to one feature dependency chain list. In the case where a feature dependency chain list has been constructed for a necessary feature, the necessary feature and the pointing relationship of the tail node pointing to the feature dependency chain list corresponding to the necessary feature can be associated together. For example, the preset necessary feature list includes necessary feature units, and a necessary feature and a pointing relationship pointing to the tail node of the feature dependency list corresponding to the necessary feature are filled in a necessary feature unit, and different necessary feature units correspond to different necessary features. In addition, the necessary feature unit may also include a pointing relationship pointing to the start node of the feature dependency list corresponding to the necessary feature. Figure 3As shown, the left dotted box is a preset necessary feature list, which includes multiple necessary feature units. Figure 3 Two essential feature units are shown in FIG. 1 , each of which includes an essential feature, a pointing relationship pointing to the tail node of the feature dependency list, and a pointing relationship pointing to the starting node of the feature dependency list; the dotted box on the right is a computational topology diagram constructed according to the embodiment provided by the present application. It should be noted that: Figure 3 The content in is only an example of a feature dependency list and an exemplary description of the pointing relationship between the starting node and the tail node of the feature dependency list. Figure 3 In , the feature information placed at each feature includes the feature, operator function, and dependent features. If the calculated feature depends on other features, the calculation dependency relationship with the next feature will continue to be described. Because when reading the feature engineering operator configuration file, the dependency relationship is read in reverse order, and parallel calculation is sequential calculation, so the feature will record the calculation dependency relationship between the calculated features on which it depends. In addition, when a node is added to the position of the tail node following an existing feature dependency list, the pointing relationship pointing to the tail node of the feature dependency list is updated to point to the newly added node. In addition, if Figure 3 As shown, feature dFn1 corresponds to necessary feature F1, and feature information of necessary feature F1 is placed at the position of feature dFn1... Feature dFnn corresponds to necessary feature Fn, and feature information of necessary feature Fn is placed at the position of feature dFnn.

[0038] Figure 4 This is a logical diagram of a method for constructing a feature engineering calculation topology map provided by another embodiment of the present invention. Figure 4 The method for constructing a feature engineering calculation topology graph provided in an embodiment of the present invention is exemplarily introduced.

[0039] Initialize the preset necessary feature list. Pre-fill the necessary features included in the configuration file of the feature engineering in the preset necessary feature list to initialize the preset necessary feature list. After initializing the preset necessary feature list, traverse all operator functions in the configuration file. Among them, the following content can be included when traversing any operator function. Read each feature that needs to be calculated corresponding to the operator function. Traverse the tail nodes of the preset necessary feature list and the existing feature dependency list in turn to determine the feature dependency list corresponding to the read feature. Among them, no matter how many features the operator function being traversed corresponds to at this time, it is necessary to determine the corresponding feature dependency list for each feature, wherein how to determine the corresponding feature dependency list can refer to the content described in the above embodiment. Match the feature information of the read feature to the determined feature dependency list and use pointers to describe the calculation dependency relationship between adjacent features if there are adjacent features. It should be noted that for each feature corresponding to the operator function being traversed at this time, the relevant content of matching its feature information to the feature dependency list is executed. Among them, generally, when a feature is a necessary feature, the corresponding feature dependency list determined is a newly established feature dependency list, which does not have adjacent features; when a feature is not a necessary feature, the corresponding feature dependency list determined is an existing feature dependency list, which has adjacent features in the determined feature dependency list. At this time, all the features corresponding to the operator function being traversed have completed matching the feature information to the feature dependency list, and it is determined whether all the operator functions in the configuration file of the feature engineering are traversed. If so, it ends and the construction of the feature engineering calculation topology map is completed; if not, the next operator function of the operator function being traversed at this time is determined according to the reverse order of the front and back dependency order between the operator functions in the configuration file, and the above convenient operation is repeated until all the operator functions in the configuration file are traversed. In addition, in an embodiment of the present invention, when reading the feature that needs to be calculated corresponding to the operator function each time, if the feature that the operator function calculation depends on has a feature_process prefix, then the feature will depend on other features. If it does not have a feature_process field, then the feature dependency list corresponding to the feature has ended, and the feature dependency list corresponding to the feature has been constructed.

[0040] Because the number of features with dependencies between operators after completing a feature engineering operation is relatively much smaller than the total number of features, in the embodiment of the present invention, an adjacency linked list data structure is innovatively designed in the process of constructing a topological feature engineering execution flow to record the dependency relationship in feature calculation. The data structure diagram of the adjacency linked list is shown in FIG. Figure 3 As shown in the dashed box, the feature engineering calculation topology diagram is constructed.

[0041] In the prior art, online feature engineering must be linearly calculated because feature engineering is relatively complex, there are many operator functions, and there are sequential dependencies. In an embodiment of the present invention, a topological structure such as a topological graph is applied to the online feature engineering operator calculation, the dependency between the operator functions is fully explored, and the feature corresponding to each operator function is used as a node in the topological graph. According to the existing configuration file, the feature engineering calculation topological graph is reshaped, the feature calculation key path is combed, and the features that can be calculated in parallel are calculated as much as possible in parallel, thereby improving the performance of online feature engineering. According to the technical solution provided by the embodiment of the present invention, when a sorting request arrives, after obtaining various types of features, concurrent feature engineering is performed for multiple materials. In the feature engineering process of each material, according to the operator configuration file of the topological structure, the features with the dependency order in front or the features without dependency are first calculated concurrently and simultaneously, and then according to the order of the topological graph, the features behind are calculated in sequence. The feature dependency linked lists are calculated in parallel without affecting each other, and the calculation of the entire feature engineering is finally completed, thereby improving the performance of the entire sorting service.

[0042] Accordingly, another aspect of an embodiment of the present invention provides a device for constructing a feature engineering calculation topology map. The device includes a traversal module. The traversal module is used to determine a feature dependency list according to the reverse order of the forward and backward dependency order between the operator functions in the configuration file of the feature engineering to construct a feature engineering calculation topology map, wherein an operator function corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and operator function. The feature engineering calculation topology map includes a feature dependency list, and a feature dependency list is for a group of features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations. The feature dependency list corresponding to a group of features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without corresponding calculation dependencies includes the feature information of the features.

[0043] Optionally, in an embodiment of the present invention, when traversing any operator function, the following operations are performed: for any feature corresponding to the operator function, based on the feature information of the feature, a feature dependency list corresponding to the feature is determined, the feature information of the feature is matched to the determined feature dependency list, and when there are adjacent features in the determined feature dependency list, the computational dependency relationship with the adjacent features is described; when the feature information of all features corresponding to the operator function is matched to the feature dependency list, it is determined whether all operator functions in the configuration file are traversed; and when not all operator functions in the configuration file are traversed, the next operator function of the operator function is determined in reverse order to traverse the next operator function.

[0044] Optionally, in an embodiment of the present invention, for any feature corresponding to the operator function, a feature dependency linked list corresponding to the feature is determined according to the feature information of the feature, the feature information of the feature is matched to the determined feature dependency linked list, and when there are adjacent features in the determined feature dependency linked list, the computational dependency relationship with the adjacent features is described, including: determining whether the feature is a necessary feature that needs to be input into the deep learning model; and in response to the feature being the necessary feature, constructing a new feature dependency linked list for the feature and placing the feature information of the feature at the starting node of the constructed feature dependency linked list; and / or in response to the feature not being a necessary feature, finding a feature dependency linked list in which the feature information of the tail node in the existing feature dependency linked list includes the same dependency feature as the feature, and adding a node at the position following the tail node of the found feature dependency linked list, placing the feature information of the feature at the added node and describing the computational dependency relationship between the added node and the nodes adjacent to the added node.

[0045] Optionally, in an embodiment of the present invention, the device also includes: an updating module, for updating the pointing relationship between the tail node of the found feature dependency list and the preset necessary feature list to a pointing relationship to the added node in response to the feature not being a necessary feature.

[0046] Optionally, in an embodiment of the present invention, for any feature corresponding to the operator function, determining whether the feature is a necessary feature that needs to be input into the deep learning model includes: comparing the feature with the necessary features included in a preset necessary feature list to determine whether the feature is a necessary feature; and / or in response to the feature not being a necessary feature, finding a feature dependency list in which the dependency features included in the feature information of the tail node in an existing feature dependency list are the same as the feature, including: obtaining feature information at the tail of the existing feature dependency list based on a pointing relationship between the tail node included in the preset necessary feature list and the existing feature dependency list; and comparing the dependency features in the obtained feature information with the feature, wherein the feature dependency list corresponding to the dependency features that are the same as the feature is the feature dependency list in which the dependency features included in the feature information at the tail of the existing feature dependency list are the same as the feature.

[0047] The specific working principle and benefits of the device for constructing a feature engineering calculation topology map provided by an embodiment of the present invention are similar to the specific working principle and benefits of the method for constructing a feature engineering calculation topology map provided by an embodiment of the present invention, and will not be repeated here.

[0048] The device for constructing a feature engineering calculation topology map includes a processor and a memory. The above-mentioned traversal modules, etc. are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions.

[0049] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the performance of feature engineering can be improved by adjusting the kernel parameters.

[0050] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0051] In addition, another aspect of an embodiment of the present invention further provides a feature engineering calculation method, which performs calculations according to a feature engineering calculation topology map constructed according to the method for constructing a feature engineering calculation topology map described in the above embodiment.

[0052] In addition, another aspect of an embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the method for constructing a feature engineering calculation topology map described in the above embodiment.

[0053] In addition, another aspect of an embodiment of the present invention further provides a processor for running a program, wherein the program, when run, is used to execute the method for constructing a feature engineering calculation topology graph described in the above embodiment.

[0054] In addition, another aspect of the embodiment of the present invention further provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the method for constructing a feature engineering calculation topology map described in the above embodiment when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0055] In addition, another aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for constructing a feature engineering calculation topology graph described in the above embodiment.

[0056] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0057] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0060] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0061] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0062] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0063] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0064] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for constructing a feature engineering computing topology graph, characterized in that: The method includes: According to the reverse order of the forward and backward dependency order between the operator functions in the configuration file of the feature engineering, the operator functions in the configuration file are traversed to determine the feature dependency list to construct the feature engineering calculation topology map, wherein one of the operator functions corresponds to at least one feature, and the feature information of any feature includes the feature, the dependent features and the operator function. The feature engineering calculation topology map includes a feature dependency list, and one of the feature dependency lists is for a group of the features with calculation dependencies or a single feature without the corresponding calculation dependencies when performing feature engineering calculations. The feature dependency list corresponding to a group of the features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without the corresponding calculation dependencies includes the feature information of the features. When performing feature engineering calculations, feature dependency lists are used as units for calculation, and feature dependency lists can be calculated in parallel.

2. The method according to claim 1, characterized in that When traversing any of the operator functions, the following operations are performed: For any of the features corresponding to the operator function, determine a feature dependency linked list corresponding to the feature according to feature information of the feature, match the feature information of the feature to the determined feature dependency linked list, and describe the computational dependency relationship with adjacent features if there are adjacent features in the determined feature dependency linked list; In the case where the feature information of all the features corresponding to the operator function matches the feature dependency linked list, determining whether all the operator functions in the configuration file are traversed; and In the case that all the operator functions in the configuration file have not been traversed, the next operator function of the operator function is determined according to the reverse order to traverse the next operator function.

3. The method according to claim 2, characterized in that For any of the features corresponding to the operator function, determining a feature dependency linked list corresponding to the feature according to feature information of the feature, matching the feature information of the feature to the determined feature dependency linked list, and describing the computational dependency relationship with the adjacent features if there are adjacent features in the determined feature dependency linked list, including: Determine whether the feature is a necessary feature that needs to be input into the deep learning model; and In response to the feature being the necessary feature, constructing a new feature dependency linked list for the feature and placing the feature information of the feature at the starting node of the constructed feature dependency linked list; and / or In response to the fact that the feature is not the necessary feature, find the feature dependency linked list in which the feature information of the tail node of the existing feature dependency linked list includes the same dependency feature as the feature, add a node at the position following the tail node of the found feature dependency linked list, place the feature information of the feature at the added node and describe the computational dependency relationship between the added node and the nodes adjacent to the added node.

4. The method according to claim 3, characterized in that In response to the feature not being the required feature, the method further comprises: The pointing relationship between the tail node of the found feature dependency linked list and the preset necessary feature list is updated to a pointing relationship to the added node.

5. The method according to claim 3, characterized in that: For any of the above features corresponding to the operator function, The determining whether the feature is a necessary feature that needs to be input into the deep learning model includes: comparing the feature with the necessary features included in a preset necessary feature list to determine whether the feature is the necessary feature; and / or In response to the feature not being the necessary feature, finding the feature dependency linked list in which the dependency feature included in the feature information of the tail node in the existing feature dependency linked list is the same as the feature comprises: According to the pointing relationship between the tail node of the existing feature dependency linked list and the preset necessary feature list, the feature information at the tail of the existing feature dependency linked list is obtained; and The dependent feature in the acquired feature information is compared with the feature, wherein the feature dependency linked list corresponding to the dependent feature identical to the feature is the feature dependency linked list in which the dependent feature included in the feature information at the tail of the existing feature dependency linked list is identical to the feature.

6. A device for constructing a feature engineering computing topology graph, characterized in that: The device includes: A traversal module is used to traverse the operator functions in the configuration file of the feature engineering in reverse order of the forward and backward dependency order between the operator functions in the configuration file of the feature engineering to determine a feature dependency list to construct the feature engineering calculation topology map, wherein one of the operator functions corresponds to at least one feature, and the feature information of any feature includes the feature, dependent features and the operator function. The feature engineering calculation topology map includes a feature dependency list, and one of the feature dependency lists is for a group of the features with calculation dependencies or a single feature without corresponding calculation dependencies when performing feature engineering calculations. The feature dependency list corresponding to a group of the features with calculation dependencies includes the feature information of the features and describes the calculation dependencies, and the feature dependency list corresponding to a single feature without corresponding calculation dependencies includes the feature information of the features. When performing feature engineering calculations, feature dependency lists are used as units for calculation, and feature dependency lists can be calculated in parallel.

7. A feature engineering calculation method, characterized in that: The feature engineering calculation method performs calculations according to a feature engineering calculation topology graph constructed according to any one of claims 1-5.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the method according to any one of claims 1 to 5.

9. A processor, characterized in that: Used to run a program, wherein the program is used to execute the method according to any one of claims 1 to 5 when run.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images