Feature engineering and graph network generation method and device, distributed system

By using directed acyclic graph network to manage feature dependencies in distributed systems, the feature engineering is solved, and the problem of high complexity and difficulty in deep feature extraction in distributed systems is achieved, and efficient and accurate feature extraction is achieved and transplant.

CN113255933BActive Publication Date: 2024-11-26SHANGHAI SENSETIME TECH DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110609583.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-01
Publication Date
2024-11-26
Estimated Expiration
2041-06-01

AI Technical Summary

Technical Problem

The existing feature engineering technology is complex and time-consuming, especially when transplanted in distributed systems, and it is easy to make mistakes. It is not efficient to build shallow features, making it difficult to achieve deep feature extraction.

Method used

Adopt the dependency and generation relationship between directed acyclic graph network management features, and pre-stored on distributed system nodes. Only feature extraction rules are retained, and specific operation methods and processing resources are not included. Each layer of the graph network is generated through iteratively. , automate feature engineering, reduce redundant features and filter important features.

Benefits of technology

It reduces the complexity and time-consuming of transplanting feature engineering between nodes in distributed system, reduces the probability of errors, realizes seamless switching of production environments, improves the efficiency and accuracy of feature engineering, and can extract deep features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113255933B_ABST
    Figure CN113255933B_ABST
Patent Text Reader

Abstract

The disclosed embodiments provide a feature engineering and graph network generation method and device, and a distributed system. The feature engineering method is applied to a slave node of a distributed system, and the method comprises: receiving first data to be processed sent by a master node of the distributed system; performing feature extraction on the first data to be processed based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph; and the first target feature is returned to the master node; wherein each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of feature engineering technology, and in particular to feature engineering and graph network generation methods and devices, and distributed systems. Background Art

[0002] Feature engineering is an extremely important part of machine learning and deep learning. However, in practice, feature engineering is very complex and time-consuming, and even requires very professional expert knowledge. Therefore, a good feature engineering is often the longest part of the entire project and requires a lot of experiments and verification. At present, most feature engineering methods are still in the process of building shallow features and constantly performing random batch searches. The efficiency of building features is not very high; and when transplanting feature engineering to nodes in distributed systems, it is very time-consuming and error-prone. Summary of the invention

[0003] The present disclosure provides a feature engineering and graph network generation method and device, and a distributed system to solve at least part of the above technical problems.

[0004] According to a first aspect of an embodiment of the present disclosure, a feature engineering method is provided, which is applied to a slave node of a distributed system, and the method comprises: receiving first data to be processed sent by a master node of the distributed system; performing feature extraction on the first data to be processed based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph; the first target feature is returned to the master node; each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule.

[0005] The disclosed embodiment manages the dependencies and generation relationships between features through a directed acyclic graph, and then pre-stores the graph network on the nodes of the distributed system. Since the graph network only retains the logic of generating features (i.e., feature extraction rules) but does not include the specific operation method and processing resources for executing the logic, when the graph network is transplanted to each node of the distributed system, there is no need to define the specific operation method and processing resources used by each node in the distributed system, thereby reducing the complexity of transplanting the graph network between nodes of the distributed system, reducing the time consumed in the transplantation process, and reducing the probability of errors, thereby achieving seamless switching of the production environment and switching of the computing engine.

[0006] In some embodiments, the method further includes: obtaining features corresponding to leaf nodes of the original graph network; extracting features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; updating the original graph network based on features corresponding to leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; using the updated graph network as the original graph network, and returning to the step of obtaining features corresponding to leaf nodes of the original graph network, and determining the updated graph network that meets preset conditions as the graph network. The disclosed embodiment generates nodes and edges at each layer in the graph network in an iterative manner, and has low complexity.

[0007] In some embodiments, after obtaining the new feature, the method further includes: randomly generating feature values ​​for features corresponding to the first-layer nodes in the update graph network; obtaining feature values ​​corresponding to other nodes in the update graph network except the first-layer nodes based on the feature values ​​corresponding to the first-layer nodes; comparing the feature values ​​of the new feature with the feature values ​​of other features, the nodes corresponding to the other features are nodes in the update graph network except the nodes corresponding to the new feature; in the presence of other features having the same feature values ​​as the new feature, deleting the new feature and the feature extraction rules used to generate the new feature from the update graph network. This embodiment removes redundant features by deleting features with the same feature values, thereby reducing the number of features and avoiding excessive growth in data processing volume as the number of iterations increases due to an excessive number of features.

[0008] In some embodiments, after obtaining the new feature, the method further includes: filtering out a second target feature from the new feature through a pre-trained tree model; the importance parameter of the second target feature is lower than a preset importance parameter threshold, and / or the proportion of the second target feature is greater than or equal to a preset proportion, and the importance parameter of the second target feature is lower than the importance parameters of other features other than the second target feature. This embodiment reduces the number of features by deleting features with smaller importance parameters, thereby avoiding excessive growth of data processing volume as the number of iterations increases due to an excessive number of features.

[0009] In some embodiments, the features corresponding to the nodes of the first layer of the graph network are obtained by extracting features from the second data to be processed; the method further includes: extracting a first original feature from the first data to be processed; determining the statistical features of the first original feature; and outputting an alarm message when the statistical features of the second original feature extracted from the second data to be processed do not match the statistical features of the first original feature. This embodiment outputs an alarm message based on the statistical features of the data, so that the user can take timely processing measures when the statistical features of the data actually processed do not match the statistical features of the training data for generating the graph network, for example, updating the statistical features of the second original feature, thereby reducing errors caused by data mismatch.

[0010] In some embodiments, the determining of the statistical characteristics of the first original feature includes: determining the category of the first original feature; and determining the statistical characteristics of the first original feature based on the category of the first original feature. First original features of different categories have different statistical characteristics. For example, the maximum value of the first original feature of the age category is about 100, and the minimum value is 0; the minimum value of the first original feature of the shopping amount category is 0, and the maximum value can be several thousand or tens of thousands. Determining the statistical characteristics of the first original feature based on its category has high accuracy and low operational complexity.

[0011] In some embodiments, the category of the first original feature is a numerical category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of at least one feature value of the first original feature; or the category of the first original feature is a non-numerical character category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of the length of the characters included in at least one feature value of the first original feature. In the above manner, statistical features can be determined for first original features of various categories such as numbers, Chinese characters, and symbols.

[0012] In some embodiments, the features corresponding to the first-layer nodes of the graph network are obtained by extracting features from the second data to be processed; the second data to be processed includes at least two groups of structured data; the method further includes: merging the at least two groups of structured data by primary key to obtain merged group structured data; extracting features from the merged group structured data to obtain the features corresponding to the first-layer nodes of the graph network. Through the multi-table merging function module, the data of multiple tables can be converted into single-table data, thereby facilitating processing.

[0013] According to a second aspect of an embodiment of the present disclosure, a graph network generation method is provided, wherein the graph network is used to extract features from first data to be processed, and the method comprises: obtaining features corresponding to leaf nodes of an original graph network, and extracting features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, and each node of the graph network is used to record information about a feature, and each edge of the graph network is used to record information about a feature extraction rule; the original graph network is updated based on the features corresponding to leaf nodes of the original graph network, the new features, and the feature extraction rules to obtain an updated graph network; determining feature values ​​corresponding to each node in the updated graph network, and extracting features from leaf nodes in the updated graph network. When the feature value corresponding to a node is the same as the feature value corresponding to a non-leaf node in the updated graph network, the leaf node and the edges connecting the leaf nodes are deleted from the updated graph network to obtain a first optimized graph network; a second target feature is screened out from the new feature by a pre-trained tree model; the importance of the second target feature is lower than the importance of features other than the second target feature in the new feature, and the nodes corresponding to the second target feature and the edges connecting the nodes corresponding to the second target feature are deleted from the first optimized graph network to obtain a second optimized graph network; the second optimized graph network is used as the original graph network, and the step of obtaining the features corresponding to the leaf nodes of the original graph network is returned, and the second optimized graph network that meets the preset conditions is determined as the graph network.

[0014] The disclosed embodiment manages the dependency and generation relationship between features through a directed acyclic graph, generates each layer of nodes of the graph network in an iterative manner, realizes automated feature engineering without the need for human intervention, and improves the efficiency of feature engineering; when constructing each layer of nodes of the graph network, prunes the nodes and / or edges on at least two paths that obtain the same feature value, and screens the features corresponding to each layer of newly generated nodes through a neural network, thereby avoiding the number of features growing too fast as the number of iterations increases, thereby being able to extract deep features by increasing the number of iterations.

[0015] According to a third aspect of an embodiment of the present disclosure, a distributed system is provided, the system comprising: a master node, for distributing data to be processed to each slave node for processing; and at least one slave node, each slave node having a graph network pre-deployed thereon, for performing feature extraction on the data to be processed based on the graph network, obtaining a first target feature, and returning the first target feature to the master node; the graph network is a directed acyclic graph; each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule.

[0016] The disclosed embodiment manages the dependencies and generation relationships between features through a directed acyclic graph, and then pre-stores the graph network on the nodes of the distributed system. Since the graph network only retains the logic of generating features (i.e., feature extraction rules) but does not include the specific operation method and processing resources for executing the logic, when the graph network is transplanted to each node of the distributed system, there is no need to define the specific operation method and processing resources used by each node in the distributed system, thereby reducing the complexity of transplanting the graph network between nodes of the distributed system, reducing the time consumed in the transplantation process, and reducing the probability of errors, thereby achieving seamless switching of the production environment and switching of the computing engine.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a feature engineering device is provided, which is applied to a slave node of a distributed system, and the device includes: a receiving module, which is used to receive first data to be processed sent by a master node of the distributed system; a first feature extraction module, which is used to extract features of the first data to be processed based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph; a first return module, which is used to return the first target feature to the master node; each node of the graph network is used to record information about a feature, and each edge of the graph network is used to record information about a feature extraction rule.

[0018] The disclosed embodiment manages the dependencies and generation relationships between features through a directed acyclic graph, and then pre-stores the graph network on the nodes of the distributed system. Since the graph network only retains the logic of generating features (i.e., feature extraction rules) but does not include the specific operation method and processing resources for executing the logic, when the graph network is transplanted to each node of the distributed system, there is no need to define the specific operation method and processing resources used by each node in the distributed system, thereby reducing the complexity of transplanting the graph network between nodes of the distributed system, reducing the time consumed in the transplantation process, and reducing the probability of errors, thereby achieving seamless switching of the production environment and switching of the computing engine.

[0019] In some embodiments, the device further includes: a feature acquisition module for acquiring features corresponding to leaf nodes of the original graph network; a third feature extraction module for extracting features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; an updated graph network acquisition module for updating the original graph network based on features corresponding to leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; using the updated graph network as the original graph network, and returning to execute the function of the feature acquisition module to determine the updated graph network that meets preset conditions as the graph network. The disclosed embodiment generates nodes and edges at each layer in the graph network in an iterative manner, and has low complexity.

[0020] In some embodiments, after obtaining the new feature, the device further includes: a first feature value acquisition module, which is used to randomly generate feature values ​​for features corresponding to the first-layer nodes in the update graph network; a second feature value acquisition module, which is used to obtain feature values ​​corresponding to other nodes in the update graph network except the first-layer nodes based on the feature values ​​corresponding to the first-layer nodes; a comparison module, which is used to compare the feature value of the new feature with the feature values ​​of other features, and the nodes corresponding to the other features are nodes in the update graph network except the nodes corresponding to the new feature; a deletion module, which is used to delete the new feature and the feature extraction rules used to generate the new feature from the update graph network when there are other features with the same feature value as the new feature. This embodiment removes redundant features by deleting features with the same feature value, thereby reducing the number of features and avoiding excessive growth of data processing volume with the increase of iteration times due to excessive number of features.

[0021] In some embodiments, after obtaining the new feature, the device further includes: a screening module for screening out a second target feature from the new feature through a pre-trained tree model; the importance parameter of the second target feature is lower than a preset importance parameter threshold, and / or the proportion of the second target feature is greater than or equal to a preset proportion, and the importance parameter of the second target feature is lower than the importance parameters of other features other than the second target feature. This embodiment reduces the number of features by deleting features with smaller importance parameters, thereby avoiding excessive growth of data processing volume as the number of iterations increases due to an excessive number of features.

[0022] In some embodiments, the features corresponding to the first layer nodes of the graph network are obtained by extracting features from the second data to be processed; the device also includes: a fourth feature extraction module for extracting the first original feature from the first data to be processed; a statistical feature determination module for determining the statistical features of the first original feature; and an alarm module for outputting an alarm message when the statistical features of the second original feature extracted from the second data to be processed do not match the statistical features of the first original feature. This embodiment outputs alarm information based on the statistical features of the data, so that the user can take timely processing measures when the actual processed data does not match the statistical features of the training data for generating the graph network, for example, updating the statistical features of the second original feature, thereby reducing errors caused by data mismatch.

[0023] In some embodiments, the statistical feature determination module includes: a category determination unit, which is used to determine the category of the first original feature; and a statistical feature determination unit, which is used to determine the statistical feature of the first original feature based on the category of the first original feature. First original features of different categories have different statistical features. For example, the maximum value of the first original feature of the age category is about 100, and the minimum value is 0; the minimum value of the first original feature of the shopping amount category is 0, and the maximum value can be several thousand or tens of thousands. Determining the statistical feature based on the category of the first original feature has high accuracy and low operational complexity.

[0024] In some embodiments, the category of the first original feature is a numerical category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of at least one feature value of the first original feature; or the category of the first original feature is a non-numerical character category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of the length of the characters included in at least one feature value of the first original feature. In the above manner, statistical features can be determined for first original features of various categories such as numbers, Chinese characters, and symbols.

[0025] In some embodiments, the features corresponding to the first-layer nodes of the graph network are obtained by extracting features from the second data to be processed; the second data to be processed includes at least two groups of structured data; the device further includes: a primary key merging module for performing primary key merging on the at least two groups of structured data to obtain merged group structured data; a fifth feature extraction module for performing feature extraction on the merged group structured data to obtain the features corresponding to the first-layer nodes of the graph network. Through the multi-table merging function module, the data of multiple tables can be converted into single-table data, which is convenient for processing.

[0026] According to a fifth aspect of an embodiment of the present disclosure, a graph network generating device is provided, wherein the graph network is used to extract features from first data to be processed, and the device comprises: a second feature extraction module, used to obtain features corresponding to leaf nodes of an original graph network, and to extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, and each node of the graph network is used to record information about a feature, and each edge of the graph network is used to record information about a feature extraction rule; an updating module, used to update the original graph network based on features corresponding to leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; a first optimization module, used to determine feature values ​​corresponding to each node in the updated graph network, and extract features from the features in the updated graph network; When the feature value corresponding to the leaf node in the updated graph network is the same as the feature value corresponding to the non-leaf node in the updated graph network, the leaf node and the edge connecting the leaf node are deleted from the updated graph network to obtain a first optimized graph network; a second optimization module is used to filter out a second target feature from the new feature through a pre-trained tree model; the importance of the second target feature is lower than the importance of the features other than the second target feature in the new feature, and the nodes corresponding to the second target feature and the edges connecting the nodes corresponding to the second target feature are deleted from the first optimized graph network to obtain a second optimized graph network; a second return module is used to take the second optimized graph network as the original graph network, return to the step of obtaining the features corresponding to the leaf nodes of the original graph network, and determine the second optimized graph network that meets the preset conditions as the graph network.

[0027] The disclosed embodiment manages the dependency and generation relationship between features through a directed acyclic graph, generates each layer of nodes of the graph network in an iterative manner, realizes automated feature engineering without the need for human intervention, and improves the efficiency of feature engineering; when constructing each layer of nodes of the graph network, prunes the nodes and / or edges on at least two paths that obtain the same feature value, and screens the features corresponding to each layer of newly generated nodes through a neural network, thereby avoiding the number of features growing too fast as the number of iterations increases, thereby being able to extract deep features by increasing the number of iterations.

[0028] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the embodiments is implemented.

[0029] According to a seventh aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the embodiments is implemented.

[0030] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0032] Figure 1 is a schematic diagram of a distributed system of some embodiments.

[0033] Figure 2 It is a flow chart of the feature engineering method of the embodiment of the present disclosure.

[0034] Figure 3 It is a schematic diagram of a graph network according to an embodiment of the present disclosure.

[0035] Figure 4 is a schematic diagram of an updated graph network according to an embodiment of the present disclosure.

[0036] Figure 5 It is a schematic diagram of the graph network construction process of an embodiment of the present disclosure.

[0037] Figure 6 It is an overall flow chart of an embodiment of the present disclosure.

[0038] Figure 7 It is a flowchart of a graph network generation method according to an embodiment of the present disclosure.

[0039] Figure 8 It is a block diagram of a feature engineering device according to an embodiment of the present disclosure.

[0040] Fig. 9 It is a block diagram of a graph network generating device according to an embodiment of the present disclosure.

[0041] Fig.10 It is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0044] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0046] Feature engineering is a series of engineering processes on raw data to refine it into features as input for algorithms and models. For example, if the raw data is a user information table, the features extracted through feature engineering may include the user's name, age, gender, income, and the average income of users with the gender "male". Of course, this is only an example, and the features extracted by actual feature engineering may be more complex. In the process of feature engineering, multiple iterations are often performed to extract deep features. However, as the number of iterations increases, the number of processed features will increase exponentially. In order to improve data processing efficiency, feature engineering needs to be transplanted to a distributed system.

[0047] like Figure 1 As shown, it is a schematic diagram of a distributed system of some embodiments. The distributed system may include multiple nodes, and data can be exchanged between the multiple nodes to jointly perform a task. A node can be a processor (such as a CPU, a GPU), a server, or a server cluster. The multiple nodes may include one or more master nodes 101 and one or more slave nodes 102. The master node 101 is used to schedule the data processed by each slave node 102 and summarize the data returned by each slave node 102. Each slave node 102 can perform data processing under the scheduling of the master node.

[0048] Because different nodes in a distributed system may use the same or different processing methods and resources when processing data, when related technologies migrate feature engineering to different nodes in a distributed system, they need to define the specific operation methods and processing resources of each node separately, which makes the migration process time-consuming and error-prone.

[0049] Based on this, the present disclosure provides a feature engineering method, which is applied to the slave nodes of a distributed system. Figure 2 , the method comprising:

[0050] Step 201: receiving first data to be processed sent by a master node of the distributed system;

[0051] Step 202: extracting features from the first data to be processed based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph;

[0052] Step 203: Returning the first target feature to the master node;

[0053] Each node of the graph network is used to record information about a feature, and each edge of the graph network is used to record information about a feature extraction rule.

[0054] The method of the embodiment of the present disclosure can be Figure 1 The slave node 102 of the distributed system shown in FIG. 201 is executed. In step 201, the slave node may receive the first data to be processed sent by the master node of the distributed system, and the first data to be processed may be data in various forms such as text, picture, video, audio, etc. In some embodiments, the first data to be processed is structured data (e.g., a table) or data generated in a fixed typesetting manner (e.g., an ID card image). The master node may divide the first data to be processed into multiple groups, and schedule each slave node to process different data groups according to a certain scheduling strategy.

[0055] In step 202, each slave node may pre-store a graph network. Each node of the graph network corresponds to a feature, and each edge of the graph network corresponds to a feature extraction rule, such as Figure 3 As shown in Figure 1, it is a schematic diagram of a graph network. A circle in the figure represents the feature corresponding to a node, where A ij Represents the feature corresponding to the jth node of the i-th layer of the graph network; a directed arrow represents a feature extraction rule, where r ik The kth feature extraction rule corresponds to generating the features corresponding to the i+1th layer nodes from the features corresponding to the i-th layer nodes of the graph network; the direction of the arrow indicates the generation relationship of the features, for example, r 11 By Node A11 Point to Node A 21 , indicating that node A is generated first 11 The corresponding features are then represented by node A 11 The corresponding feature generation node A 21 Corresponding features. The features corresponding to a node can be generated by the features corresponding to one or more nodes in the previous layer and the feature extraction rules. For example, node A 21 The corresponding feature is only represented by a node A in the previous layer 11 The corresponding features are calculated according to r 11 The corresponding feature extraction rules are generated, and node A 22 The corresponding features are obtained by node A in the previous layer. 11 The corresponding features are calculated according to r 21 Corresponding feature extraction rules and node A 12 The corresponding features are calculated according to r 13 Corresponding feature extraction rules are jointly generated. A feature can generate another feature through one or more feature extraction rules. These one or more feature extraction rules are called the path for generating the feature. 12 The corresponding feature extraction rules and r 13 The corresponding feature extraction rules can generate A 22 The corresponding feature is r 12 The corresponding feature extraction rules and r 13 The corresponding feature extraction rule is to generate A 22 The path of the corresponding feature. Those skilled in the art will understand that the figure is only an exemplary illustration. In actual applications, the number of layers and nodes of the graph network is not limited to the situation shown in the figure. By constructing a graph network, parallel data calculation can be performed based on the dependency relationship between features, because the graph network only retains the corresponding calculation order and logic, and the calculation engine can be easily switched, which is convenient for application in distributed systems.

[0056] A slave node can extract features from the first data to be processed according to all feature extraction rules in the entire graph network to obtain the first target feature, or can extract features from the first data to be processed according to only some feature extraction rules in the graph network to obtain the first target feature. 11 to r 17 Corresponding feature extraction rules are used to extract features from the first data to be processed, and the features of the second layer of the graph network, i.e., node A, are obtained. 21 , A 22 , A 23 and A 24 For example, the slave node can only use r in the graph network 11 、r 12 、r21 and r 22 Corresponding feature extraction rules are used to extract features of the first data to be processed, and A is obtained. 31 Corresponding features.

[0057] In step 203, the slave node may return the extracted first target feature to the master node. After receiving the first target feature returned by the slave node, the master node may send the first target feature to other slave nodes for subsequent processing. Alternatively, the master node may also aggregate the first target features returned by multiple slave nodes to obtain a complete first target feature.

[0058] In related technologies, when feature engineering is performed through a distributed system, specific operation methods and processing resources are defined on each node of the distributed system. For example, for the feature extraction rule of "exponential operation", node 1 calls a processing core in a multi-core CPU for processing, and the processing core converts the data that needs to be exponentially operated into 32-bit floating-point numbers during processing; while node 2 calls multiple processing cores in a multi-core CPU for processing, and the processing core converts the data that needs to be exponentially operated into 64-bit floating-point numbers during processing. Therefore, for node 1 and node 2, the number of CPU processing cores used when executing the feature extraction rule of exponential operation and the number of bits of floating-point numbers during the operation are usually defined respectively.

[0059] The disclosed embodiment manages the dependencies and generation relationships between features through a graph network, and then pre-stores the graph network on the nodes of a distributed system. Since the graph network only retains the logic of generating features (i.e., feature extraction rules) and does not include the specific operation method and processing resources for executing the logic, when the graph network is transplanted to each node of the distributed system, there is no need to define the specific operation method and processing resources used by each node in the distributed system, thereby reducing the complexity of transplanting the graph network between nodes in the distributed system, reducing the time consumed in the transplantation process, and reducing the probability of errors. In the above example, when processing, Node 1 and Node 2 only need to read the corresponding processing logic, i.e., "exponential operation", from the graph network, and then use their respective processing methods and processing resources for processing.

[0060] The graph network used in the above embodiment can be generated in an iterative manner. Specifically, the features corresponding to the leaf nodes of the original graph network can be obtained; the features corresponding to the leaf nodes of the original graph network are extracted based on the pre-generated feature extraction rules to obtain new features; based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules, the original graph network is updated to obtain an updated graph network; the updated graph network is used as the original graph network, and the step of obtaining the features corresponding to the leaf nodes of the original graph network is returned until the preset conditions are met, and the updated graph network that meets the preset conditions is determined as the graph network (i.e., the pre-stored graph network).

[0061] In some embodiments, the features corresponding to the first layer nodes of the graph network can be determined by extracting features from the second data to be processed. The second data to be processed may include the same data items as the first data to be processed. For example, both the first data to be processed and the second data to be processed include data items such as the user's name, gender, and age. The data items in the first data to be processed may be exactly the same as the data items in the second data to be processed; or, the data items in the first data to be processed are a subset of the data items in the second data to be processed. For example, the data items in the first data to be processed include the user's name and gender, and the data items in the second data to be processed include the user's name, gender, and age.

[0062] In some embodiments, the second data to be processed includes at least two groups of structured data, for example, tables. The at least two groups of structured data can be merged by primary key to obtain merged group structured data; the merged group structured data can be feature extracted to obtain the features corresponding to the first layer nodes of the graph network. The automatic multi-table merging process can be performed by the BFS (BreathFirst Search) algorithm. Through the functional module of multi-table merging, the data of multiple tables can be converted into single-table data, which is convenient for processing. The primary key can be pre-specified. For example, the order table includes columns such as user's name, transaction amount and transaction time. The "name" column can be designated as the primary key, and the transaction amounts and transaction times of different users in the two order tables are integrated. In some embodiments, Table 1 and Table 2 before merging and Table 3 after merging are shown in the following table.

[0063] Table 1

[0064]

[0065] Table 2

[0066] Name Transaction amount Trading Hours Li Si 20 April 19, 2021 Wang Wu 231 April 20, 2021

[0067] Table 3

[0068]

[0069] After extracting the features corresponding to the nodes in the first layer of the graph network, the pre-generated feature extraction rules can be used to extract the features corresponding to the nodes in the first layer to obtain the features corresponding to the nodes in the second layer; the pre-generated feature extraction rules can then be used to extract the features corresponding to the nodes in the second layer to obtain the features corresponding to the nodes in the third layer; and so on. Among them, the feature extraction rules used to extract features corresponding to nodes in different layers can be the same or different. In some embodiments, feature extraction rule 1 can be used to first extract features from the 1st to the nth layers of the graph network, and then feature extraction rule 2 can be used to extract features from the n+1th to the n+tth layers of the graph network, where n and t are both positive integers greater than or equal to 1. The feature categories generated in a loop will change with the number of loops, preventing the generated features from being too single and nested, and preventing the loop generation of features from causing feature explosion.

[0070] Each time a new feature is obtained, the new feature and the feature extraction rule for generating the new feature can be updated to the graph network, that is, the new feature is used as the leaf node of the updated graph network, and the feature extraction rule for generating the new feature is used as the edge, from the node corresponding to the feature used to generate the new feature to the node corresponding to the new feature. Figure 3 and Figure 4 , assuming that by r 31 The corresponding feature extraction rule pair A 31 The corresponding features are extracted to obtain A 41 The corresponding features are obtained by 32 The corresponding feature extraction rule pair A 31 The corresponding features are extracted and the 33 The corresponding feature extraction rule pair A 32 The corresponding features are extracted and A is obtained together 42 Corresponding features, then A 41 and A 42 As a leaf node, r 31 As from A 31 Point to A 41 The edge of r 32 As from A 31 Point to A 42 The edge of r 33 As from A 32 Point to A 42 , thus obtaining the updated graph network.

[0071] Iterative processing is performed in the above manner until a preset condition is met, where the preset condition may be that the number of features reaches a preset number, the number of iterations reaches a preset number, or the performance (for example, accuracy) of a neural network trained based on the extracted features reaches a preset performance indicator.

[0072] In order to avoid generating too many features, the generated new features can be screened after each iteration, thereby reducing the number of new features generated each time. The screening can include at least one of static feature screening and dynamic feature screening. Static feature screening refers to screening out substantially the same feature extraction methods, and dynamic feature screening refers to screening out features with lower importance. Figure 5 As shown, after each iteration generates new features, static feature screening and dynamic feature screening can be performed alternately. For example, after the first iteration, the features corresponding to the second-layer nodes are generated based on the features corresponding to the first-layer nodes, and the features corresponding to the second-layer nodes are subjected to static feature screening and dynamic feature screening; then the filtered graph network is iterated for the second time, and the features corresponding to the third-layer nodes are generated based on the features corresponding to the second-layer nodes, and the features corresponding to the third-layer nodes are subjected to static feature screening and dynamic feature screening; then the filtered graph network is iterated for the third time; and so on. The order of static feature screening and dynamic feature screening is not limited in the present disclosure, and static feature screening can be performed first and then dynamic feature screening, or dynamic feature screening can be performed first and then static feature screening.

[0073] Specifically, during static feature screening, feature values ​​can be randomly generated for features corresponding to the first-layer nodes in the update graph network; feature values ​​corresponding to other nodes in the update graph network except the first-layer nodes are obtained based on the feature values ​​corresponding to the first-layer nodes; the feature value of the new feature is compared with the feature values ​​of other features, and the nodes corresponding to the other features are the nodes in the update graph network except the node corresponding to the new feature; if there are other features with the same feature value as the new feature, the new feature and the feature extraction rule used to generate the new feature are deleted from the update graph network.

[0074] The features corresponding to the first-layer nodes in the update graph network may be features extracted from the first data to be processed, and the process of extracting the features corresponding to the first-layer nodes may be referred to as data understanding. First, the first data to be processed may be divided into a plurality of categories based on the character type of the first data to be processed, each category corresponding to a character type, and the character type may include but is not limited to numbers, letters, Chinese characters, etc. Then, the first data to be processed of each category may be further divided into a plurality of subcategories, for example, the first data to be processed of the number category may be divided into age subcategories, amount subcategories, time subcategories, etc.; the first data to be processed of the Chinese character category may be divided into name subcategories, gender subcategories, address subcategories, occupation subcategories, etc. Each subcategory may be determined as a feature corresponding to the first-layer node. By statistically distinguishing the first data to be processed, the statistical features of the first data to be processed of each subcategory may also be determined. The statistical features include but are not limited to mean, variance, maximum value, minimum value, etc. For the first data to be processed of the number category, the statistical features such as the mean and variance of the numerical value may be directly calculated; for the first data to be processed of the character category, the statistical features corresponding to the number of characters may be calculated.

[0075] In some embodiments, one or more feature values ​​may be randomly generated for the features corresponding to the first-layer nodes in the update graph network based on the subcategories of the features (e.g., age, gender, etc.). For example, for features of age category, feature values ​​within the range of 0 to 100 may be randomly generated, and for features of gender category, feature values ​​of "male" or "female" may be randomly generated, or numerical feature values ​​representing "male" or numerical feature values ​​representing "female" may be randomly generated.

[0076] Then, the characteristic values ​​corresponding to the nodes other than the first-layer nodes in the update graph network can be obtained based on the characteristic values ​​corresponding to the first-layer nodes. Figure 3 In the embodiment shown, it is assumed that r 12 and r 13 The corresponding feature extraction rule is summation, then A 11 and A 12 The corresponding eigenvalues ​​are summed to obtain A 22 Corresponding eigenvalues. Thus, the eigenvalues ​​corresponding to each node in the updated graph network can be obtained.

[0077] For each new feature in the updated graph network, the feature value of the new feature can be compared with the feature values ​​of each feature generated in the historical iteration process to determine whether the two are the same. If they are the same, it means that the feature extraction rule used to generate the new feature is a redundant rule, and the rule that generates the new feature and the new feature can be deleted. Figure 3 For example, generate new features (i.e. A 31 and A 32The corresponding features) can be 31 The corresponding eigenvalues ​​are 11 , A 12 , A 13 , A 21 、A22、A 23 , A 24 The corresponding eigenvalues ​​can be compared. Similarly, A 32 The corresponding eigenvalue is compared with the other eigenvalues ​​mentioned above. 31 The corresponding eigenvalue is the same as any other eigenvalue, delete A 31 and r 21 and r 22 .

[0078] During dynamic feature screening, the second target feature can be screened out from the new feature by using a pre-trained tree model; and the node corresponding to the second target feature and the edge connecting the node corresponding to the second target feature are deleted from the update graph network. The importance parameter of the second target feature is lower than a preset importance parameter threshold, and / or the proportion of the second target feature is greater than or equal to a preset proportion, and the importance parameter of the second target feature is lower than the importance parameters of other features other than the second target feature. For example, when the number of features whose importance parameters are lower than the preset importance parameter threshold is greater than or equal to a preset proportion, each feature whose importance parameter is lower than the preset importance parameter threshold can be determined as the second target feature. For another example, when the number of features whose importance parameters are lower than the preset importance parameter threshold is less than a preset proportion, a preset number of features with importance parameters ranging from small to large can be determined as the second target feature. The importance parameter of a feature is used to reflect the value of the feature, that is, the influence of the feature on the output result of the machine learning model. For example, when the machine learning model is used to perform a classification task, the importance parameter of the feature is used to reflect the degree of influence of the feature on the classification result, and the greater the importance parameter, the greater the degree of influence of the feature on the classification result. The importance parameter of the feature can be determined based on the number of splits of the feature in the tree model and / or the information gain of the feature. The more times the splits are, the higher the importance of the feature is, and the greater the information gain is, the higher the importance of the feature is. The proportion of the second target feature can be calculated by the ratio of the number of the second target features to the total number of all features.

[0079] In some embodiments, the first original feature can also be extracted from the first data to be processed; and the statistical features of the first original feature can be determined. In the case where the statistical features of the second original feature extracted from the second data to be processed do not match the statistical features of the first original feature, an alarm message can be output. The statistical features of the second original feature do not match the statistical features of the first original feature, which may be due to the unreasonable selection or insufficient number of the second data to be processed, resulting in the unreasonable statistical features of the second original feature. For example, in actual situations, the age distribution of users in online transaction orders is 15 to 65 years old, and the average age is 40 years old; but the age distribution of users in the second data to be processed is 20 to 25 years old, which fails to cover all possible situations, resulting in the average age of 22.5 years old. In this case, if there is an alarm message output, the user can update the statistical features of the second original feature. In other cases, the user can also perform other processing according to actual needs when the alarm message is output, and the present disclosure does not limit this.

[0080] The statistical features of the first original feature can be determined based on the category of the first original feature. For example, if the category of the first original feature is a numerical category, the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of at least one characteristic value of the first original feature. For another example, if the category of the first original feature is a non-numeric character category, the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of the character length included in at least one characteristic value of the first original feature.

[0081] like Figure 6 As shown, it is an overall flow chart of an embodiment of the present disclosure. For multiple groups of first data to be processed (data 1, data 2, ..., data N), initial features can be extracted from them through data understanding, and used as features corresponding to the first-layer nodes of the graph network. In the case where the first data to be processed is a table, the initial features can be the header information of each column in the table. Then, the primary keys of multiple tables can be merged. The merged tables are automatically constructed into a graph network, that is, each layer of nodes and edges of the graph network are iteratively generated, and dynamic feature screening and static feature screening are performed alternately. After the graph network is constructed, feature calculations are performed through the graph network. The graph network can also be deployed to each slave node of the distributed system, and each slave node can perform feature calculations under the scheduling of the master node of the distributed system.

[0082] The solution of the disclosed embodiment can be used for information push, wherein the first data to be processed may be a user information table, which may include information of multiple dimensions of the user, for example, age, name, gender, occupation, address, transaction information, etc., and a first target feature may be extracted from the user information table as sample data, a neural network may be trained based on the first target feature, and the user information table to be processed may be input into the neural network to obtain recommended information predicted by the neural network, and the recommended information may include but is not limited to videos pushed to users by video platforms, products pushed to users by online shopping platforms, news pushed to users by news platforms, etc.

[0083] The solution of the embodiment of the present disclosure can also be used in financial scenarios, such as insurance scenarios. The solution of the embodiment of the present disclosure can be used for information push, wherein the first data to be processed can be the user's insurance policy, and the insurance policy can include information of multiple dimensions of the user, such as age, name, gender, occupation, historical hospitalization information, historical illness information, etc. The first target feature can be extracted from the insurance policy sample, and the neural network can be trained based on the first target feature, and the insurance policy to be processed is input into the neural network to obtain the user's insurance information predicted by the neural network, and the user's insurance information can include but is not limited to the user's insurance category, insurance amount, etc.

[0084] In addition to the two application scenarios listed above, the solution of the embodiment of the present disclosure can also be used in other application scenarios, which will not be listed one by one.

[0085] like Figure 7 As shown, the embodiment of the present disclosure also provides a method for generating a graph network, wherein the graph network is used to extract features from first data to be processed, and the method includes:

[0086] Step 701: Acquire features corresponding to leaf nodes of the original graph network, and extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule;

[0087] Step 702: updating the original graph network based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network;

[0088] Step 703: Determine the eigenvalues ​​corresponding to the nodes in the update graph network, and when the eigenvalues ​​corresponding to the leaf nodes in the update graph network are the same as the eigenvalues ​​corresponding to the non-leaf nodes in the update graph network, delete the leaf nodes and the edges connecting the leaf nodes from the update graph network to obtain a first optimized graph network;

[0089] Step 704: Filter out a second target feature from the new features by using a pre-trained tree model; the importance of the second target feature is lower than the importance of features other than the second target feature in the new features, and delete nodes corresponding to the second target feature and edges connecting nodes corresponding to the second target feature from the first optimized graph network to obtain a second optimized graph network;

[0090] Step 705: Use the second optimized graph network as the original graph network, return to the step of obtaining the features corresponding to the leaf nodes of the original graph network, and determine the second optimized graph network that meets the preset conditions as the graph network.

[0091] The disclosed embodiments manage the dependencies and generation relationships between features through a graph network, and generate each layer of nodes of the graph network in an iterative manner, thereby realizing automated feature engineering without the need for human intervention and improving the efficiency of feature engineering. When constructing each layer of nodes of the graph network, the nodes and / or edges on at least two paths that obtain the same feature values ​​are pruned, and the features corresponding to each layer of newly generated nodes are screened through a neural network, thereby preventing the number of features from growing too fast as the number of iterations increases, thereby being able to extract deep features by increasing the number of iterations.

[0092] The graph network generated by the embodiments of the present disclosure can be used in the feature engineering method of any of the aforementioned embodiments to extract features from the first data to be processed, which will not be described in detail here.

[0093] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0094] like Figure 8 As shown, the embodiment of the present disclosure also provides a feature engineering device, which is applied to a slave node of a distributed system, and the device includes:

[0095] A receiving module 801 is used to receive first data to be processed sent by a master node of the distributed system;

[0096] A first feature extraction module 802 is used to extract features from the first to-be-processed data based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph;

[0097] A first returning module 803, configured to return the first target feature to the master node;

[0098] Each node of the graph network is used to record information about a feature, and each edge of the graph network is used to record information about a feature extraction rule.

[0099] The disclosed embodiment manages the dependencies and generation relationships between features through a directed acyclic graph, and then pre-stores the graph network on the nodes of the distributed system. Since the graph network only retains the logic of generating features (i.e., feature extraction rules) but does not include the specific operation method and processing resources for executing the logic, when the graph network is transplanted to each node of the distributed system, there is no need to define the specific operation method and processing resources used by each node in the distributed system, thereby reducing the complexity of transplanting the graph network between nodes of the distributed system, reducing the time consumed in the transplantation process, and reducing the probability of errors, thereby achieving seamless switching of the production environment and switching of the computing engine.

[0100] In some embodiments, the device further includes: a feature acquisition module for acquiring features corresponding to leaf nodes of the original graph network; a third feature extraction module for extracting features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; an updated graph network acquisition module for updating the original graph network based on features corresponding to leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; using the updated graph network as the original graph network, and returning to execute the function of the feature acquisition module to determine the updated graph network that meets preset conditions as the graph network. The disclosed embodiment generates nodes and edges at each layer in the graph network in an iterative manner, and has low complexity.

[0101] In some embodiments, after obtaining the new feature, the device further includes: a first feature value acquisition module, which is used to randomly generate feature values ​​for features corresponding to the first-layer nodes in the update graph network; a second feature value acquisition module, which is used to obtain feature values ​​corresponding to other nodes in the update graph network except the first-layer nodes based on the feature values ​​corresponding to the first-layer nodes; a comparison module, which is used to compare the feature value of the new feature with the feature values ​​of other features, and the nodes corresponding to the other features are nodes in the update graph network except the nodes corresponding to the new feature; a deletion module, which is used to delete the new feature and the feature extraction rules used to generate the new feature from the update graph network when there are other features with the same feature value as the new feature. This embodiment removes redundant features by deleting features with the same feature value, thereby reducing the number of features and avoiding excessive growth of data processing volume with the increase of iteration times due to excessive number of features.

[0102] In some embodiments, after obtaining the new feature, the device further includes: a screening module for screening out a second target feature from the new feature through a pre-trained tree model; the importance parameter of the second target feature is lower than a preset importance parameter threshold, and / or the proportion of the second target feature is greater than or equal to a preset proportion, and the importance parameter of the second target feature is lower than the importance parameters of other features other than the second target feature. This embodiment reduces the number of features by deleting features with smaller importance parameters, thereby avoiding excessive growth of data processing volume as the number of iterations increases due to an excessive number of features.

[0103] In some embodiments, the features corresponding to the first layer nodes of the graph network are obtained by extracting features from the second data to be processed; the device also includes: a fourth feature extraction module for extracting the first original feature from the first data to be processed; a statistical feature determination module for determining the statistical features of the first original feature; and an alarm module for outputting an alarm message when the statistical features of the second original feature extracted from the second data to be processed do not match the statistical features of the first original feature. This embodiment outputs alarm information based on the statistical features of the data, so that the user can take timely processing measures when the actual processed data does not match the statistical features of the training data for generating the graph network, for example, updating the statistical features of the second original feature, thereby reducing errors caused by data mismatch.

[0104] In some embodiments, the statistical feature determination module includes: a category determination unit, which is used to determine the category of the first original feature; and a statistical feature determination unit, which is used to determine the statistical feature of the first original feature based on the category of the first original feature. First original features of different categories have different statistical features. For example, the maximum value of the first original feature of the age category is about 100, and the minimum value is 0; the minimum value of the first original feature of the shopping amount category is 0, and the maximum value can be several thousand or tens of thousands. Determining the statistical feature based on the category of the first original feature has high accuracy and low operational complexity.

[0105] In some embodiments, the category of the first original feature is a numerical category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of at least one feature value of the first original feature; or the category of the first original feature is a non-numerical character category, and the statistical features of the first original feature include the mean, variance, maximum and / or minimum value of the length of the characters included in at least one feature value of the first original feature. In the above manner, statistical features can be determined for first original features of various categories such as numbers, Chinese characters, and symbols.

[0106] In some embodiments, the features corresponding to the first-layer nodes of the graph network are obtained by extracting features from the second data to be processed; the second data to be processed includes at least two groups of structured data; the device further includes: a primary key merging module for performing primary key merging on the at least two groups of structured data to obtain merged group structured data; a fifth feature extraction module for performing feature extraction on the merged group structured data to obtain the features corresponding to the first-layer nodes of the graph network. Through the multi-table merging function module, the data of multiple tables can be converted into single-table data, which is convenient for processing.

[0107] like Fig. 9 As shown, the embodiment of the present disclosure further provides a graph network generating device, wherein the graph network is used to extract features from the first data to be processed, and the device includes:

[0108] The second feature extraction module 901 is used to obtain features corresponding to leaf nodes of the original graph network, and extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule;

[0109] An updating module 902, configured to update the original graph network based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network;

[0110] A first optimization module 903 is used to determine the eigenvalues ​​corresponding to the nodes in the update graph network, and when the eigenvalues ​​corresponding to the leaf nodes in the update graph network are the same as the eigenvalues ​​corresponding to the non-leaf nodes in the update graph network, delete the leaf nodes and the edges connecting the leaf nodes from the update graph network to obtain a first optimized graph network;

[0111] The second optimization module 904 is used to filter out a second target feature from the new features by using a pre-trained tree model; the importance of the second target feature is lower than the importance of features other than the second target feature in the new features, and delete the nodes corresponding to the second target feature and the edges connecting the nodes corresponding to the second target feature from the first optimization graph network to obtain a second optimization graph network;

[0112] The second returning module 905 is used to take the second optimized graph network as the original graph network, and return to the step of obtaining the features corresponding to the leaf nodes of the original graph network, and determine the second optimized graph network that meets the preset conditions as the graph network.

[0113] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] The present disclosure also provides a distributed system, the system comprising:

[0115] A master node is used to distribute the data to be processed to each slave node for processing; and

[0116] At least one slave node, each slave node pre-deploys a graph network, and is used to extract features from the data to be processed based on the graph network, obtain a first target feature, and return the first target feature to the master node; wherein each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule. The graph network is a directed acyclic graph.

[0117] The structure of the distributed system can be as follows Figure 1 The structure shown will not be described in detail here.

[0118] An embodiment of the present specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0119] Fig.10A more specific hardware structure diagram of a computing device provided in an embodiment of this specification is shown, and the device may include: a processor 1001, a memory 1002, an input / output interface 1003, a communication interface 1004, and a bus 1005. The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are connected to each other in communication within the device through the bus 1005.

[0120] The processor 1001 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 1001 can also include a graphics card, which can be a Nvidia titan X graphics card or a 1080Ti graphics card, etc.

[0121] The memory 1002 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1002 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1002 and are called and executed by the processor 1001.

[0122] The input / output interface 1003 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0123] The communication interface 1004 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0124] The bus 1005 includes a path that transmits information between various components of the device (eg, the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ).

[0125] It should be noted that, although the above device only shows the processor 1001, the memory 1002, the input / output interface 1003, the communication interface 1004 and the bus 1005, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0126] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the embodiments of this specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of this specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of this specification.

[0129] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0130] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of this specification. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0131] The above is only a specific implementation of the embodiments of this specification. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiments of this specification. These improvements and modifications should also be regarded as the protection scope of the embodiments of this specification.

Claims

1. A feature engineering method, characterized in that: Applied to a slave node of a distributed system, the method comprises: Receiving first data to be processed sent by a master node of the distributed system, where the first data to be processed is in a form of text, picture, video or audio; Extracting features of the first data to be processed based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph; Returning the first target feature to the master node, where the first target feature is used to train a neural network, where the neural network is used to obtain recommendation information pushed to the user based on the input user information table; Each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule; The graph network is obtained based on the following method: Get the features corresponding to the leaf nodes of the original graph network; Extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; Based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules, the original graph network is updated to obtain an updated graph network; The updated graph network is taken as the original graph network, and the step of obtaining the features corresponding to the leaf nodes of the original graph network is returned to determine the updated graph network that meets the preset conditions as the graph network.

2. The method according to claim 1, characterized in that After obtaining the new feature, the method further includes: Randomly generate feature values ​​for features corresponding to the first layer nodes in the update graph network; Based on the characteristic values ​​corresponding to the first-layer nodes, obtaining characteristic values ​​corresponding to other nodes in the update graph network except the first-layer nodes; Compare the feature value of the new feature with the feature values ​​of other features, where the nodes corresponding to the other features are the nodes in the update graph network other than the node corresponding to the new feature; In the case that there are other features having the same feature value as the new feature, the new feature and the feature extraction rule used to generate the new feature are deleted from the update graph network.

3. The method according to claim 1, characterized in that: After obtaining the new feature, the method further includes: Filtering a second target feature from the new features by using a pre-trained tree model; The importance parameter of the second target feature is lower than a preset importance parameter threshold, and / or The proportion of the second target feature is greater than or equal to a preset proportion, and an importance parameter of the second target feature is lower than importance parameters of other features other than the second target feature.

4. The method according to claim 1, characterized in that: The features corresponding to the first layer nodes of the graph network are obtained by extracting features from the second data to be processed; The method further comprises: Extracting a first original feature from the first data to be processed; Determining statistical characteristics of the first original features; When the statistical features of the second original features extracted from the second data to be processed do not match the statistical features of the first original features, an alarm message is output.

5. The method according to claim 4, characterized in that The determining of the statistical feature of the first original feature comprises: determining a category of the first original feature; The statistical features of the first original features are determined based on the categories of the first original features.

6. The method according to claim 4, characterized in that The category of the first original feature is a numerical category, and the statistical feature of the first original feature includes a mean, a variance, a maximum value and / or a minimum value of at least one feature value of the first original feature; or The category of the first original feature is a non-numeric character category, and the statistical features of the first original feature include a mean, a variance, a maximum value and / or a minimum value of a character length included in at least one feature value of the first original feature.

7. The method according to any one of claims 1 to 6, characterized in that: The features corresponding to the first layer nodes of the graph network are obtained by extracting features from the second data to be processed; The second data to be processed includes at least two groups of structured data; the method further includes: Performing primary key merging on the at least two groups of structured data to obtain merged group structured data; Feature extraction is performed on the merged group structured data to obtain features corresponding to the first layer nodes of the graph network.

8. A graph network generation method, characterized in that: The graph network is used to extract features from first data to be processed, and the method includes: Acquire features corresponding to leaf nodes of the original graph network, and extract features corresponding to the leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule; The original graph network is updated based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; Determine the eigenvalues ​​corresponding to the nodes in the update graph network, and when the eigenvalues ​​corresponding to the leaf nodes in the update graph network are the same as the eigenvalues ​​corresponding to the non-leaf nodes in the update graph network, delete the leaf nodes and the edges connecting the leaf nodes from the update graph network to obtain a first optimized graph network; A second target feature is selected from the new feature by using a pre-trained tree model; the importance of the second target feature is lower than the importance of features other than the second target feature in the new feature, and nodes corresponding to the second target feature and edges connecting nodes corresponding to the second target feature are deleted from the first optimized graph network to obtain a second optimized graph network; The second optimized graph network is used as the original graph network, and the step of obtaining the features corresponding to the leaf nodes of the original graph network is returned, and the second optimized graph network that meets the preset conditions is determined as the graph network, wherein the slave nodes of the distributed system perform feature extraction based on the first data to be processed to obtain a first target feature, and the first data to be processed is in the form of text, picture, video or audio, and the first target feature is used to train a neural network, and the neural network is used to obtain recommendation information pushed to the user based on the input user information table.

9. A distributed system, characterized in that: The system comprises: A master node is used to distribute the data to be processed to each slave node for processing; and At least one slave node, each slave node pre-deploys a graph network, and is used to extract features of the data to be processed based on the graph network to obtain a first target feature, and return the first target feature to the master node; the graph network is a directed acyclic graph; the data to be processed includes text, pictures, videos or audio, and the first target feature is used to train a neural network, and the neural network is used to obtain recommended information pushed to the user based on an input user information table; Each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule; The graph network is obtained based on the following method: Get the features corresponding to the leaf nodes of the original graph network; Extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; Based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules, the original graph network is updated to obtain an updated graph network; The updated graph network is taken as the original graph network, and the step of obtaining the features corresponding to the leaf nodes of the original graph network is returned to determine the updated graph network that meets the preset conditions as the graph network.

10. A feature engineering device, characterized in that: Applied to a slave node of a distributed system, the device comprises: A receiving module, configured to receive first data to be processed sent by a master node of the distributed system, wherein the first data to be processed is in a form of text, picture, video or audio; A first feature extraction module, used to extract features from the first to-be-processed data based on a pre-stored graph network to obtain a first target feature; the graph network is a directed acyclic graph; A first return module, used to return the first target feature to the master node, the first target feature is used to train a neural network, and the neural network is used to obtain recommendation information pushed to the user based on an input user information table; Each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule; The graph network is obtained based on the following modules: A second feature extraction module is used to obtain features corresponding to leaf nodes of the original graph network; based on pre-generated feature extraction rules, the features corresponding to the leaf nodes of the original graph network are extracted to obtain new features; An updating module is used to update the original graph network based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; use the updated graph network as the original graph network, and return to the step of obtaining the features corresponding to the leaf nodes of the original graph network, and determine the updated graph network that meets the preset conditions as the graph network.

11. A graph network generating device, characterized in that: The graph network is used to extract features from first data to be processed, and the device includes: A second feature extraction module is used to obtain features corresponding to leaf nodes of the original graph network, and extract features corresponding to leaf nodes of the original graph network based on pre-generated feature extraction rules to obtain new features; the original graph network is a directed acyclic graph, each node of the graph network is used to record information of a feature, and each edge of the graph network is used to record information of a feature extraction rule; An updating module, configured to update the original graph network based on the features corresponding to the leaf nodes of the original graph network, the new features and the feature extraction rules to obtain an updated graph network; A first optimization module is used to determine the eigenvalues ​​corresponding to the nodes in the update graph network, and when the eigenvalues ​​corresponding to the leaf nodes in the update graph network are the same as the eigenvalues ​​corresponding to the non-leaf nodes in the update graph network, delete the leaf nodes and the edges connecting the leaf nodes from the update graph network to obtain a first optimized graph network; A second optimization module is used to filter out a second target feature from the new features through a pre-trained tree model; the importance of the second target feature is lower than the importance of features other than the second target feature in the new features, and delete nodes corresponding to the second target feature and edges connecting nodes corresponding to the second target feature from the first optimization graph network to obtain a second optimization graph network; A second return module is used to take the second optimized graph network as the original graph network, and return to the step of obtaining the features corresponding to the leaf nodes of the original graph network, and determine the second optimized graph network that meets the preset conditions as the graph network, wherein the slave nodes of the distributed system perform feature extraction based on the first data to be processed to obtain a first target feature, the first data to be processed is in the form of text, picture, video or audio, and the first target feature is used to train a neural network, and the neural network is used to obtain recommendation information pushed to the user based on the input user information table.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.