Feature synthesis method and device

Abstract

The embodiment of the present application provides a method and device for feature synthesis, which includes: obtaining a configuration file input by a user, the configuration file including a storage path of original data; obtaining the original data according to the storage path of the original data; obtaining feature definition information of the original data, the feature definition information is used to indicate the feature calculation logic of the original data; and synthesizing the features of the original data according to the feature definition information of the original data to obtain multiple synthesized features of the original data. That is, the embodiment of the present application realizes the automation of feature synthesis, which solves the problem that manual feature synthesis is time-consuming and labor-intensive, and the number of synthesized features is limited.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a feature synthesis method and device. Background Art

[0002] The application of machine learning requires a lot of manual intervention, which is manifested in various aspects of machine learning, such as feature extraction, model selection, and parameter adjustment. In order to reduce manual intervention in machine learning, AutoML (Automated Machine Learning) was born. AutoML is used to automatically learn important steps related to features, models, optimization, and evaluation, so that machine learning models can be applied without manual intervention.

[0003] Feature engineering is a core component of AutoML, including steps such as feature synthesis, feature generation, and feature screening. Feature synthesis is an important means of producing new features. It produces new features through different transformation operations designed by experienced modelers for data. Usually, most of the work of modelers revolves around features, and in order to improve the modeling effect, they will try different features.

[0004] The current mainstream feature synthesis method is to perform feature synthesis through manual design, which means that modelers design and implement different feature synthesis operators based on application scenarios and modeling experience to produce new features. However, manual feature synthesis is not only time-consuming and labor-intensive, but also has a limited number of synthesized features. Summary of the invention

[0005] The embodiments of the present application provide a feature synthesis method and device for improving the scheduling reliability between microservices.

[0006] In a first aspect, an embodiment of the present application provides a feature synthesis method, comprising:

[0007] Obtaining a configuration file input by a user, wherein the configuration file includes a storage path of original data;

[0008] Obtaining the original data according to the storage path of the original data;

[0009] Acquire feature definition information of the original data, where the feature definition information is used to indicate feature calculation logic of the original data;

[0010] According to the feature definition information of the original data, the features of the original data are synthesized to obtain a plurality of synthesized features of the original data.

[0011] In a possible implementation manner of the first aspect, the acquiring feature definition information of the original data includes:

[0012] When detecting that the user inputs a feature definition file, obtaining feature definition information of the original data from the feature definition file;

[0013] When the user input feature definition file is not detected, a first feature operator corresponding to the original data is determined, and feature definition information of the original data is obtained according to the first feature operator corresponding to the original data and the original data.

[0014] In a possible implementation manner of the first aspect, determining the first feature operator corresponding to the original data includes:

[0015] Obtaining the user-defined operator file input by the user, and generating a first feature operator corresponding to the original data according to the user-defined operator file; and / or,

[0016] According to the second feature operator information specified by the user in the configuration file, the second feature operator is obtained, and the first feature operator corresponding to the original data is determined according to the second feature operator.

[0017] In a possible implementation manner of the first aspect, obtaining feature definition information of the original data according to the first feature operator corresponding to the original data and the original data includes:

[0018] Based on the first feature operator corresponding to the original data and a preset feature synthesis principle, a quantity threshold of the synthetic features of the original data and a calculation logic of the synthetic features are obtained;

[0019] The feature definition information is formed according to the quantity threshold of the synthetic features and the calculation logic of the synthetic features.

[0020] In a possible implementation manner of the first aspect, synthesizing features of the original data according to feature definition information of the original data to obtain multiple synthesized features of the original data includes:

[0021] Acquire a quantity threshold of the synthetic features from the feature definition information;

[0022] When it is determined that the number of the synthetic features currently generated is less than the number threshold, the synthetic features of the original data continue to be generated according to the feature definition information of the original data.

[0023] In a possible implementation manner of the first aspect, before synthesizing features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data, the method further includes:

[0024] Checking the data type of the original data, and if the data type of the original data is not the target data type, converting the data type of the original data to the target data type;

[0025] The step of synthesizing the features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data includes:

[0026] According to the feature definition information of the original data, feature synthesis is performed on the original data of the target data type to obtain a plurality of synthesized features of the original data.

[0027] In a possible implementation manner of the first aspect, the original data is data in a single data table, or the original data is artificially synthesized initial features.

[0028] In a possible implementation manner of the first aspect, performing feature synthesis on the original data according to feature definition information of the original data to obtain multiple synthesized features of the original data includes:

[0029] sending the feature definition information of the original data and the original data to a first computing device, so that the first computing device performs feature synthesis on the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data; or

[0030] The original data is divided into N sub-data, and the feature definition information of the original data is sent to each second computing device on N different second computing devices, and the N sub-data are sent one-to-one to the N second computing devices, so that each of the N second computing devices performs feature synthesis on the sub-data according to the feature definition information of the sub-data on the second computing device and the original data to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each of the sub-data constitutes multiple synthetic features of the original data.

[0031] In a second aspect, an embodiment of the present application provides a feature synthesis device, including:

[0032] A first acquisition unit, configured to acquire a configuration file input by a user, wherein the configuration file includes a storage path of original data;

[0033] A second acquisition unit, configured to acquire the original data according to a storage path of the original data;

[0034] A processing unit, used for acquiring feature definition information of the original data, wherein the feature definition information is used for indicating feature calculation logic of the original data;

[0035] The synthesis unit is used to perform feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data.

[0036] In a possible implementation of the second aspect, the processing unit is specifically used to obtain feature definition information of the original data from the feature definition file when the user input feature definition file is detected; when the user input feature definition file is not detected, determine the first feature operator corresponding to the original data, and obtain the feature definition information of the original data based on the first feature operator corresponding to the original data and the original data.

[0037] In a possible implementation of the second aspect, the processing unit is specifically used to obtain the custom operator file input by the user, and generate a first feature operator corresponding to the original data based on the custom operator file; and / or, obtain the second feature operator based on the second feature operator information specified by the user in the configuration file, and determine the first feature operator corresponding to the original data based on the second feature operator.

[0038] In a possible implementation of the second aspect, the processing unit is specifically used to obtain a quantity threshold of synthetic features of the original data and a calculation logic of the synthetic features based on a first feature operator corresponding to the original data and a preset feature synthesis principle; and form the feature definition information according to the quantity threshold of synthetic features and the calculation logic of the synthetic features.

[0039] In a possible implementation manner of the second aspect, the synthesis unit is specifically used to obtain a quantity threshold of the synthetic features from the feature definition information; when it is determined that the number of the synthetic features currently generated is less than the quantity threshold, continue to generate the synthetic features of the original data according to the feature definition information of the original data.

[0040] In a possible implementation of the second aspect, the synthesis unit is specifically used to check the data type of the original data, and if the data type of the original data is not the target data type, convert the data type of the original data into the target data type; and perform feature synthesis on the original data of the target data type according to feature definition information of the original data to obtain multiple synthetic features of the original data.

[0041] In a possible implementation manner of the second aspect, the original data is data in a single data table, or the original data is artificially synthesized initial features.

[0042] In a possible implementation manner of the second aspect, the synthesis unit is specifically configured to send the feature definition information of the original data and the original data to a first computing device, so that the first computing device performs feature synthesis on the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data; or

[0043] The original data is divided into N sub-data, and the feature definition information of the original data is sent to each second computing device on N different second computing devices, and the N sub-data are sent one-to-one to the N second computing devices, so that each of the N second computing devices performs feature synthesis on the sub-data according to the feature definition information of the sub-data on the second computing device and the original data to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each of the sub-data constitutes multiple synthetic features of the original data.

[0044] In a third aspect, an embodiment of the present application further provides a computing device, the computing device comprising a memory and a processor;

[0045] The memory is used to store computer programs;

[0046] The processor is used to execute the computer program to implement the method described in any one of the first aspects above.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium includes computer instructions. When the instructions are executed by a computer, the computer implements the method as described in any one of the first aspects.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a readable storage medium. At least one processor of a computer can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the computer implements any method described in the first aspect.

[0049] The feature synthesis method and device provided in the embodiment of the present application obtains a configuration file input by a user, wherein the configuration file includes a storage path of the original data; obtains the original data according to the storage path of the original data; obtains feature definition information of the original data, wherein the feature definition information is used to indicate the feature calculation logic of the original data; and performs feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data. That is, the embodiment of the present application realizes the automation of feature synthesis, which solves the problem that manual feature synthesis is time-consuming and labor-intensive, and the number of synthesized features is limited. In addition, it can support distributed systems, thereby solving the problem of long feature synthesis calculation time when the amount of data is large. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a feature synthesis method according to an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a distributed system according to an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a flow chart of a feature synthesis method provided in another embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of a feature synthesis device provided in an embodiment of the present application;

[0056] Figure 6 A block diagram of a computing device involved in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0058] It should be understood that in the embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based only on A, and B can also be determined based on A and / or other information.

[0059] In the description of the present application, unless otherwise specified, “plurality” means two or more than two.

[0060] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0061] In order to facilitate understanding of the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are briefly introduced as follows:

[0062] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0063] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0064] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0065] From the perspective of machine learning, AutoML can be seen as a system with very strong learning and generalization capabilities on given data and tasks. From the perspective of automation, AutoML can be seen as designing a series of advanced control systems to operate machine learning models, so that the models can automatically learn the appropriate parameters and configurations without human intervention.

[0066] The core tasks of AutoML: better performance, no human intervention, and lower computational cost.

[0067] The main problems of AutoML can be composed of three parts: feature engineering, model selection, and algorithm selection. The embodiments of this application mainly involve the above-mentioned feature engineering.

[0068] Feature engineering plays a vital role in machine learning. In AutoML, the purpose of automatic feature engineering is to automatically discover and construct relevant features so that the model can have the best performance. In addition, it also includes some specific feature enhancement methods, such as feature synthesis, feature generation, feature screening, etc. These steps have not yet reached the stage of automation.

[0069] For feature synthesis in feature engineering, modelers currently design and implement different feature synthesis operators based on application scenarios and modeling experience to produce new features. However, manual feature synthesis is not only time-consuming and labor-intensive, but also has a limited number of synthesized features.

[0070] In order to solve the above technical problems, the embodiments of the present application perform feature synthesis through a feature synthesis tool to realize the automation of feature synthesis, thereby solving the problem that manual feature synthesis is time-consuming and labor-intensive, and the number of synthesized features is limited.

[0071] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0072] Figure 1 Schematic diagram of application scenarios of the embodiments of the present application, such as Figure 1 Shown include database, feature synthesis tools.

[0073] The database includes original data.

[0074] The feature synthesis tool includes multiple feature operators, such as operator 1, operator 2, operator 3 and operator 4. At the same time, the feature synthesis tool also includes a stand-alone or distributed system, that is, the feature synthesis method of the embodiment of the present application is not only applicable to a stand-alone machine, but also to a distributed system. For example, when the amount of original data is small, a stand-alone machine can be selected for feature synthesis, and when the amount of original data is large, a distributed system can be selected for feature synthesis.

[0075] A single machine or distributed system uses multiple feature operators in a feature synthesis tool to synthesize features of the original data to obtain multiple synthetic features of the original data.

[0076] In some embodiments, the original features can be understood as the original data itself, so that the synthetic features and original features of the original data can be used as feature data of the original data for subsequent feature screening and model training steps. For the feature screening and model training steps, refer to the description of the existing technology, and this application will not repeat them here.

[0077] Combine the following Figure 2 The feature synthesis method provided in the embodiment of the present application is introduced.

[0078] Figure 2 A schematic diagram of a feature synthesis method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the feature synthesis method includes:

[0079] S201. Obtain a configuration file input by a user.

[0080] S202: Obtain original data according to the storage path of the original data.

[0081] The execution subject of the embodiments of the present application is a device with a feature synthesis function, such as a feature synthesis device. The feature synthesis device can be a separate computing device. In some embodiments, the feature synthesis device can also be a component with a data processing function in the computing device, such as a processor in the computing device.

[0082] Optionally, the above computing device can be understood as Figure 1 The feature synthesis tool shown in .

[0083] In some embodiments, the computing device can be any device with data computing capabilities, such as a smart phone, a desktop computer, a notebook, an AI device, a smart TV, a vehicle-mounted device, a wearable device, an industrial device, a server, etc.

[0084] The synthetic features involved in this application can be understood as: using different feature calculation methods to calculate new features with data or features as input information.

[0085] The feature operators involved in this application can be understood as methods of operating features to obtain new features, usually multiple calculation methods such as aggregation and transformation.

[0086] The above configuration file (conf) includes the storage path of the original data. For example, the original data is stored under a certain path, and the user needs to configure the storage path of the original data in the above configuration file. After the computing device obtains the configuration file uploaded by the user, it parses the configuration file to obtain the storage path of the original data, and reads the original data according to the storage path of the original data.

[0087] In some embodiments, the storage path of the original data is the storage path of the original data in the database, so that the computing device can obtain the original data according to the storage path of the original data. Figure 1 Read the original data from the database shown.

[0088] In some embodiments, the above original data is data in a single data table, that is, single table data. In other words, the method of the embodiment of the present application can support feature synthesis in the form of a single table. In actual business, most scenarios are based on single table feature synthesis. Users can easily access a single table for feature synthesis through the method of the present application.

[0089] In some embodiments, the above raw data is an artificially synthesized initial feature. That is, the embodiments of the present application can use the original artificially synthesized initial feature as the raw data, perform deep synthesis on the initial feature, obtain high-order synthetic features, realize the combination of artificial and automatic feature synthesis, and help users obtain more synthetic features.

[0090] In some embodiments, the configuration file also includes feature operator indication information, for example, indicating one or more feature operators from a plurality of preset feature operators for calculating the synthetic features of the original data.

[0091] S203: Obtain feature definition information of original data.

[0092] The feature definition information is used to indicate the feature calculation logic of the original data.

[0093] In some embodiments, in addition to inputting a configuration file, the user may optionally input a feature definition file, so that the method for obtaining the feature definition information of the original data is different depending on whether the user inputs the feature definition file. That is, the method for obtaining the feature definition information of the original data in S203 includes but is not limited to the following:

[0094] Method 1: when it is detected that the user inputs a feature definition file, the feature definition information of the original data is obtained from the feature definition file.

[0095] The feature definition file input by the user has described in detail the feature calculation logic and other information of the original data. The feature definition information of the original data can be obtained by parsing the feature definition file input by the user.

[0096] Mode 2: When the user input feature definition file is not detected, the above S203 includes: S203-A1 and S203-A2:

[0097] S203-A1, determining a first characteristic operator corresponding to the original data;

[0098] S203-A2. Obtain feature definition information of the original data according to the first feature operator corresponding to the original data and the original data.

[0099] When the user does not input the feature definition file, the present application can adopt the method of mode 2 to obtain the feature definition information of the original data, specifically, determine the first feature operator corresponding to the original data, and obtain the feature definition information of the original data according to the first feature operator corresponding to the original data and the original data. For example, the model includes a training phase and a test phase. In the training phase, it is necessary to generate a training set, that is, to synthesize the features of the original data according to the first feature operator to form a new synthetic feature, and use the new synthetic feature as the training set. At the same time, the feature definition file generated when the synthetic feature is formed is saved. The feature definition file is used to save the feature calculation logic of the original data. In the later model testing phase, when the original data is also selected to generate a test set, the feature definition file formed by the training set can be used as the feature definition file of the test set, and then the feature definition information of the original data can be obtained from the feature definition file.

[0100] The embodiment of the present application supports two scenarios: the feature definition is known (i.e., the above method 1) and unknown (i.e., the above method 2). After the feature synthesis is completed through the training set, new synthetic features will be derived, and the definitions of these new synthetic features will also be obtained. At this time, there is no need to re-implement the calculation logic on the test set, but directly calculate with reference to the new feature definition on the training set.

[0101] In some embodiments, the methods for determining the first characteristic operator corresponding to the original data in S203-A1 include but are not limited to the following:

[0102] Method 1: If the user inputs a custom operator file, the custom operator file input by the user is obtained, and the first feature operator corresponding to the original data is generated according to the custom operator file.

[0103] For example, the computing device can provide a user with a custom operator interface, and the user inputs a custom operator file on the interface. For example, the user simply defines an operator function. The computing device can convert the custom operator file input by the user, automatically generate a custom feature operator, and determine the custom operator as the first feature operator corresponding to the original data for subsequent feature synthesis calculations.

[0104] In the second method, the configuration file includes the second characteristic operator information specified by the user, so that the second characteristic operator is obtained according to the second characteristic operator information specified by the user in the configuration file, and the first characteristic operator corresponding to the original data is determined according to the second characteristic operator.

[0105] In some embodiments, the computing device of the present application supports at least one type of feature operator as follows:

[0106] Supports built-in operators and self-built operators using feature tools. Usually the default parameters are not adjustable.

[0107] Supports operators with passable parameters. The operator types cover built-in operators and provide the function of passable parameters.

[0108] Supports text operators. For text features, the feature synthesis tool processes the data by default to meet the requirements of Feature tools and uses its operators to calculate text features.

[0109] Supports time operators. For time type data, the feature synthesis tool will convert the data to meet the requirements of Feature tools and use a variety of time operators to calculate time features.

[0110] That is to say, the computing device of the present application comes with at least one type of feature operator mentioned above, and each type of feature operator includes at least one feature operator. Based on this, the user can automatically select a second feature operator from multiple feature operators in the computing device, and carry the selected second feature operator information (such as the identifier of the second feature operator) in the configuration file and upload it to the computing device. The computing device parses the configuration file to obtain the second feature operator information carried by the configuration file, determines the second feature operator from the multiple feature operators it comes with according to the second feature operator information, and uses the determined second feature operator as the first feature operator corresponding to the original data.

[0111] Method 3: If the user inputs a custom operator file, and the configuration file includes the second characteristic operator information specified by the user, the custom operator file input by the user is obtained, and a part of the first characteristic operator corresponding to the original data is generated according to the custom operator file, and the details are referred to in the above method 1. In addition, according to the second characteristic operator information specified by the user in the configuration file, the second characteristic operator is obtained, and the second characteristic operator is used to determine another part of the first characteristic operator corresponding to the original data, and the details are referred to in the above method 2. These two parts of the first characteristic operator form multiple first characteristic operators corresponding to the original data.

[0112] This application supports the convenience of users to use multiple types of feature synthesis operators. Actual business has personalized requirements for feature operators in different scenarios. Through the method of this application, users can easily access a large number of predefined operators, and can also easily access custom synthesis operators. Users do not need to do additional operations, and the computing device will automatically parse different types of operators in the background.

[0113] According to the above-mentioned method 1, method 2 or method 3, the first feature operator corresponding to the original data is obtained, and then S203-A2 is executed to obtain the feature definition information of the original data according to the first feature operator corresponding to the original data and the original data.

[0114] In some embodiments, the above S203-A2 includes: obtaining the quantity threshold of the synthetic features of the original data and the calculation logic of the synthetic features based on the first feature operator corresponding to the original data and the preset feature synthesis principle; and forming feature definition information according to the quantity threshold of the synthetic features and the calculation logic of the synthetic features.

[0115] In some embodiments, the present application performs feature synthesis based on a deep feature synthesis (DFS) method, and the corresponding above-mentioned preset feature synthesis principle is the deep feature synthesis principle of DFS.

[0116] Among them, DFS is a feature that automatically generates relational datasets. Essentially, the algorithm traces the relationships in the data to the base fields, and then sequentially applies mathematical functions along that path to create the final features. By stacking the calculations in sequence, each new feature can be defined as having a certain depth d. Therefore, the algorithm is called "deep feature" synthesis.

[0117] According to the above steps, the computing device can obtain the feature definition information of the original data, and then execute the following step S204.

[0118] S204: Perform feature synthesis on the original data according to feature definition information of the original data to obtain multiple synthesized features of the original data.

[0119] like Figure 1 As shown, the method of the embodiment of the present application can support feature synthesis in a single machine and also support feature synthesis in a distributed system, that is, the implementation method of the above S204 includes at least the following two implementation methods.

[0120] Method 1: sending feature definition information of the original data and the original data to a first computing device, so that the first computing device synthesizes features of the original data according to the feature definition information of the original data to obtain multiple synthesized features of the original data.

[0121] That is, a single machine can be used for feature synthesis. For example, the feature definition information of the original data and the original data are sent to a first computing device, so that the first computing device synthesizes the features of the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data.

[0122] Optionally, the first computing device may be a computing device that executes steps S201 to S203.

[0123] Optionally, the first computing device is a computing device different from the computing device that executes steps S201 to S203.

[0124] If the first computing device is different from the computing device that executes the method of the present application, the first computing device calculates multiple synthetic features of the original data and then sends the calculated synthetic features to the computing device that executes the method of the present application.

[0125] In some embodiments, when the amount of original data is small, the above-mentioned method 1 is used to perform feature synthesis.

[0126] Method 2: divide the original data into N sub-data, and send the feature definition information of the original data to each second computing device among N different second computing devices, and send the N sub-data to the N second computing devices one-to-one, so that each second computing device among the N second computing devices synthesizes the features of the sub-data according to the feature definition information of the sub-data and the original data on the second computing device to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each sub-data constitutes multiple synthetic features of the original data.

[0127] like Figure 3 As shown, the original data is sliced ​​to form N sub-data, and the N sub-data are sent to the N second computing devices in the distributed system respectively, so that each second computing device includes one sub-data. At the same time, the feature definition information of the original data obtained in the above steps is sent to each second computing device, where the feature definition information of the original data includes definition information of multiple feature operators, such as Figure 3 The feature definition information of the original data shown includes definition information of four feature operators.

[0128] The process of feature synthesis performed by the N second computing devices is the same. Figure 3 The first second computing device in the example is used for explanation. Assume that the second computing device 1 includes sub-data 1. The second computing device 1 defines information according to the characteristics of the original data, for example Figure 3 The four feature operators in , perform feature synthesis on sub-data 1, and generate multiple synthetic features of the sub-data. Assuming that the preset feature synthesis depth is 1, then Figure 3 As shown, each of the four feature operators synthesizes features on sub-data 1 to generate four synthetic features, that is, each feature operator calculates a synthetic feature. Assuming that the preset feature synthesis depth is 2, each feature operator synthesizes a synthetic feature, and each operator is nested in pairs to synthesize a new synthetic feature.

[0129] like Figure 3 As shown, each second computing device synthesizes multiple synthetic features and outputs the synthesized synthetic features, so that the synthetic features output by each second computing device constitute the synthetic features of the original data.

[0130] In actual business, there are scenarios where the amount of data is very large. The above method 2 uses a distributed system to perform feature synthesis, which solves the difficulties of long feature synthesis calculation time when the amount of data is large.

[0131] In addition, this application supports distributed feature synthesis, that is, based on the distributed principle of Spark, it realizes the distributed computing of feature operators. Through data parallelism, the data is divided into different small sub-data sets, the feature synthesis operator is implemented as Spark's UDF, and applied to each sub-data set to obtain the synthesized features.

[0132] Optionally, in addition to being used in Spark distributed systems, the embodiments of the present application can also be applied to other distributed systems, such as Hadoop distributed systems.

[0133] The feature synthesis method provided in the embodiment of the present application obtains a configuration file input by a user, wherein the configuration file includes a storage path of the original data; obtains the original data according to the storage path of the original data; obtains feature definition information of the original data, wherein the feature definition information is used to indicate the feature calculation logic of the original data; and performs feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data. That is, the embodiment of the present application realizes the automation of feature synthesis, which solves the problem that manual feature synthesis is time-consuming and labor-intensive, and the number of synthesized features is limited. In addition, it can support distributed systems, thereby solving the problem of long feature synthesis calculation time when the amount of data is large.

[0134] Figure 4 A schematic diagram of a feature synthesis method according to another embodiment of the present application is shown below. Figure 5 Schematic diagram of the principle of feature synthesis involved in the embodiment of this application. Figure 4 and Figure 5 As shown, the feature synthesis method includes:

[0135] S301, obtaining a configuration file input by a user, where the configuration file includes a storage path of original data;

[0136] S302: Obtain original data according to the storage path of the original data.

[0137] The specific implementation process of the above S301 and S302 refers to the specific description of the above S201 and S202, which will not be repeated here.

[0138] S303: Check the data type of the original data. If the data type of the original data is not the target data type, convert the data type of the original data into the target data type.

[0139] For example, check the data type and missing values ​​of the original data. If the data type of the original data is time type or text type data, the time type or text type data needs to be processed and converted to meet the target data type, such as converting it to a data type that can be calculated by Feature tools.

[0140] S304. Obtain the custom operator file input by the user, and generate a first feature operator corresponding to the original data according to the custom operator file; and / or, obtain a second feature operator according to the second feature operator information specified by the user in the configuration file, and determine the first feature operator corresponding to the original data according to the second feature operator.

[0141] The specific implementation process of the above S304 refers to the specific description of the method 1, method 2 and method 3 in the above S203-A1, which will not be repeated here.

[0142] It should be noted that there is no order in which the above S304 and the above S303 are executed. For example, S304 is executed before the above S303, or after S303, or simultaneously with the above S303. This application does not impose any restrictions on this.

[0143] S305: Verify the first characteristic operator corresponding to the original data.

[0144] According to the method of S304 above, after the first characteristic operator corresponding to the original data is obtained, the first characteristic operator is checked, for example, the parameter-transfer operator, the time operator, and the user-defined operator in the first characteristic operator are checked in detail.

[0145] In some embodiments, this step also includes processing and assembling different first feature operators as required according to the operator assembly information in the configuration file, in preparation for the next feature calculation. For example, the first feature operator includes f(x) and g(x). If the feature synthesis depth is 2, the first feature operator including f(x) and g(x) is assembled to obtain f(g(x)), g(f(x)), etc. It should be noted that the assembly method of the above two feature operators includes other assembly methods besides nesting each other, and the present application does not impose any restrictions on this.

[0146] S306: Determine whether a feature definition file input by the user is detected. If a feature definition file input by the user is detected, execute S307; if no feature definition file input by the user is detected, execute S308.

[0147] S307: Obtain feature definition information of the original data from the feature definition file input by the user.

[0148] S308. Obtain feature definition information of the original data according to the first feature operator corresponding to the original data and the original data.

[0149] For example, based on the first feature operator corresponding to the original data and the preset feature synthesis principle, the number threshold of the synthetic features of the original data and the calculation logic of the synthetic features are obtained; according to the number threshold of the synthetic features and the calculation logic of the synthetic features, feature definition information is formed. For details, refer to the description of the second method in S203 above, which will not be repeated here.

[0150] When there are too many feature operators and the synthesis depth is too deep, a large number of meaningless features will be generated and the amount of calculation will be significantly increased, so it is necessary to constrain the feature dimension. Specifically, the following steps S309 to S312 are performed.

[0151] S309: Obtain a quantity threshold T of synthetic features from the feature definition information.

[0152] S310. When it is determined that the number i of synthetic features currently generated is less than the number threshold T, execute S311. When it is determined that the number i of synthetic features currently generated is equal to the number threshold T, execute S312.

[0153] S311. Continue to synthesize synthetic features of the original data according to the feature definition information of the original data.

[0154] For example, according to the feature definition information of the original data, the synthetic features of the original data are synthesized using the DFS method.

[0155] S312: Save the synthetic features of the original data.

[0156] In some embodiments, Figure 5 As shown, the synthetic features are saved in Csv / pickle format.

[0157] In some embodiments, a feature definition file is also saved to form a feature definition file corresponding to the original data.

[0158] In some embodiments, Figure 5 As shown, save the feature definition file in Json / npz format.

[0159] S313, end.

[0160] In some embodiments, the present application provides a feature synthesis tool based on Feature tools, which is generally a feature tool that can be used in actual production. It is easy for users to access. By providing a configuration file, the tool can complete the feature synthesis calculation in the background. The tool parses the user's data path, feature operator type, and application scenario in the background, and selects the corresponding implementation method in the program according to different configurations. This process does not require any additional operations for the user.

[0161] The feature synthesis method provided in the embodiment of the present application can help junior modelers obtain more features without investing too much effort when they do not have much experience and are not familiar with the business. In addition, in the feature extraction stage at the initial stage of modeling, features can be directly synthesized from the original data as a tool for feature extraction. It can also be used to produce more features through feature synthesis tools when artificially designed features are insufficient.

[0162] It should be understood that Figure 1 and Figure 5 This is only an example of the present application and should not be construed as limiting the present application.

[0163] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0164] Combination of the above Figure 1 and Figure 4 , describes in detail the method embodiment of the present application, and the following is combined with Figure 5 to Figure 6 , describe in detail the device embodiments of the present application.

[0165] Figure 5 A schematic diagram of the structure of a feature synthesis device provided in an embodiment of the present application.

[0166] like Figure 5 As shown, the feature synthesis device 100 may include:

[0167] A first acquisition unit 110 is used to acquire a configuration file input by a user, wherein the configuration file includes a storage path of original data;

[0168] A second acquisition unit 120 is used to obtain the original data according to the storage path of the original data;

[0169] A processing unit 130 is used to obtain feature definition information of the original data, where the feature definition information is used to indicate feature calculation logic of the original data;

[0170] The synthesis unit 140 is used to perform feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthesized features of the original data.

[0171] In some embodiments, the processing unit 130 is specifically used to obtain feature definition information of the original data from the feature definition file when the user input feature definition file is detected; when the user input feature definition file is not detected, determine the first feature operator corresponding to the original data, and obtain the feature definition information of the original data based on the first feature operator corresponding to the original data and the original data.

[0172] In some embodiments, the processing unit 130 is specifically used to obtain the custom operator file input by the user, and generate a first feature operator corresponding to the original data based on the custom operator file; and / or, obtain the second feature operator based on the second feature operator information specified by the user in the configuration file, and determine the first feature operator corresponding to the original data based on the second feature operator.

[0173] In some embodiments, the processing unit 130 is specifically used to obtain a quantity threshold of synthetic features of the original data and a calculation logic of the synthetic features based on a first feature operator corresponding to the original data and a preset feature synthesis principle; and form the feature definition information according to the quantity threshold of synthetic features and the calculation logic of the synthetic features.

[0174] In some embodiments, the synthesis unit 140 is specifically used to obtain a quantity threshold of the synthetic features from the feature definition information; when it is determined that the number of the synthetic features currently generated is less than the quantity threshold, continue to generate the synthetic features of the original data according to the feature definition information of the original data.

[0175] In some embodiments, the synthesis unit 140 is specifically used to check the data type of the original data, and if the data type of the original data is not the target data type, convert the data type of the original data to the target data type; and perform feature synthesis on the original data of the target data type according to feature definition information of the original data to obtain multiple synthetic features of the original data.

[0176] In some embodiments, the original data is data in a single data table, or the original data is artificially synthesized initial features.

[0177] In some embodiments, the synthesis unit 140 is specifically used to send the feature definition information of the original data and the original data to a first computing device, so that the first computing device performs feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data; or

[0178] The original data is divided into N sub-data, and the feature definition information of the original data is sent to each second computing device on N different second computing devices, and the N sub-data are sent one-to-one to the N second computing devices, so that each of the N second computing devices performs feature synthesis on the sub-data according to the feature definition information of the sub-data on the second computing device and the original data to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each of the sub-data constitutes multiple synthetic features of the original data.

[0179] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 5 The device 100 shown can execute the method of the embodiment of the present application, and the aforementioned and other operations and / or functions of each unit in the device 100 are respectively for implementing the corresponding processes in each method, which will not be repeated here for the sake of brevity.

[0180] Figure 6 This is a block diagram of a computing device involved in an embodiment of the present application. The computing device is used to execute the action generation method of the above embodiment. For details, please refer to the description in the above method embodiment.

[0181] Figure 6 The computing device 400 shown includes a memory 401, a processor 402, and a communication interface 403. The memory 401, the processor 402, and the communication interface 403 are connected to each other in communication. For example, the memory 401, the processor 402, and the communication interface 403 can be connected to each other through a network connection. Alternatively, the computing device 400 may further include a bus 404. The memory 401, the processor 402, and the communication interface 403 are connected to each other in communication through the bus 404. Figure 6 The computing device 400 is a memory 401 , a processor 402 , and a communication interface 403 that are communicatively connected to each other via a bus 404 .

[0182] The memory 401 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 401 may store a program, and when the program stored in the memory 401 is executed by the processor 402, the processor 402 and the communication interface 403 are used to execute the above method.

[0183] The processor 402 may be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0184] The processor 402 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, the method of the present application may be completed by an integrated logic circuit of hardware in the processor 402 or instructions in the form of software. The above-mentioned processor 402 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 402 reads the information in the memory 401, and completes the method of the embodiment of the present application in combination with its hardware.

[0185] The communication interface 403 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device 400 and other devices or communication networks. For example, a data set can be obtained through the communication interface 403 .

[0186] When the computing device 400 described above includes a bus 404 , the bus 404 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 401 , the processor 402 , and the communication interface 403 ).

[0187] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0188] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0189] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0190] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0191] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0192] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0193] In summary, the above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A feature synthesis method, characterized in that: include: Obtaining a configuration file input by a user, wherein the configuration file includes a storage path of original data; Obtaining the original data according to the storage path of the original data; When detecting that the user inputs a feature definition file, obtaining feature definition information of the original data from the feature definition file; When the user input feature definition file is not detected, the user-defined operator file input by the user is obtained, and a first feature operator corresponding to the original data is generated according to the user-defined operator file, and / or, according to the second feature operator information specified by the user in the configuration file, the second feature operator is obtained, and the first feature operator corresponding to the original data is determined according to the second feature operator, and feature definition information of the original data is obtained according to the first feature operator corresponding to the original data and the original data, wherein the feature definition information is used to indicate feature calculation logic of the original data; Performing feature synthesis on the original data according to feature definition information of the original data to obtain multiple synthesized features of the original data; The step of synthesizing the features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data includes: sending the feature definition information of the original data and the original data to a first computing device, so that the first computing device performs feature synthesis on the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data; or The original data is divided into N sub-data, and the feature definition information of the original data is sent to each second computing device on N different second computing devices, and the N sub-data are sent one-to-one to the N second computing devices, so that each of the N second computing devices performs feature synthesis on the sub-data according to the feature definition information of the sub-data on the second computing device and the original data to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each of the sub-data constitutes multiple synthetic features of the original data.

2. The method according to claim 1, characterized in that: The obtaining feature definition information of the original data according to the first feature operator corresponding to the original data and the original data includes: Based on the first feature operator corresponding to the original data and a preset feature synthesis principle, a quantity threshold of the synthetic features of the original data and a calculation logic of the synthetic features are obtained; The feature definition information is formed according to the quantity threshold of the synthetic features and the calculation logic of the synthetic features.

3. The method according to claim 1 or 2, characterized in that: The step of synthesizing the features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data includes: Acquire a quantity threshold of the synthetic features from the feature definition information; When it is determined that the number of the synthetic features currently generated is less than the number threshold, the synthetic features of the original data continue to be generated according to the feature definition information of the original data.

4. The method according to claim 3, characterized in that: Before synthesizing the features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data, the method further includes: Checking the data type of the original data, and if the data type of the original data is not the target data type, converting the data type of the original data to the target data type; The step of synthesizing the features of the original data according to the feature definition information of the original data to obtain a plurality of synthesized features of the original data includes: According to the feature definition information of the original data, feature synthesis is performed on the original data of the target data type to obtain a plurality of synthesized features of the original data.

5. The method according to claim 1, characterized in that: The original data is data in a single data table, or the original data is initial features synthesized artificially.

6. A feature synthesis device, characterized in that: include: A first acquisition unit, configured to acquire a configuration file input by a user, wherein the configuration file includes a storage path of original data; A second acquisition unit, configured to acquire the original data according to a storage path of the original data; A processing unit, configured to obtain feature definition information of the original data from the feature definition file when detecting that the user inputs the feature definition file; When the user input feature definition file is not detected, the user-defined operator file input by the user is obtained, and a first feature operator corresponding to the original data is generated according to the user-defined operator file, and / or, according to the second feature operator information specified by the user in the configuration file, the second feature operator is obtained, and the first feature operator corresponding to the original data is determined according to the second feature operator, and feature definition information of the original data is obtained according to the first feature operator corresponding to the original data and the original data, wherein the feature definition information is used to indicate feature calculation logic of the original data; A synthesis unit is used to send the feature definition information of the original data and the original data to a first computing device, so that the first computing device performs feature synthesis on the original data according to the feature definition information of the original data to obtain multiple synthetic features of the original data; or divide the original data into N sub-data, and send the feature definition information of the original data to each second computing device on N different second computing devices, and send the N sub-data one-to-one to the N second computing devices, so that each of the N second computing devices performs feature synthesis on the sub-data according to the sub-data on the second computing device and the feature definition information of the original data to obtain at least one synthetic feature of the sub-data, wherein the at least one synthetic feature of each of the sub-data constitutes the multiple synthetic features of the original data.

7. A computing device, characterized in that include: Memory, processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 5.

Images