Artificial intelligence-based feature data extraction method and device, and related equipment

Abstract

The application relates to the technical field of big data, and provides a feature data extraction method and device based on artificial intelligence and related equipment, the method comprising the following steps: logically processing a flow type data set according to a data warehouse type and a preset processing engine to generate a target flow type data set; performing first preprocessing and second preprocessing on the target flow type data set to obtain a second feature data set; performing first-order derivation on the second feature data set to obtain a third feature data set; performing feature scoring on each feature data in the third feature data set by using a plurality of preset evaluation systems to obtain a plurality of feature score values of each feature data, and then performing feature data extraction to obtain a feature data extraction result. After the feature scoring on each feature data by using the plurality of preset evaluation systems, the feature data extraction is performed, effective features are screened out, and the accuracy and effectiveness of the feature data extraction are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device and related equipment for extracting feature data based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence, machine learning methods have become widely used. This requires feature engineering of datasets. The effectiveness of feature engineering directly impacts the final performance of the resulting model. Traditional feature engineering often requires extensive manual intervention to obtain feature datasets.

[0003] However, manual intervention means that the extraction of feature data sets is limited to certain developed personal thinking spaces and cannot be truly "broad and deep", resulting in low integrity and accuracy of the extracted feature data sets.

[0004] Therefore, it is necessary to propose a method that can quickly and accurately extract feature data sets. Summary of the Invention

[0005] In view of the above content, it is necessary to propose a feature data extraction method, device and related equipment based on artificial intelligence. By adopting multiple preset evaluation systems to score each feature data and then extract the feature data, effective features are screened out, thereby improving the accuracy and effectiveness of feature data extraction.

[0006] A first aspect of the present invention provides a method for extracting feature data based on artificial intelligence, the method comprising:

[0007] Parsing the received feature data extraction request to obtain a pipeline data set and a data warehouse type corresponding to the pipeline data set;

[0008] Performing logical processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set;

[0009] Performing a first preprocessing on the target pipeline data set to obtain a first feature data set;

[0010] performing a second preprocessing on the first feature data set to obtain a second feature data set;

[0011] Performing a first-order derivative on the second feature data set using a preset first-order derivative algorithm to obtain a third feature data set;

[0012] Using a plurality of preset evaluation systems to perform feature scoring on each feature data in the third feature data set, to obtain a plurality of feature score values ​​for each feature data;

[0013] Based on the multiple feature score values ​​of each feature data, feature data extraction is performed on the third feature data set to obtain a feature data extraction result of the feature data extraction request.

[0014] Optionally, the performing logical processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set includes:

[0015] Obtaining a template identification code from the configuration requirements in the feature data extraction request;

[0016] Acquire a corresponding configuration template based on the template identification code, and select an aggregate primary key, a date field, a numeric field, and a character field according to the configuration template;

[0017] The pipeline data set is configured based on the aggregate primary key, the date field, the numeric field, and the character field to obtain a pipeline data set;

[0018] The processing logic is automatically generated according to the data warehouse type and the preset processing engine, and the processing logic is used to perform logical processing on the pipeline data set to generate a target pipeline data set.

[0019] Optionally, performing a first preprocessing on the target pipeline data set to obtain a first feature data set includes:

[0020] Performing a first division on the target pipeline data set according to a plurality of preset first field types to obtain a sub-data set of each preset first field type;

[0021] Using a preset operator corresponding to each first field type to perform operator processing on each sub-data in each preset sub-data set of the first field type, to obtain a preset feature data set of each first field type;

[0022] The plurality of feature data sets of the preset plurality of first field types are merged, and the merged plurality of feature data sets are determined as the first feature data set.

[0023] Optionally, performing a second preprocessing on the first feature data set to obtain a second feature data set includes:

[0024] The first feature data set is divided for the second time according to the preset multiple second field types to obtain a preset second feature sub-data set of each second field type, and the multiple second feature sub-data corresponding to the preset multiple second field types are determined as the second feature data set.

[0025] Optionally, the performing a first-order derivation on the second feature data set using a preset first-order derivation algorithm to obtain a third feature data set includes:

[0026] Extracting a preset second feature sub-data set of each second field type from the second feature set;

[0027] Performing a first-order derivation on each characteristic sub-data set in the second characteristic sub-data set corresponding to each preset second field type using a preset first-order derivation algorithm corresponding to each second field type, to obtain a preset third characteristic sub-data set of each second field type;

[0028] The plurality of third characteristic sub-datasets of the preset plurality of second field types are merged, and the merged plurality of third sub-datasets are determined as the third characteristic dataset.

[0029] Optionally, performing feature data extraction on the third feature data set based on the multiple feature score values ​​of each feature data to obtain the feature data extraction result of the feature data extraction request includes:

[0030] Using a preset weighting algorithm to perform weighted calculation on multiple feature score values ​​of each feature data in the third feature data set to obtain a weighted value of each feature data;

[0031] A plurality of feature data with larger weighted values ​​are selected from the calculated weighted values, and determined as the feature data extraction result of the feature data extraction request.

[0032] Optionally, the preset first field type includes a numerical field type, a category field type and a date field type, and the preset second field type includes a numerical type and a character type.

[0033] A second aspect of the present invention provides a feature data extraction device based on artificial intelligence, the device comprising:

[0034] A parsing and acquisition module, configured to parse a received feature data extraction request, and obtain a pipeline data set and a data warehouse type corresponding to the pipeline data set;

[0035] A logic processing module, configured to perform logic processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set;

[0036] A first preprocessing module, configured to perform a first preprocessing on the target pipeline data set to obtain a first feature data set;

[0037] A second preprocessing module, configured to perform a second preprocessing on the first feature data set to obtain a second feature data set;

[0038] a first-order derivative module, configured to perform a first-order derivative on the second feature data set using a preset first-order derivative algorithm to obtain a third feature data set;

[0039] a scoring module, configured to perform a feature scoring on each feature data in the third feature data set using a plurality of preset evaluation systems, and obtain a plurality of feature scoring values ​​for each feature data;

[0040] The extraction module is configured to perform feature data extraction on the third feature data set based on a plurality of feature score values ​​of each feature data to obtain a feature data extraction result of the feature data extraction request.

[0041] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the artificial intelligence-based feature data extraction method when executing a computer program stored in the memory.

[0042] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the artificial intelligence-based feature data extraction method when executed by a processor.

[0043] In summary, the artificial intelligence-based feature data extraction method, device, and related equipment of the present invention generate a target pipeline data set by performing logical processing on the pipeline data set according to the data warehouse type and the preset processing engine, thereby avoiding manual intervention and improving the efficiency and accuracy of generating the target pipeline data set. By performing a first preprocessing on the target pipeline data set to obtain a first feature data set, and using a preset first-order derivative algorithm to perform a first-order derivative on the second feature data set to obtain a third feature data set, the number of target pipeline data sets is expanded from different dimensions, the data dimensions of the third feature data set are further enriched, and the integrity of the third feature data set is improved. A plurality of preset evaluation systems are used to perform feature scoring on each feature data in the third feature data set, and after obtaining multiple feature score values ​​for each feature data, feature data is extracted. When calculating the feature score values, the third feature data set is used for model training to obtain multiple feature score values ​​after the model training is completed, thereby improving the accuracy of the feature score values, and screening out effective features based on the multiple feature score values, thereby improving the accuracy and effectiveness of the feature data extraction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of the feature data extraction method based on artificial intelligence provided in Example 1 of the present invention.

[0045] Figure 2This is a structural diagram of the feature data extraction device based on artificial intelligence provided in Example 2 of the present invention.

[0046] Figure 3 It is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] Example 1

[0050] Figure 1 This is a flowchart of the feature data extraction method based on artificial intelligence provided in Example 1 of the present invention.

[0051] In this embodiment, the artificial intelligence-based feature data extraction method can be applied to electronic devices. For electronic devices that need to perform artificial intelligence-based feature data extraction, the artificial intelligence-based feature data extraction function provided by the method of the present invention can be directly integrated into the electronic device, or run in the electronic device in the form of a software development kit (SDK).

[0052] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0053] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, machine learning, and deep learning.

[0054] like Figure 1As shown, the feature data extraction method based on artificial intelligence specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0055] S11, parsing the received feature data extraction request, obtaining a pipeline data set and a data warehouse type corresponding to the pipeline data set.

[0056] In this embodiment, when the user extracts feature data, he / she initiates a feature data extraction request to the server through the client. Specifically, the client can be a smart phone, IPAD or other existing smart devices, and the server can be a feature data extraction subsystem. During the feature data extraction process, the client can send a feature data extraction request to the feature data extraction subsystem, and the feature data extraction subsystem is used to receive the feature data extraction request sent by the client.

[0057] In this embodiment, when the feature data extraction subsystem receives the feature data extraction request, it parses the feature extraction request to obtain the pipeline data set and the data warehouse type corresponding to the pipeline data set. Specifically, the data warehouse type may be a hive data warehouse.

[0058] In this embodiment, the transaction flow data set refers to a transaction flow type data set, for example, transaction flow data for a credit card, which contains transaction records, transaction time, repayment information and other transaction flow data.

[0059] In an optional embodiment, parsing the received feature data extraction request to obtain the pipeline data set includes:

[0060] Parse the received feature data extraction request and obtain multiple data call requests;

[0061] Obtaining a corresponding calling interface according to each of the data calling requests;

[0062] According to the multiple data call requests and the corresponding call interfaces, a query link call interface list is formed;

[0063] Starting from the head of the call interface list in the query phase, each call interface is called in sequence to obtain the pipeline data corresponding to each call interface, and the multiple pipeline data corresponding to the multiple data call requests are determined as a pipeline data set.

[0064] In this embodiment, it supports obtaining a stream-type data set from a relational database and a hive data warehouse, wherein the relational database may include a MySQL database, an Oracle database, and other relational databases that can obtain a stream-type data set.

[0065] In this embodiment, the feature extraction request may include a data calling interface. Specifically, the data calling interface refers to the calling interface of the database. Different databases have different corresponding calling interfaces. By obtaining the corresponding pipeline data from the corresponding calling interface, it is targeted and improves the accuracy and efficiency of the obtained pipeline data set.

[0066] S12, performing logical processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set.

[0067] In this embodiment, the feature extraction request further includes configuration requirements, wherein the configuration requirements may include other requirements such as aggregation dimension requirements of pipeline data and pipeline time requirements.

[0068] In an optional embodiment, the logically processing the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set includes:

[0069] Obtaining a template identification code from the configuration requirements in the feature data extraction request;

[0070] Acquire a corresponding configuration template based on the template identification code, and select an aggregate primary key, a date field, a numeric field, and a character field according to the configuration template;

[0071] The pipeline data set is configured based on the aggregate primary key, the date field, the numeric field, and the character field to obtain a pipeline data set;

[0072] The processing logic is automatically generated according to the data warehouse type and the preset processing engine, and the processing logic is used to perform logical processing on the pipeline data set to generate a target pipeline data set.

[0073] In this embodiment, the template identification code uniquely identifies a configuration template, which refers to a template pre-set by the user for configuring data. Based on the configuration template, a corresponding aggregate primary key, date field, numeric field, and character field can be selected. The aggregate primary key refers to the target field for aggregate statistics corresponding to each configuration template. For example, for a streamlined dataset of customer A, if aggregate statistics are performed based on customer A's card number, the aggregate primary key is customer A's card number.

[0074] In this embodiment, the preset processing engine may be a spark computing engine, a hive computing engine, a presto computing engine or other offline big data computing engines.

[0075] In this embodiment, the data acquisition range of stream data supports partitioned data acquisition and custom filtering conditions. For example, when selecting data based on a date field, data can be acquired by partition or by custom filtering conditions.

[0076] For example, when it is necessary to perform aggregate statistics based on the flow data of a time period, the time period can be selected according to the configuration requirements, such as data from the past week, the past month, or the past three months. If the date of the flow data is accurate to the hour, minute, and second, the calculation can be further accurately divided. For example, various types of fields between 9 and 10 o'clock every day in the past month can be selected for logical processing and calculation.

[0077] In this embodiment, the aggregate primary key, date field, numeric field and character field are selected through the configuration template to perform pipeline data configuration. After the configuration is completed, the preset processing engine will be executed according to the data warehouse type, and the corresponding processing logic will be automatically generated. For example, the Hive data warehouse calls the Hive computing engine, automatically generates Hive SQL processing logic, and submits the processing task to the Hive data warehouse cluster for data processing.

[0078] In this embodiment, by automatically and efficiently performing logical processing on the pipeline data set according to the data warehouse type and a preset processing engine, manual intervention is avoided and the efficiency and accuracy of generating the target pipeline data set are improved.

[0079] S13: Perform a first preprocessing on the target pipeline data set to obtain a first feature data set.

[0080] In this embodiment, the first preprocessing includes dividing and performing operator processing on the target pipeline data.

[0081] In an optional embodiment, performing a first preprocessing on the target pipeline data set to obtain a first feature data set includes:

[0082] Performing a first division on the target pipeline data set according to a plurality of preset first field types to obtain a sub-data set of each preset first field type;

[0083] Using a preset operator corresponding to each first field type to perform operator processing on each sub-data in each preset sub-data set of the first field type, to obtain a preset feature data set of each first field type;

[0084] The plurality of feature data sets of the preset plurality of first field types are merged, and the merged plurality of feature data sets are determined as the first feature data set.

[0085] In this embodiment, a plurality of first field types may be preset. Specifically, the preset first field types may include a numerical field type, a category field type, and a date field type.

[0086] In this embodiment, each first field type corresponds to different operator processing. Specifically, the operators corresponding to the numerical field type include: maximum value, minimum value, median, mean, standard deviation, coefficient of variation, sum and kurtosis, etc.; the category field type operators include: number of categories, highest frequency, top2, top3, lowest frequency, total amount and null value amount, etc.; the date field type operators include: day of the month, day of the week, month, highest frequency date, highest frequency day of the week and highest frequency month, etc.

[0087] In this embodiment, the target pipeline data set is subjected to a first preprocessing. During the first preprocessing process, corresponding operators are used to perform operator processing on each sub-data set of each preset sub-data set of the first field type according to different field types, thereby expanding the number of target pipeline data sets from different dimensions, thereby obtaining a first feature data set, and improving the integrity of the first feature data set.

[0088] S14: Perform a second preprocessing on the first feature data set to obtain a second feature data set.

[0089] In this embodiment, the second preprocessing includes performing a second division on the data of the first feature data set. Specifically, performing the second preprocessing on the first feature data set to obtain the second feature data set includes:

[0090] The first feature data set is divided for the second time according to the preset multiple second field types to obtain a preset second feature sub-data set of each second field type, and the multiple second feature sub-data corresponding to the preset multiple second field types are determined as the second feature data set.

[0091] In this embodiment, the second field type may be preset. Specifically, the preset second field type may include a numeric type and a character type.

[0092] In this embodiment, since the feature data corresponding to the numerical field type, category field type and date field type in the first feature data contains both numerical data and character data, the first feature data set is subjected to a second preprocessing according to the preset multiple second field types to facilitate the subsequent first-order derivative processing of the second feature data set.

[0093] S15: Perform a first-order derivation on the second feature data set using a preset first-order derivation algorithm to obtain a third feature data set.

[0094] In this embodiment, the first-order derivation refers to further dimensional expansion of the second feature data set.

[0095] In an optional embodiment, performing a first-order derivative on the second feature data set using a preset first-order derivative algorithm to obtain a third feature data set includes:

[0096] Extracting a preset second feature sub-data set of each second field type from the second feature set;

[0097] Performing a first-order derivation on each characteristic sub-data set in the second characteristic sub-data set corresponding to each preset second field type using a preset first-order derivation algorithm corresponding to each second field type, to obtain a preset third characteristic sub-data set of each second field type;

[0098] The plurality of third characteristic sub-datasets of the preset plurality of second field types are merged, and the merged plurality of third sub-datasets are determined as the third characteristic dataset.

[0099] In this embodiment, the first-order derivative algorithms corresponding to each second field type are different. Specifically, the first-order derivative algorithms corresponding to the numerical type include: logarithmic transformation, exponential transformation, square transformation, cubic transformation, upward and downward truncation, etc.; the first-order derivative algorithms corresponding to the character type include: Woe and onehot encoding, etc., among which Woe (Weight of Evidence) is often used for feature transformation.

[0100] In this embodiment, for different second field types, different first-order derivatives are further performed on the feature data in the second feature data set, and the number of target pipeline data sets is secondarily expanded from different dimensions, thereby further enriching the data dimensions of the third feature data set and improving the integrity of the third feature data set.

[0101] S16, using a plurality of preset evaluation systems to perform feature scoring on each feature data in the third feature data set, to obtain a plurality of feature scoring values ​​for each feature data.

[0102] In this embodiment, an evaluation system can be set in advance, and the preset evaluation system may include: a random forest algorithm, an information value algorithm and a correlation coefficient algorithm. Model training is performed by inputting the target label value in the third feature data set and the feature data extraction request into the random forest algorithm, the information value algorithm and the correlation coefficient algorithm respectively. After the training is completed, the random forest coefficient of the random forest algorithm, the iv (Information Value) value coefficient in the information value algorithm and the correlation coefficient in the correlation coefficient algorithm are obtained. Among them, the training processes of the random forest algorithm, the information value algorithm and the correlation coefficient algorithm are all existing technologies, and this embodiment will not be described in detail here.

[0103] In this embodiment, the feature score values ​​refer to the random forest coefficient, the iv (Information Value) value coefficient and the correlation coefficient.

[0104] S17: Perform feature data extraction on the third feature data set based on the multiple feature score values ​​of each feature data to obtain a feature data extraction result of the feature data extraction request.

[0105] In this embodiment, the feature data extraction result includes valid feature data filtered based on the multiple feature score values.

[0106] In an optional embodiment, performing feature data extraction on the third feature data set based on the multiple feature score values ​​of each feature data to obtain the feature data extraction result of the feature data extraction request includes:

[0107] Using a preset weighting algorithm to perform weighted calculation on multiple feature score values ​​of each feature data in the third feature data set to obtain a weighted value of each feature data;

[0108] A plurality of feature data with larger weighted values ​​are selected from the calculated weighted values, and determined as the feature data extraction result of the feature data extraction request.

[0109] In this embodiment, a weighted algorithm may be preset. Specifically, the preset weighted algorithm may be a weighted random algorithm, a weighted average method, etc., wherein the weighted random algorithm and the weighted average method are both existing technologies and will not be described in detail in this embodiment.

[0110] In this embodiment, for different second field types, different first-order derivations are further performed on the feature data in the second feature data set, thereby enriching the third feature data set. When the feature score value is subsequently calculated, the third feature data set is used for model training, and multiple feature score values ​​are obtained after the model training is completed, thereby improving the accuracy of the feature score value, and screening out effective features based on the multiple feature score values, thereby improving the accuracy and effectiveness of the feature data extraction results.

[0111] In summary, the artificial intelligence-based feature data extraction method described in this embodiment generates a target pipeline data set by performing logical processing on the pipeline data set according to the data warehouse type and the preset processing engine, thereby avoiding manual intervention and improving the efficiency and accuracy of generating the target pipeline data set. By performing a first preprocessing on the target pipeline data set to obtain a first feature data set, and using a preset first-order derivative algorithm to perform a first-order derivative on the second feature data set to obtain a third feature data set, the number of target pipeline data sets is expanded from different dimensions, the data dimensions of the third feature data set are further enriched, and the integrity of the third feature data set is improved. A plurality of preset evaluation systems are used to perform feature scoring on each feature data in the third feature data set, and after obtaining multiple feature score values ​​for each feature data, feature data is extracted. When calculating the feature score values, the third feature data set is used for model training to obtain multiple feature score values ​​after the model training is completed, thereby improving the accuracy of the feature score values, and screening out effective features based on the multiple feature score values, thereby improving the accuracy and effectiveness of the feature data extraction results.

[0112] Example 2

[0113] Figure 2 This is a structural diagram of the feature data extraction device based on artificial intelligence provided in Example 2 of the present invention.

[0114] In some embodiments, the feature data extraction device 20 based on artificial intelligence may include a plurality of functional modules composed of program code segments. The program code of each program segment in the feature data extraction device 20 based on artificial intelligence may be stored in the memory of the electronic device and executed by the at least one processor to perform (see Figure 1 Description) Function of feature data extraction based on artificial intelligence.

[0115] In this embodiment, the artificial intelligence-based feature data extraction device 20 can be divided into multiple functional modules based on the functions it performs. These functional modules may include: a parsing and acquisition module 201, a logic processing module 202, a first preprocessing module 203, a second preprocessing module 204, a first-order derivative module 205, a scoring module 206, and an extraction module 207. As used herein, a module refers to a series of computer-readable instruction segments that can be executed by at least one processor and can perform a fixed function, and is stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0116] The parsing and obtaining module 201 is used to parse the received feature data extraction request and obtain the pipeline data set and the data warehouse type corresponding to the pipeline data set.

[0117] In this embodiment, when the user extracts feature data, he / she initiates a feature data extraction request to the server through the client. Specifically, the client can be a smart phone, IPAD or other existing smart devices, and the server can be a feature data extraction subsystem. During the feature data extraction process, the client can send a feature data extraction request to the feature data extraction subsystem, and the feature data extraction subsystem is used to receive the feature data extraction request sent by the client.

[0118] In this embodiment, when the feature data extraction subsystem receives the feature data extraction request, it parses the feature extraction request to obtain the pipeline data set and the data warehouse type corresponding to the pipeline data set. Specifically, the data warehouse type may be a hive data warehouse.

[0119] In this embodiment, the transaction flow data set refers to a transaction flow type data set, for example, transaction flow data for a credit card, which contains transaction records, transaction time, repayment information and other transaction flow data.

[0120] In an optional embodiment, the parsing and obtaining module 201 parses the received feature data extraction request to obtain the pipeline data set, including:

[0121] Parse the received feature data extraction request and obtain multiple data call requests;

[0122] Obtaining a corresponding calling interface according to each of the data calling requests;

[0123] According to the multiple data call requests and the corresponding call interfaces, a query link call interface list is formed;

[0124] Starting from the head of the call interface list in the query phase, each call interface is called in sequence to obtain the pipeline data corresponding to each call interface, and the multiple pipeline data corresponding to the multiple data call requests are determined as a pipeline data set.

[0125] In this embodiment, it supports obtaining a stream-type data set from a relational database and a hive data warehouse, wherein the relational database may include a MySQL database, an Oracle database, and other relational databases that can obtain a stream-type data set.

[0126] In this embodiment, the feature extraction request may include a data calling interface. Specifically, the data calling interface refers to the calling interface of the database. Different databases have different corresponding calling interfaces. By obtaining the corresponding pipeline data from the corresponding calling interface, it is targeted and improves the accuracy and efficiency of the obtained pipeline data set.

[0127] The logic processing module 202 is used to perform logic processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set.

[0128] In this embodiment, the feature extraction request further includes configuration requirements, wherein the configuration requirements may include other requirements such as aggregation dimension requirements of pipeline data and pipeline time requirements.

[0129] In an optional embodiment, the logic processing module 202 performs logic processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set, including:

[0130] Obtaining a template identification code from the configuration requirements in the feature data extraction request;

[0131] Acquire a corresponding configuration template based on the template identification code, and select an aggregate primary key, a date field, a numeric field, and a character field according to the configuration template;

[0132] The pipeline data set is configured based on the aggregate primary key, the date field, the numeric field, and the character field to obtain a pipeline data set;

[0133] The processing logic is automatically generated according to the data warehouse type and the preset processing engine, and the processing logic is used to perform logical processing on the pipeline data set to generate a target pipeline data set.

[0134] In this embodiment, the template identification code uniquely identifies a configuration template, which refers to a template pre-set by the user for configuring data. Based on the configuration template, a corresponding aggregate primary key, date field, numeric field, and character field can be selected. The aggregate primary key refers to the target field for aggregate statistics corresponding to each configuration template. For example, for a streamlined dataset of customer A, if aggregate statistics are performed based on customer A's card number, the aggregate primary key is customer A's card number.

[0135] In this embodiment, the preset processing engine may be a spark computing engine, a hive computing engine, a presto computing engine or other offline big data computing engines.

[0136] In this embodiment, the data acquisition range of stream data supports partitioned data acquisition and custom filtering conditions. For example, when selecting data based on a date field, data can be acquired by partition or by custom filtering conditions.

[0137] For example, when it is necessary to perform aggregate statistics based on the flow data of a time period, the time period can be selected according to the configuration requirements, such as data from the past week, the past month, or the past three months. If the date of the flow data is accurate to the hour, minute, and second, the calculation can be further accurately divided. For example, various types of fields between 9 and 10 o'clock every day in the past month can be selected for logical processing and calculation.

[0138] In this embodiment, the aggregate primary key, date field, numeric field and character field are selected through the configuration template to perform pipeline data configuration. After the configuration is completed, the preset processing engine will be executed according to the data warehouse type, and the corresponding processing logic will be automatically generated. For example, the Hive data warehouse calls the Hive computing engine, automatically generates Hive SQL processing logic, and submits the processing task to the Hive data warehouse cluster for data processing.

[0139] In this embodiment, by automatically and efficiently performing logical processing on the pipeline data set according to the data warehouse type and a preset processing engine, manual intervention is avoided and the efficiency and accuracy of generating the target pipeline data set are improved.

[0140] The first preprocessing module 203 is configured to perform a first preprocessing on the target pipeline data set to obtain a first feature data set.

[0141] In this embodiment, the first preprocessing includes dividing and performing operator processing on the target pipeline data.

[0142] In an optional embodiment, the first preprocessing module 203 performs a first preprocessing on the target pipeline data set to obtain a first feature data set including:

[0143] Performing a first division on the target pipeline data set according to a plurality of preset first field types to obtain a sub-data set of each preset first field type;

[0144] Using a preset operator corresponding to each first field type to perform operator processing on each sub-data in each preset sub-data set of the first field type, to obtain a preset feature data set of each first field type;

[0145] The plurality of feature data sets of the preset plurality of first field types are merged, and the merged plurality of feature data sets are determined as the first feature data set.

[0146] In this embodiment, a plurality of first field types may be preset. Specifically, the preset first field types may include a numerical field type, a category field type, and a date field type.

[0147] In this embodiment, each first field type corresponds to different operator processing. Specifically, the operators corresponding to the numerical field type include: maximum value, minimum value, median, mean, standard deviation, coefficient of variation, sum and kurtosis, etc.; the category field type operators include: number of categories, highest frequency, top2, top3, lowest frequency, total amount and null value amount, etc.; the date field type operators include: day of the month, day of the week, month, highest frequency date, highest frequency day of the week and highest frequency month, etc.

[0148] In this embodiment, the target pipeline data set is subjected to a first preprocessing. During the first preprocessing process, corresponding operators are used to perform operator processing on each sub-data set of each preset sub-data set of the first field type according to different field types, thereby expanding the number of target pipeline data sets from different dimensions, thereby obtaining a first feature data set, and improving the integrity of the first feature data set.

[0149] The second preprocessing module 204 is configured to perform a second preprocessing on the first feature data set to obtain a second feature data set.

[0150] In this embodiment, the second preprocessing includes performing a second division on the data of the first feature data set. Specifically, the second preprocessing module 204 performs a second preprocessing on the first feature data set to obtain a second feature data set including:

[0151] The first feature data set is divided for the second time according to the preset multiple second field types to obtain a preset second feature sub-data set of each second field type, and the multiple second feature sub-data corresponding to the preset multiple second field types are determined as the second feature data set.

[0152] In this embodiment, the second field type may be preset. Specifically, the preset second field type may include a numeric type and a character type.

[0153] In this embodiment, since the feature data corresponding to the numerical field type, category field type and date field type in the first feature data contains both numerical data and character data, the first feature data set is subjected to a second preprocessing according to the preset multiple second field types to facilitate the subsequent first-order derivative processing of the second feature data set.

[0154] The first-order derivative module 205 is configured to perform a first-order derivative on the second feature data set using a preset first-order derivative algorithm to obtain a third feature data set.

[0155] In this embodiment, the first-order derivation refers to further dimensional expansion of the second feature data set.

[0156] In an optional embodiment, the first-order derivation module 205 performs a first-order derivation on the second feature data set using a preset first-order derivation algorithm to obtain a third feature data set including:

[0157] Extracting a preset second feature sub-data set of each second field type from the second feature set;

[0158] Performing a first-order derivation on each characteristic sub-data set in the second characteristic sub-data set corresponding to each preset second field type using a preset first-order derivation algorithm corresponding to each second field type, to obtain a preset third characteristic sub-data set of each second field type;

[0159] The plurality of third characteristic sub-datasets of the preset plurality of second field types are merged, and the merged plurality of third sub-datasets are determined as the third characteristic dataset.

[0160] In this embodiment, the first-order derivative algorithms corresponding to each second field type are different. Specifically, the first-order derivative algorithms corresponding to the numerical type include: logarithmic transformation, exponential transformation, square transformation, cubic transformation, upward and downward truncation, etc.; the first-order derivative algorithms corresponding to the character type include: Woe and onehot encoding, etc., among which Woe (Weight of Evidence) is often used for feature transformation.

[0161] In this embodiment, for different second field types, different first-order derivatives are further performed on the feature data in the second feature data set, and the number of target pipeline data sets is secondarily expanded from different dimensions, thereby further enriching the data dimensions of the third feature data set and improving the integrity of the third feature data set.

[0162] The scoring module 206 is configured to perform a feature scoring on each feature data in the third feature data set using a plurality of preset evaluation systems to obtain a plurality of feature scoring values ​​for each feature data.

[0163] In this embodiment, an evaluation system can be set in advance, and the preset evaluation system may include: a random forest algorithm, an information value algorithm and a correlation coefficient algorithm. Model training is performed by inputting the target label value in the third feature data set and the feature data extraction request into the random forest algorithm, the information value algorithm and the correlation coefficient algorithm respectively. After the training is completed, the random forest coefficient of the random forest algorithm, the iv (Information Value) value coefficient in the information value algorithm and the correlation coefficient in the correlation coefficient algorithm are obtained. Among them, the training processes of the random forest algorithm, the information value algorithm and the correlation coefficient algorithm are all existing technologies, and this embodiment will not be described in detail here.

[0164] In this embodiment, the feature score values ​​refer to the random forest coefficient, the iv (Information Value) value coefficient and the correlation coefficient.

[0165] The extraction module 207 is configured to perform feature data extraction on the third feature data set based on multiple feature score values ​​of each feature data to obtain a feature data extraction result of the feature data extraction request.

[0166] In this embodiment, the feature data extraction result includes valid feature data filtered based on the multiple feature score values.

[0167] In an optional embodiment, the extraction module 207 performs feature data extraction on the third feature data set based on multiple feature score values ​​of each feature data, and obtains the feature data extraction result of the feature data extraction request including:

[0168] Using a preset weighting algorithm to perform weighted calculation on multiple feature score values ​​of each feature data in the third feature data set to obtain a weighted value of each feature data;

[0169] A plurality of feature data with larger weighted values ​​are selected from the calculated weighted values, and determined as the feature data extraction result of the feature data extraction request.

[0170] In this embodiment, a weighted algorithm may be preset. Specifically, the preset weighted algorithm may be a weighted random algorithm, a weighted average method, etc., wherein the weighted random algorithm and the weighted average method are both existing technologies and will not be described in detail in this embodiment.

[0171] In this embodiment, for different second field types, different first-order derivations are further performed on the feature data in the second feature data set, thereby enriching the third feature data set. When the feature score value is subsequently calculated, the third feature data set is used for model training, and multiple feature score values ​​are obtained after the model training is completed, thereby improving the accuracy of the feature score value, and screening out effective features based on the multiple feature score values, thereby improving the accuracy and effectiveness of the feature data extraction results.

[0172] In summary, the artificial intelligence-based feature data extraction device described in this embodiment generates a target pipeline data set by performing logical processing on the pipeline data set according to the data warehouse type and the preset processing engine, thereby avoiding manual intervention and improving the efficiency and accuracy of generating the target pipeline data set. By performing a first preprocessing on the target pipeline data set to obtain a first feature data set, and using a preset first-order derivative algorithm to perform a first-order derivative on the second feature data set to obtain a third feature data set, the number of target pipeline data sets is expanded from different dimensions, the data dimensions of the third feature data set are further enriched, and the integrity of the third feature data set is improved. A plurality of preset evaluation systems are used to perform feature scoring on each feature data in the third feature data set, and after obtaining multiple feature score values ​​for each feature data, feature data is extracted. When calculating the feature score values, the third feature data set is used for model training to obtain multiple feature score values ​​after the model training is completed, thereby improving the accuracy of the feature score values, and screening out effective features based on the multiple feature score values, thereby improving the accuracy and effectiveness of the feature data extraction results.

[0173] Example 3

[0174] See Figure 3 FIG. 3 is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .

[0175] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention, and can be either a bus structure or a star structure. The electronic device 3 can also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0176] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include a client device, which includes but is not limited to any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice-controlled device, such as a personal computer, tablet computer, smartphone, digital camera, etc.

[0177] It should be noted that the electronic device 3 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0178] In some embodiments, the memory 31 is used to store program codes and various data, such as the artificial intelligence-based feature data extraction device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0179] In some embodiments, the at least one processor 32 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The at least one processor 32 is the control core (Control Unit) of the electronic device 3, connecting the various components of the entire electronic device 3 using various interfaces and lines, and executing or executing programs or modules stored in the memory 31, as well as calling data stored in the memory 31, to perform various functions of the electronic device 3 and process data.

[0180] In some embodiments, the at least one communication bus 33 is configured to implement connection and communication between the memory 31 and the at least one processor 32 .

[0181] Although not shown, the electronic device 3 may also include a power source (such as a battery) to power various components. Optionally, the power source may be logically connected to the at least one processor 32 via a power management device, thereby enabling the power management device to manage charging, discharging, and power consumption. The power source may also include one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be detailed here.

[0182] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0183] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module stored in a storage medium includes a number of instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) or a processor to execute portions of the methods described in various embodiments of the present invention.

[0184] In a further embodiment, in combination Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed applications (such as the artificial intelligence-based feature data extraction device 20), program codes, etc., for example, the above-mentioned modules.

[0185] The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to execute related functions. For example, Figure 2 The modules described are program codes stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of feature data extraction based on artificial intelligence.

[0186] Exemplarily, the program code may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to implement the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which are used to describe the execution process of the program code in the electronic device 3. For example, the program code may be divided into a parsing and acquisition module 201, a logic processing module 202, a first preprocessing module 203, a second preprocessing module 204, a first-order derivative module 205, a scoring module 206, and an extraction module 207.

[0187] In one embodiment of the present invention, the memory 31 stores a plurality of computer-readable instructions, and the plurality of computer-readable instructions are executed by the at least one processor 32 to implement a function of extracting feature data based on artificial intelligence.

[0188] Specifically, the specific implementation method of the at least one processor 32 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0189] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and actual implementation may employ other division methods.

[0190] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, and may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0191] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0192] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive from any point of view, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all variations that fall within the meaning and scope of the equivalents of the claims be encompassed within the present invention. Any reference numerals in the claims should not be construed as limiting the claims to which they relate. Furthermore, it is apparent that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. The multiple units or devices set forth in the present invention may also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A feature data extraction method based on artificial intelligence, characterized in that: The method comprises: Parsing the received feature data extraction request to obtain a pipeline data set and a data warehouse type corresponding to the pipeline data set, wherein the parsing of the received feature data extraction request to obtain the pipeline data set includes: parsing the received feature data extraction request to obtain multiple data call requests; obtaining a corresponding call interface according to each of the data call requests, wherein the call interface refers to a call interface of a database; forming a query link call interface list according to the multiple data call requests and the corresponding call interfaces; starting from the head of the query link call interface list, sequentially calling each of the call interfaces to obtain the pipeline data corresponding to each of the call interfaces, and determining the multiple pipeline data corresponding to the multiple data call requests as the pipeline data set; logically processing the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set, including: obtaining a template identification code from the configuration requirements in the feature data extraction request; obtaining a corresponding configuration template based on the template identification code, and selecting an aggregate primary key, a date field, a numeric field, and a character field according to the configuration template; configuring the pipeline data set based on the aggregate primary key, date field, numeric field, and character field to obtain a pipeline data set; automatically generating processing logic according to the data warehouse type and a preset processing engine, and using the processing logic to logically process the pipeline data set to generate a target pipeline data set; performing a first preprocessing on the target pipeline data set to obtain a first feature data set, including: performing a first division on the target pipeline data set according to a plurality of preset first field types to obtain a sub-data set of each preset first field type; performing operator processing on each sub-data in the sub-data set of each preset first field type using an operator corresponding to each of the preset first field types to obtain a preset feature data set of each first field type; merging a plurality of feature data sets of the plurality of preset first field types, and determining the merged plurality of feature data sets as the first feature data set; performing a second preprocessing on the first feature data set to obtain a second feature data set, including: performing a second division on the first feature data set according to a plurality of preset second field types to obtain a second feature sub-data set of each preset second field type, and determining a plurality of second feature sub-data corresponding to the plurality of preset second field types as the second feature data set, wherein the second field types include a numeric type and a character type; Performing a first-order derivation on the second feature data set using a preset first-order derivation algorithm to obtain a third feature data set, including: extracting a preset second feature sub-data set of each second field type from the second feature set; performing a first-order derivation on each feature sub-data in the second feature sub-data set corresponding to each second field type using a preset first-order derivation algorithm corresponding to each second field type to obtain a preset third feature sub-data set of each second field type, wherein the first-order derivation algorithm corresponding to each second field type is different; merging multiple third feature sub-data sets of the preset multiple second field types, and determining the merged multiple third sub-data sets as the third feature data set; Using a plurality of preset evaluation systems to perform feature scoring on each feature data in the third feature data set, to obtain a plurality of feature score values ​​for each feature data; Based on the multiple feature score values ​​of each feature data, feature data extraction is performed on the third feature data set to obtain a feature data extraction result of the feature data extraction request.

2. The method for extracting feature data based on artificial intelligence according to claim 1, wherein: The extracting feature data from the third feature data set based on the multiple feature score values ​​of each feature data to obtain the feature data extraction result of the feature data extraction request includes: Using a preset weighting algorithm to perform weighted calculation on multiple feature score values ​​of each feature data in the third feature data set to obtain a weighted value of each feature data; A plurality of feature data with larger weighted values ​​are selected from the calculated weighted values, and determined as the feature data extraction result of the feature data extraction request.

3. The feature data extraction method based on artificial intelligence according to claim 1, characterized in that: The preset first field types include a numerical field type, a category field type, and a date field type, and the preset second field types include a numerical type and a character type.

4. A feature data extraction device based on artificial intelligence, characterized in that: The device comprises: The parsing and acquisition module is used to parse the received feature data extraction request, obtain the pipeline data set and the data warehouse type corresponding to the pipeline data set, wherein the parsing of the received feature data extraction request and the acquisition of the pipeline data set include: parsing the received feature data extraction request to obtain multiple data call requests; obtaining a corresponding call interface according to each of the data call requests, wherein the call interface refers to the call interface of the database; forming a query link call interface list according to the multiple data call requests and the corresponding call interfaces; starting from the head of the query link call interface list, calling each of the call interfaces in sequence to obtain the pipeline data corresponding to each of the call interfaces, and determining the multiple pipeline data corresponding to the multiple data call requests as the pipeline data set; a logic processing module, configured to perform logic processing on the pipeline data set according to the data warehouse type and a preset processing engine to generate a target pipeline data set, including: obtaining a template identification code from the configuration requirements in the feature data extraction request; obtaining a corresponding configuration template based on the template identification code, and selecting an aggregate primary key, a date field, a numeric field, and a character field according to the configuration template; configuring the pipeline data set based on the aggregate primary key, date field, numeric field, and character field to obtain a pipeline data set; automatically generating processing logic according to the data warehouse type and a preset processing engine, and using the processing logic to perform logic processing on the pipeline data set to generate a target pipeline data set; A first preprocessing module is configured to perform a first preprocessing on the target pipeline data set to obtain a first feature data set, including: performing a first division on the target pipeline data set according to a plurality of preset first field types to obtain a sub-data set of each preset first field type; performing operator processing on each sub-data in the sub-data set of each preset first field type using an operator corresponding to each of the preset first field types to obtain a feature data set of each preset first field type; merging a plurality of feature data sets of the plurality of preset first field types, and determining the merged plurality of feature data sets as the first feature data set; A second preprocessing module is configured to perform a second preprocessing on the first feature data set to obtain a second feature data set, including: performing a second division on the first feature data set according to a plurality of preset second field types to obtain a second feature sub-data set of each preset second field type, and determining a plurality of second feature sub-data corresponding to the plurality of preset second field types as the second feature data set, wherein the second field types include numeric and character types; a first-order derivation module, configured to perform a first-order derivation on the second feature data set using a preset first-order derivation algorithm to obtain a third feature data set, including: extracting a preset second feature sub-data set of each second field type from the second feature set; performing a first-order derivation on each feature sub-data in the second feature sub-data set corresponding to each preset second field type using a preset first-order derivation algorithm corresponding to each second field type to obtain a preset third feature sub-data set of each second field type, wherein the first-order derivation algorithm corresponding to each second field type is different; merging multiple third feature sub-data sets of the preset multiple second field types, and determining the merged multiple third sub-data sets as the third feature data set; a scoring module, configured to perform a feature scoring on each feature data in the third feature data set using a plurality of preset evaluation systems, and obtain a plurality of feature scoring values ​​for each feature data; The extraction module is configured to perform feature data extraction on the third feature data set based on a plurality of feature score values ​​of each feature data to obtain a feature data extraction result of the feature data extraction request.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is used to implement the artificial intelligence-based feature data extraction method as described in any one of claims 1 to 3 when executing the computer program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the artificial intelligence-based feature data extraction method according to any one of claims 1 to 3 is implemented.

Images