Risk feature backtracking method and device based on big data platform and medium

Abstract

The invention relates to the technical field of data processing, in particular to a risk feature backtracking method based on a big data platform, and the method comprises the following steps: S1, taking historical detail data as source data, and storing the source data in a steady state library; s2, processing the source data to obtain configuration features, and storing the configuration features in a sensitive state library; s3, constructing a backtracking feature list based on the configuration features; s4, constructing a feature backtracking task based on the backtracking feature list; and S5, executing a feature backtracking task to obtain a backtracking result, and completing feature backtracking. The method has the advantages that the source data is processed, the configuration data is acquired and stored in the database, the business can timely adjust the feature business logic caliber through the front-end function, the feature backtracking front-end function supports selection of the backtracking feature range and the backtracking time end, the backtracking feature is updated in real time by the background through the timed task, and the backtracking efficiency is improved. And a business field data table is generated, so that a backtracking task is conveniently established to realize feature backtracking.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a risk feature backtracking method, device and medium based on a big data platform. Background Art

[0002] Feature backtracking is an important function in the big data platform and is used in various business scenarios. The existing feature backtracking has the following defects: 1. The backtracking data link dependency judgment requires offline agreement. If the upstream data has not been processed, adding a new backtracking task will be problematic. 2. When the backtracking task involves too many features or the features have too many dependencies, opening multiple threads concurrently to execute the backtracking task will cause excessive CPU pressure.

[0003] In summary, there is an urgent need for a risk feature backtracking method, equipment and medium based on a big data platform to solve the problems of the existing technology. Summary of the invention

[0004] The purpose of the present invention is to provide a risk feature backtracking method, device and medium based on a big data platform. The specific technical solution is as follows:

[0005] A risk feature backtracking method based on a big data platform comprises the following steps:

[0006] S1: Save the historical detailed data as source data into the steady-state database;

[0007] S2: Process the source data to obtain configuration features, and save the configuration features into the agile database;

[0008] S3: Build a backtracking feature list based on the configuration features;

[0009] S4: Construct feature backtracking task based on the backtracking feature list;

[0010] S5: Execute the feature backtracking task, obtain the backtracking result, and complete the feature backtracking.

[0011] Preferably, the process of processing the source data is as follows:

[0012] According to different business fields, the historical detailed data is divided into business field data tables, the data fields in the business field data tables are set as atomic features, and the atomic features are configured with feature results to obtain the first feature.

[0013] Preferably, the first feature includes a derived feature, an SQL feature and a composite feature;

[0014] The derived features are calculated from atomic features;

[0015] The composite feature is calculated from the derived feature;

[0016] The SQL feature is calculated by combining the composite feature and the derived feature.

[0017] Preferably, the process of processing the source data also includes setting a scheduled task, which is executed according to a set period, specifically:

[0018] Read the first feature and business domain data table to check whether there are any new features;

[0019] When no new features are added or offline, the first feature is put into a temporary table, assembled into json format through a function, and then the GET_JSON_OBJECT function of HSQL is used to parse the json data and insert it into the result table, which is a business domain data table;

[0020] When a feature is added or taken offline, determine whether the business domain to which the newly added feature belongs exists. If the business domain already exists, directly add a column named after the first feature in the business domain data table. If the business domain does not exist, create a new business domain data table corresponding to the business domain, and add a column named after the feature in the business domain data table.

[0021] Preferably, the scheduled task further includes querying whether there is a feature offline, and for a failed first feature, setting the value of the first feature in the business domain data table to null.

[0022] Preferably, the backtracking feature list includes the backtracking features and the time range corresponding to the backtracking features. The process of constructing the feature backtracking task based on the backtracking feature list is as follows:

[0023] The user sets the backtracking features and time range to complete the task of constructing feature backtracking.

[0024] Preferably, in the process of executing the feature backtracking task, backtracking of multiple backtracking features is performed synchronously based on multiple threads.

[0025] Preferably, the backtracking result includes the backtracking feature and the time range. After the feature backtracking is completed, the backtracking result is recalled to the agile library for storage for real-time viewing by the user.

[0026] In addition, the present invention also includes a computer device, including a memory and a processor;

[0027] The memory is used to store a computer program executable on the processor;

[0028] The processor is used to implement the steps of the risk feature backtracking method as described above when executing the computer program.

[0029] In addition, the present invention also includes a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the risk feature backtracking method as described above are implemented.

[0030] The application of the technical solution of the present invention has the following beneficial effects:

[0031] The present invention processes the source data, obtains the configuration data and stores it in the warehouse. The business can adjust the characteristic business logic caliber in time through the front-end function. The characteristic backtracking front-end function supports the selection of the characteristic range and the backtracking time end of the backtracking. The backtracking characteristics are updated in real time through the scheduled task in the background to generate the business field data table. The present invention constructs the backtracking task and executes the backtracking task, and synchronizes it to the specified data space for the user to view the backtracking characteristics.

[0032] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of the steps of the risk feature backtracking method in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] Glossary:

[0037] Atomic features: data directly obtained from the data set table, such as a field, without any other processing.

[0038] Derived features: further processing based on atomic features, such as summing or taking the maximum value of a field. The similarity between derived features and atomic feature data is that they come from the same business domain. The difference is that derived features can perform numerical and functional processing (through configuration functions) based on atomic features.

[0039] Composite features: Based on derived features, the features of the data business domain are further expanded. It can combine and process features of two different data tables / business domains. For example, it can add the two fields in the credit basic information table and the credit loan information table, or add the number of judicial cases in the judicial execution information table and the number of industrial and commercial time in the industrial and commercial information table. Unlike derived features, different fields in different tables and business domains can be selected for processing during configuration.

[0040] SQL feature: In order to configure the features set by complex functions, considering that the target data value is a derived feature or composite feature obtained by processing the atomic features of different business domains, and then further combined and processed, the SQL feature is set to use the configured features to further process the feature processing method based on the already configured features.

[0041] It should be noted that, in this embodiment, the atomic feature is a certain field information, the derived feature is the function processing of the field information, and the composite feature is the composite processing of different field information together. The SQL feature is the composite field information that can be further processed using the processed derived and composite features.

[0042] like Figure 1 As shown, this embodiment discloses a risk feature backtracking method based on a big data platform, comprising the following steps:

[0043] S1: Save the historical detailed data as source data into the steady-state database.

[0044] S2: Process the source data to obtain configuration features, and save the configuration features into the agile library.

[0045] Specifically, the process of processing the source data is as follows:

[0046] According to different business fields (such as industrial and commercial, judicial business fields), the historical detailed data is divided into business field data tables, the data fields in the business field data tables are set as atomic features, and the atomic features are configured with feature results to obtain the first feature. In this embodiment, the first feature is configured as an HSQL statement to facilitate front-end reading.

[0047] It should be noted that the atomic feature is the source data of a certain field selected by the user as the data source, in which the specific field information (such as date, amount, etc.) is saved as the atomic feature. Business domain data is calculated based on the combination of atomic features of different business scopes. For example, the calculation logic of the simple industrial and commercial business domain feature, such as the difference between the minimum non-empty date of the industrial and commercial registration date and the latest change date, can be configured by the user.

[0048] Specifically, select the non-empty function and the MIN function and take out the industrial and commercial data related to a certain customer, that is, the industrial and commercial registration date, and calculate the date difference with the latest change date obtained by the sorting algorithm to get the number of days of the difference. The feature analysis platform provides a user interface for users to independently select functions / algorithms and atomic features of data tables in different business fields (such as judicial information tables, credit information tables, etc. After selecting the business table, the data fields in the table can be selected as atomic features), and freely calculate the desired combined features by combining functions and features, and configure the backtracking period. The feature platform will calculate the results of the combined features in each period according to the backtracking period through historical data.

[0049] Furthermore, the first feature includes a derived feature, an SQL feature and a composite feature. The embodiment of the present invention generates a plurality of first features of varying complexity and functions by configuring feature results for atomic features. The derived feature, SQL feature and composite feature have been described in the foregoing text and will not be repeated here.

[0050] In this embodiment, for different features, certain categories are divided according to the business domain of feature source and complexity. Compared with the existing implementation technology, most configurations can only realize the feature processing of the same business domain and table. In this embodiment, composite features are set to further expand the features of the business domain of feature data source. The composite features combine and process the features of two different data tables / business domains, such as adding the two fields in the credit basic information table and the credit loan information table, or adding the number of judicial cases in the judicial execution information table and the number of industrial and commercial time in the industrial and commercial information table. The composite features can realize the combined processing of data in multiple tables and multiple business domains, providing flexibility in feature configuration.

[0051] Furthermore, in addition to realizing composite processing of different data sources, for functions with high complexity, the configuration method may have multiple layers of nesting, which is difficult to achieve only through ordinary configuration combinations. In this embodiment, the above problems are also solved by designing SQL features. SQL features are a feature processing method that can further process configured features based on already configured features. SQL features can read existing derived / composite features, and combine the processing methods of existing features with the processing methods configured by the user for further processing. Such processing methods can realize feature processing with relatively high complexity, and further provide more possibilities in feature configuration.

[0052] In this embodiment, by setting several different feature configuration modes, more implementation methods are provided for flexible configuration and complex configuration of features, and diverse configuration methods are provided on the existing market feature configuration implementation technology.

[0053] The derived features are calculated from the atomic features; the composite features are calculated from the derived features; and the SQL features are calculated from the composite features and the derived features. Specifically, the calculations involved in the configuration are mainly composed of numerical calculation combinations, such as basic calculations of square, variance, standard deviation, and date sorting, and then year-on-year and month-on-month growth or decay rate, P85 quantile, etc., and algorithms. The numerical calculation combination has more selectivity, such as simple variance numerical calculations used to characterize the degree of data dispersion, and the implementation formula is: sigma^{2}=frac{sigma(X-mu)^{2}}{N}, where X is a variable, mu is the sample mean, and N is the number of samples. Complex numerical calculations such as the trapezoidal rule in the integral algorithm, configure the trapezoidal rule function and the corresponding variables a, b, and calculate I(F)≈([f(a)+f(b)] / 2)(ba) to obtain a simplified result. In terms of algorithms, the feature platform also provides some common algorithms that can be used. If the features are configured for modeling content, the correlation parameters can be calculated for feature screening in feature engineering. For example, the Pearson correlation coefficient, mutual information, and maximum information coefficient can be calculated for a certain feature.

[0054] The Pearson correlation coefficient result interval is [-1, 1], and the calculation implemented is:

[0055]

[0056] Among them, x and y represent the two selected eigenvalues. is the sample mean, l xy , l xx , l yy is the distance between the x and y vectors, n is the number of samples, the numerator is calculated to achieve covariance, and the denominator is calculated to achieve the product of standard deviations.

[0057] The result interval of mutual information and maximum information coefficient is [0,1], and the calculation implemented is:

[0058]

[0059] Among them, x and y are the two selected eigenvalues, p(x i ,y) is the probability calculation result, and the probability calculation is realized by the Bayesian formula P(Y|X)=P(X|Y)*P(Y) / P(X). Users can calculate the feature results according to the above configured algorithm, and select the features for modeling by filtering the relevance of the features.

[0060] In another implementation case, the process of processing the source data further includes setting a scheduled task, which is executed according to a set period, specifically:

[0061] Read the first feature and business domain data table to check whether there are any new features;

[0062] When no new features are added or offline, the first feature is put into a temporary table, assembled into json format through a function, and then the GET_JSON_OBJECT function of HSQL is used to parse the json data and insert it into the result table, which is a business domain data table;

[0063] When a feature is added or taken offline, determine whether the business domain to which the newly added feature belongs exists. If the business domain already exists, directly add a column named after the first feature in the business domain data table. If the business domain does not exist, create a new business domain data table corresponding to the business domain, and add a column named after the feature in the business domain data table.

[0064] In another more specific implementation example, the scheduled task further includes querying whether there is a feature offline, and for a failed first feature, setting the value of the first feature in the business domain data table to null.

[0065] S3: Build a backtracking feature list based on the configuration features.

[0066] Specifically, the backtracking feature list includes the backtracking features and the time range corresponding to the backtracking features. The process of constructing the feature backtracking task based on the backtracking feature list is as follows:

[0067] The user sets the backtracking features and time range to complete the task of constructing feature backtracking.

[0068] S4: Construct a feature backtracking task based on the backtracking feature list.

[0069] In the process of executing the feature backtracking task, backtracking of multiple backtracking features is performed synchronously based on multi-threading. In this embodiment, the big data platform has a parallel processing mode with 10 threads by default. Multiple features under the backtracking task can be backtracked at the same time. In this embodiment, the basic multi-threaded parallel task processing is optimized. In terms of data parallelism, the data source is split to avoid complex associations, block sorting is used, the input data is cut into blocks and complexity sorting is performed in multiple processing units, and the sorting results are finally merged. In terms of space and task parallelism, dynamic programming and recursive parallel methods are added to improve the computational efficiency of feature backtracking tasks. In this embodiment, if the backtracking feature depends on other features, the backtracking function will first backtrack the dependent features, and then backtrack the target features.

[0070] S5: Execute the feature backtracking task, obtain the backtracking result, and complete the feature backtracking.

[0071] Specifically, the backtracking result includes the backtracking feature and the time range. After the feature backtracking is completed, the backtracking result is recalled to the agile library for storage for real-time viewing by users.

[0072] A computer device comprising a memory and a processor;

[0073] The memory is used to store a computer program executable on the processor;

[0074] The processor is used to implement the steps of the risk feature backtracking method as described above when executing the computer program.

[0075] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device.

[0076] The computer device may be a computing device such as a mobile phone, a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory. For example, the computer device may also include an input / output device, a network access device, a bus, etc.

[0077] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0078] The memory can be used to store the computer program and / or module, and the processor implements the computer program by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0079] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0080] In addition, this embodiment further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the risk feature backtracking method as described above are implemented.

[0081] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A risk feature backtracking method based on a big data platform, characterized in that: The steps include: S1: Save the historical detailed data as source data into the steady-state database; S2: Process the source data to obtain configuration features, and save the configuration features into the agile database; S3: Build a backtracking feature list based on the configuration features; S4: Construct feature backtracking task based on the backtracking feature list; S5: Execute the feature backtracking task, obtain the backtracking result, and complete the feature backtracking.

2. The risk feature backtracking method according to claim 1 is characterized in that: The process of processing source data is as follows: According to different business fields, the historical detailed data is divided into business field data tables, the data fields in the business field data tables are set as atomic features, and the atomic features are configured with feature results to obtain the first feature.

3. The risk feature backtracking method according to claim 2 is characterized in that: The first features include derived features, SQL features and composite features; The derived features are calculated from atomic features; The composite feature is calculated from the derived feature; The SQL feature is calculated by combining the composite feature and the derived feature.

4. The risk feature backtracking method according to claim 3 is characterized in that: The process of processing source data also includes setting a scheduled task, which is executed according to the set period, specifically: Read the first feature and business domain data table to check whether there are any new features; When no new features are added or offline, the first feature is put into a temporary table, assembled into json format through a function, and then the GET_JSON_OBJECT function of HSQL is used to parse the json data and insert it into the result table, which is a business domain data table; When a feature is added or taken offline, determine whether the business domain to which the newly added feature belongs exists. If the business domain already exists, directly add a column named after the first feature in the business domain data table. If the business domain does not exist, create a new business domain data table corresponding to the business domain, and add a column named after the feature in the business domain data table.

5. The risk feature backtracking method according to claim 4 is characterized in that: The scheduled task also includes querying whether there is a feature offline, and for a failed first feature, setting the value of the first feature in the business domain data table to null.

6. The risk feature backtracking method according to claim 1 is characterized in that: The backtracking feature list includes the backtracking features and the time range corresponding to the backtracking features. The process of constructing a feature backtracking task based on the backtracking feature list is as follows: The user sets the backtracking features and time range to complete the task of constructing feature backtracking.

7. The risk feature backtracking method according to claim 1 is characterized in that: In the process of executing the feature backtracking task, backtracking of multiple backtracking features is performed synchronously based on multi-threading.

8. The risk feature backtracking method according to claim 1, characterized in that: The backtracking result includes the backtracking feature and time range. After the feature backtracking is completed, the backtracking result is called back to the agile library for storage for real-time viewing by users.

9. A computer device, characterized in that: including memory and processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the risk feature backtracking method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the risk feature backtracking method according to any one of claims 1 to 8 are implemented.

Images