Data processing method, device, equipment and storage medium

Abstract

The present invention relates to the field of financial technology, and discloses a data processing method, device, equipment and storage medium, wherein the data processing method comprises: obtaining the binning quantile of each feature bin, grouping the feature data blocks of each feature bin according to the binning quantile to generate a correspondence between each feature bin and the feature data blocks of each feature bin; if a binning adjustment instruction is detected, determining the bin to be adjusted from each feature bin, and the feature data blocks to be adjusted of the bin to be adjusted according to the binning adjustment instruction and the corresponding relationship; performing adjustment processing on the bin to be adjusted and the feature data blocks to be adjusted, and outputting the adjustment processing result. The present invention solves the technical problem that the traditional binning data adjustment method has low data processing efficiency due to untimely data response when facing massive data.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] Feature binning is a data preprocessing technique used to reduce the impact of minor observation errors. It is a method of grouping multiple continuous values ​​into a smaller number of "bins". In actual use, users will adjust the binning results based on business experience, and the binning adjustment will cause the bin quantiles to change, causing the statistical information in the bin to change, thus requiring re-counting. However, when faced with massive amounts of data, re-counting a large amount of data will make the statistical process cumbersome and time-consuming, greatly reducing the data response speed of feature binning, resulting in untimely data response, resulting in a significant reduction in operating performance, and thus reducing the data processing efficiency of the system. Summary of the invention

[0004] The main purpose of the present invention is to provide a data processing method, device, equipment and storage medium, aiming to solve the technical problem that when the traditional bin data adjustment method faces massive data, the data response is not timely resulting in low data processing efficiency.

[0005] To achieve the above object, an embodiment of the present invention provides a data processing method, the data processing method comprising:

[0006] Obtaining a binning quantile of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0007] If a bin adjustment instruction is detected, determining a bin to be adjusted from each feature bin and a feature data block to be adjusted of the bin to be adjusted according to the bin adjustment instruction and the corresponding relationship;

[0008] The to-be-adjusted bins and the to-be-adjusted characteristic data blocks are adjusted, and an adjustment processing result is output.

[0009] Optionally, grouping the feature data blocks of each feature bin according to the bin quantile point to generate a correspondence between each feature bin and the feature data blocks of each feature bin includes:

[0010] The feature data blocks of each feature bin are cached, and the feature data blocks of each feature bin are grouped according to the bin quantile point to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0011] The step of adjusting the to-be-adjusted bins and the to-be-adjusted feature data blocks and outputting the adjustment results includes:

[0012] The to-be-adjusted bins and the to-be-adjusted characteristic data blocks are adjusted in the cache, and the adjustment processing result is output.

[0013] Optionally, the adjusting process of the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache includes:

[0014] Obtaining the to-be-adjusted quantile of the to-be-adjusted feature data block in the cache, and obtaining the instruction type of the bin adjustment instruction;

[0015] According to the instruction type, the quantile to be adjusted and the bin quantile, cache adjustment processing is performed on the bin to be adjusted and the feature data block to be adjusted.

[0016] Optionally, performing cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, includes:

[0017] If the instruction type is a bin splitting type, the bin to be adjusted and the feature data block to be adjusted are split according to the quantile to be adjusted and the bin quantile to obtain a plurality of target split bins and target split data blocks corresponding to each target split bin;

[0018] First statistical information of each target split data is obtained, and a cache adjustment result is generated according to each target split bin, the target split data corresponding to each target split bin, and the first statistical information corresponding to each target split data.

[0019] Optionally, performing cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, includes:

[0020] If the instruction type is a bin merging type, then according to the quantile to be adjusted and the bin quantile, the bin to be adjusted and the feature data block to be adjusted are merged to obtain a target merged bin and a target merged data block corresponding to the target merged bin;

[0021] Obtaining second statistical information of the bin to be adjusted, and summarizing the second statistical information to generate target statistical information;

[0022] A cache adjustment result is generated according to the target merged data block and the target statistical information.

[0023] Optionally, after performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks and outputting the adjustment processing results, the method further includes:

[0024] The information value of each feature bin in the statistical adjustment processing results;

[0025] If the information value is greater than or equal to the preset value, the adjustment processing effect is confirmed to be qualified;

[0026] If the information value is less than the preset value, it is confirmed that the adjustment processing effect is unsatisfactory.

[0027] Optionally, the information value of each feature bin in the statistical adjustment processing result includes:

[0028] The event value and non-event value of each feature bin in the statistical adjustment processing result are calculated to obtain the woe value;

[0029] The information value is obtained according to the event value, the non-event value and the woe value.

[0030] The present invention also provides a data processing device, the data processing device comprising:

[0031] A relationship module, used for obtaining a binning quantile of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0032] A determination module, configured to determine, if a bin adjustment instruction is detected, a bin to be adjusted from each feature bin, and a feature data block to be adjusted of the bin to be adjusted according to the bin adjustment instruction and the corresponding relationship;

[0033] The adjustment module is used to perform adjustment processing on the to-be-adjusted bins and the to-be-adjusted characteristic data blocks, and output the adjustment processing results.

[0034] Optionally, the relationship module includes:

[0035] A cache processing unit, used for performing cache processing on the feature data blocks of each feature bin, and grouping the feature data blocks of each feature bin according to bin quantile points, so as to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0036] The adjustment module comprises:

[0037] The cache adjustment unit is used to perform adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache, and output the adjustment processing result.

[0038] Optionally, the cache adjustment unit includes:

[0039] An instruction type subunit, used for obtaining the to-be-adjusted quantile of the to-be-adjusted feature data block in the cache, and obtaining the instruction type of the bin adjustment instruction;

[0040] The cache adjustment subunit is used to perform cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile.

[0041] Optionally, the cache adjustment subunit is used to:

[0042] If the instruction type is a bin splitting type, the bin to be adjusted and the feature data block to be adjusted are split according to the quantile to be adjusted and the bin quantile to obtain a plurality of target split bins and target split data blocks corresponding to each target split bin;

[0043] First statistical information of each target split data is obtained, and a cache adjustment result is generated according to each target split bin, the target split data corresponding to each target split bin, and the first statistical information corresponding to each target split data.

[0044] Optionally, the cache adjustment subunit is used to:

[0045] If the instruction type is a bin merging type, then according to the quantile to be adjusted and the bin quantile, the bin to be adjusted and the feature data block to be adjusted are merged to obtain a target merged bin and a target merged data block corresponding to the target merged bin;

[0046] Obtaining second statistical information of the bin to be adjusted, and summarizing the second statistical information to generate target statistical information;

[0047] A cache adjustment result is generated according to the target merged data block and the target statistical information.

[0048] Optionally, the data processing device further includes:

[0049] The statistical module is used to calculate the information value of each feature bin in the statistical adjustment processing results;

[0050] The qualified module is used to confirm that the adjustment processing effect is qualified if the information value is greater than or equal to the preset value;

[0051] The unqualified module is used to confirm that the adjustment processing effect is unqualified if the information value is less than a preset value.

[0052] Optionally, the statistics module includes:

[0053] The statistical unit is used to statistically adjust the event value and non-event value of each feature bin in the processing result to obtain the woe value;

[0054] The information value unit is used to obtain the information value according to the event value, the non-event value and the woe value.

[0055] In addition, to achieve the above-mentioned purpose, the present invention further provides a device, comprising: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein:

[0056] When the data processing program is executed by the processor, the steps of the data processing method described above are implemented.

[0057] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium;

[0058] The computer storage medium stores a data processing program, and when the data processing program is executed by the processor, the steps of the above-mentioned data processing method are implemented.

[0059] The present invention obtains the binning quantile of each feature bin, groups the feature data blocks of each feature bin according to the binning quantile, so as to generate the corresponding relationship between each feature bin and the feature data block of each feature bin; if a binning adjustment instruction is detected, the bin to be adjusted and the feature data block to be adjusted of the bin to be adjusted are determined from each feature bin according to the binning adjustment instruction and the corresponding relationship; the bin to be adjusted and the feature data block to be adjusted are adjusted, and the adjustment processing result is output. The present invention can be applied to the interactive binning of features of massive data in a big data environment, and the feature data to be adjusted is directly adjusted without any operation on the data blocks that do not need to be adjusted, so that the statistical steps of a large number of irrelevant data blocks can be reduced, the statistical time consumption can be reduced, the data response speed of the feature binning can be greatly improved, and then the statistical process can be simplified, the statistical efficiency of massive data can be improved, and the operation performance and response speed can be significantly improved on the premise of ensuring the accuracy of the results, the interactive binning experience can be optimized, and the data processing efficiency can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a flow chart of an embodiment of a data processing method of the present invention;

[0062] Figure 3 It is a schematic diagram of binning and blocking data in the data processing method of the present invention;

[0063] Figure 4 A schematic diagram of merging binned data blocks in the data processing method of the present invention;

[0064] Figure 5 It is a schematic diagram of splitting the binned data blocks in the data processing method of the present invention.

[0065] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0067] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0068] The device in the embodiment of the present invention may be a PC or a server device.

[0069] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0070] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0071] like Figure 1As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data processing program.

[0072] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data processing program stored in the memory 1005 and perform the operations in each embodiment of the following data processing method.

[0073] The main idea of ​​the embodiment of the present invention is: obtain the binning quantile of each feature bin, group the feature data blocks of each feature bin according to the binning quantile, so as to generate the corresponding relationship between each feature bin and the feature data block of each feature bin; if the binning adjustment instruction is detected, determine the bin to be adjusted and the feature data block to be adjusted of the bin to be adjusted from each feature bin according to the binning adjustment instruction and the corresponding relationship; adjust the bin to be adjusted and the feature data block to be adjusted, and output the adjustment processing result. The present invention can be applied to the interactive binning of features of massive data in a big data environment, directly adjust the feature data to be adjusted, without performing any operation on the data blocks that do not need to be adjusted, so as to reduce the statistical steps of a large number of irrelevant data blocks, reduce the statistical time consumption, greatly improve the data response speed of the feature binning, and then simplify the statistical process, improve the statistical efficiency of massive data, and significantly improve the operation performance and response speed on the premise of ensuring the accuracy of the results, optimize the interactive binning experience, and greatly improve the data processing efficiency.

[0074] The embodiments of the present invention take into account that in the prior art, users may adjust the binning results based on their business experience, and the binning adjustment may cause the bin quantiles to change, causing the statistical information in the bin to change, thus requiring re-counting. However, when faced with massive amounts of data, re-counting a large amount of data will make the statistical process cumbersome, consume too much system resources, and respond in a delayed manner, greatly reducing the data processing efficiency of the system.

[0075] The present invention provides a solution that can be applied to interactive feature binning of massive data in a big data environment. The feature data to be adjusted can be directly adjusted without any operation on the data blocks that do not need to be adjusted. This can reduce the statistical steps of a large number of irrelevant data blocks, reduce statistical time consumption, greatly improve the data response speed of feature binning, and further simplify the statistical process, improve the statistical efficiency of massive data, and significantly improve the operating performance and response speed while ensuring the accuracy of the results, optimize the interactive binning experience, and greatly improve the data processing efficiency.

[0076] Based on the above hardware structure, an embodiment of the data processing method of the present invention is proposed.

[0077] The present invention belongs to the field of financial technology (Fintech), and provides a data processing method.

[0078] In an embodiment of a data processing method, referring to Figure 2 , the data processing method comprises:

[0079] Step S10, obtaining a binning quantile of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0080] Step S20, if a bin adjustment instruction is detected, then according to the bin adjustment instruction and the corresponding relationship, a bin to be adjusted and a feature data block to be adjusted of the bin to be adjusted are determined from each feature bin;

[0081] Step S30, performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted characteristic data blocks, and outputting the adjustment processing result.

[0082] The data processing method can be applied to the device, and the specific contents are as follows:

[0083] Step S10, obtaining a binning quantile of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0084] Each feature bin has its own bin quantile, and the bin quantile corresponds to the data boundary of the feature bin. For example, there is currently a group of age feature bins: feature bin A (0-10 years old), feature bin B (10-20 years old), feature bin C (20-30 years old) and feature bin D (30-40 years old). Each group of feature bins stores the relevant feature data blocks of the age feature. The system equipment obtains all feature bins and groups the feature data blocks of all feature bins according to the bin quantile. Figure 3 , Figure 3 The content on the left is bin 1...bin n. The feature data blocks are grouped according to the value range of the feature values ​​in each bin. That is, the data within the value range of each bin is mapped to the same group, and the data in the group does not need to be sorted. Figure 3Content on the right. The data of the same group can be stored in the same computing node or multiple computing nodes, and multiple grouped data can also be stored in the same computing node, just mark them and distinguish them. This grouping process can form a correspondence between each bin and the data within the bin range, and quickly read and match in advance. For example, all feature data blocks in feature bin A are mapped to bin 1 data blocks, and all feature data blocks in feature bin B are mapped to bin 2 data blocks, and so on, thereby forming bin 1 data blocks, bin 2 data blocks... bin n data blocks and other feature bin data blocks, that is, each feature bin and feature data block form a correspondence. Therefore, feature bin n and bin data block n are mapped to each other. Bin data block n is the cache data block of feature bin n, which contains all feature data blocks of feature bin n.

[0085] Step S20, if a bin adjustment instruction is detected, then according to the bin adjustment instruction and the corresponding relationship, a bin to be adjusted and a feature data block to be adjusted of the bin to be adjusted are determined from each feature bin;

[0086] When a bin adjustment instruction is detected, it proves that there is a business adjustment demand for the feature bins, and the business adjustment demand corresponds to the bins to be adjusted, and the bins to be adjusted and the corresponding feature data blocks to be adjusted can be located from each feature bin according to the corresponding relationship. For example, if the bin adjustment instruction requires the split adjustment of feature bin B of 10-20 years old, feature bin B is obtained as the bin to be adjusted and the corresponding feature data blocks to be adjusted.

[0087] In this embodiment, the bins to be adjusted can be determined in the bin adjustment instruction. For example, the feature bins include feature bin A for the 0-10 years old interval, feature bin B for the 10-20 years old interval, feature bin C for the 20-30 years old interval, and feature bin D for the 30-40 years old interval. If the feature data of the 25-year-old is to be adjusted in the bin adjustment instruction, then the feature bin C for the 20-30 years old interval and the corresponding feature data block to be adjusted can be located through the quantile of 25 and the feature bin corresponding to the quantile in the corresponding relationship.

[0088] Step S30, performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted characteristic data blocks, and outputting the adjustment processing result.

[0089] The adjustment processing method can be merging bins or splitting bins, which depends on the actual situation. Specifically, the feature data blocks of each feature bin are grouped according to the bin quantile point to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin, including:

[0090] The feature data blocks of each feature bin are cached, and the feature data blocks of each feature bin are grouped according to the bin quantile point to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0091] The step of adjusting the to-be-adjusted bins and the to-be-adjusted feature data blocks and outputting the adjustment processing results includes:

[0092] The to-be-adjusted bins and the to-be-adjusted characteristic data blocks are adjusted in the cache, and the adjustment processing result is output.

[0093] This embodiment adopts a cache grouping mechanism to adjust the data bins in the cache, avoid excessive consumption of system resources in the process of massive data statistics, and improve response speed and data processing efficiency. Specifically, assuming that the feature bins need to be adjusted according to business adjustment requirements, the device system will obtain all feature bins and map all feature bins to the cache.

[0094] Further, the adjusting process of the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache includes:

[0095] Step A1, obtaining the to-be-adjusted quantile of the to-be-adjusted feature data block in the cache, and obtaining the instruction type of the bin adjustment instruction;

[0096] The quantiles to be adjusted of the feature data to be adjusted are obtained. The feature data to be adjusted are represented in the form of data blocks. For example, there are currently 4 feature data to be adjusted: data block A in the 0-10 age range, data block B in the 10-20 age range, data block C in the 20-30 age range, and data block D in the 30-40 age range. According to the data blocks, the quantile list of all data blocks can be obtained as [10, 20, 30, 40]. Therefore, the quantiles to be adjusted of the feature data to be adjusted can be determined according to the quantile list in the cache. For example, if the feature data to be adjusted is data block D in the 30-40 age range, the corresponding quantile to be adjusted is 40.

[0097] It can be understood that the bin adjustment instructions include two types, bin splitting type and bin merging type. In order to avoid errors in the adjustment process, it is necessary to distinguish the types of bin adjustment instructions to obtain the instruction types.

[0098] Step A2: performing cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile.

[0099] Different instruction types represent different adjustment processes, and the quantile to be adjusted and the target quantile can be located to the object to be adjusted. Therefore, according to the instruction type, the quantile to be adjusted and the target quantile, the feature data to be adjusted can be adjusted to obtain the target feature data block.

[0100] Further, the cache adjustment process is performed on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, including:

[0101] Step A21, if the instruction type is a bin splitting type, then according to the quantile point to be adjusted and the bin quantile point, the bin to be adjusted and the feature data block to be adjusted are split to obtain a plurality of target split bins and target split data blocks corresponding to each target split bin;

[0102] If the instruction type is a bin splitting type, it proves that the current bin adjustment instruction is to split a specific bin among all feature bins. At this time, the feature data block to be adjusted is the object to be split.

[0103] The bin splitting needs to split the bin to be adjusted into two bins, split the feature data block to be adjusted into two data blocks, that is, generate two new bins and corresponding feature data blocks, and then increase the bin numbers of the original bins whose numbers are greater than k+1 by one, and increase the corresponding feature data numbers by one. Figure 4 , assuming that the bin k data block is the feature data block to be adjusted, k is the quantile to be adjusted, and k+1 is the target quantile. Therefore, according to k (i.e. the quantile to be adjusted) and k+1 (i.e. the target quantile), the bin k data block (i.e. the feature data block to be adjusted) can be split into bin k data blocks and bin k+1 data blocks (i.e. multiple target split data blocks), and the bin k data block maps the corresponding feature bin k, which means that the feature bin k is split into feature bin k and feature bin k+1. In other words, when splitting the feature data block to be adjusted, only the current feature data block to be adjusted is adjusted, and no operation is required on other data.

[0104] Step A22, obtaining first statistical information of each target split data, and generating a cache adjustment result according to each target split bin, the target split data corresponding to each target split bin, and the first statistical information corresponding to each target split data.

[0105] Each target split data block stores the corresponding statistical information. For example, if the target split data block allocates the feature data and other information in the original feature data block to be adjusted, the statistical information (such as event information and non-event information) in the original feature data block to be adjusted will also be reallocated, so that the statistical information corresponding to each target split data block can be obtained. Each target split data block and the corresponding statistical information can generate the cache adjustment result.

[0106] It should be noted that the essence of bin adjustment is to generate new data quantile information. For example, the original bin quantile is [10, 20, 30, 40], and the adjusted bin quantile is [10, 30, 40]. As the bin quantile changes, the statistical information in each bin changes, so it is necessary to traverse all data and recalculate the bin statistics in each bin.

[0107] Further, the cache adjustment process is performed on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, including:

[0108] Step A23, if the instruction type is a bin merging type, then according to the quantile to be adjusted and the bin quantile, the bin to be adjusted and the feature data block to be adjusted are merged to obtain a target merged bin and a target merged data block corresponding to the target merged bin;

[0109] Step A24, obtaining second statistical information of the bins to be adjusted, and summing up the second statistical information to generate target statistical information;

[0110] Step A25: Generate a cache adjustment result according to the target merged data block and the target statistical information.

[0111] If the instruction type is a bin merging type, it proves that the current bin adjustment instruction is to merge a specific bin in all feature bins, which involves multiple feature bins. At this time, the feature data to be adjusted is the object to be merged, and there are multiple feature data to be adjusted.

[0112] Bin merging requires merging multiple corresponding data into one data block, that is, merging two or more data blocks, and then reducing the original bin numbers greater than k+1 by one, and the corresponding data numbers are also reduced by one. Figure 5, assuming that the bin k data block and the bin k+1 data block are feature data blocks to be adjusted, k and k+1 are quantiles to be adjusted, and k is the bin quantile, therefore, according to k and k+1 (i.e. quantiles to be adjusted) and k (i.e. bin quantile), the bin k data and bin k+1 (i.e. feature data to be adjusted) can be merged into bin k data (i.e. target merged data), and the bin k data block and bin k+1 data block merged in the cache map the corresponding feature bin k and feature bin k+1, which means that the feature bin k and feature bin k+1 are merged into feature bin k. In other words, when merging the feature data blocks to be adjusted in the cache, only the current feature data block to be adjusted is adjusted, and no operation is required on other data blocks.

[0113] The statistical information of each feature data block to be adjusted needs to be fully acquired and then added and summarized to obtain the target statistical information. The original statistical information in the feature data block to be adjusted is acquired. Since the feature data blocks to be adjusted are merged, the corresponding statistical information also needs to be merged. In this embodiment, the statistical information is added and summarized to generate the target statistical information. For example, the statistical information a of the A data block and the statistical information b of the B data block. As the A and B data blocks are merged, the statistical information a and the statistical information b will also be merged to generate the target statistical information.

[0114] After the data blocks are merged, the original bin numbers greater than k+1 are reduced by one, and the corresponding data block numbers are also reduced by one. The merging process is defined as:

[0115]

[0116]

[0117] After obtaining the target merged data block and the target statistical information, the two are combined to generate the target feature data block.

[0118] The present invention obtains the binning quantile of each feature bin, groups the feature data blocks of each feature bin according to the binning quantile, so as to generate the corresponding relationship between each feature bin and the feature data block of each feature bin; if a binning adjustment instruction is detected, the bin to be adjusted and the feature data block to be adjusted of the bin to be adjusted are determined from each feature bin according to the binning adjustment instruction and the corresponding relationship; the bin to be adjusted and the feature data block to be adjusted are adjusted, and the adjustment processing result is output. The present invention can be applied to the interactive binning of features of massive data in a big data environment, and the feature data to be adjusted is directly adjusted without any operation on the data blocks that do not need to be adjusted, so that the statistical steps of a large number of irrelevant data blocks can be reduced, the statistical time consumption can be reduced, the data response speed of the feature binning can be greatly improved, and then the statistical process can be simplified, the statistical efficiency of massive data can be improved, and the operation performance and response speed can be significantly improved on the premise of ensuring the accuracy of the results, the interactive binning experience can be optimized, and the data processing efficiency can be greatly improved.

[0119] Further, based on the first embodiment, a second embodiment of the data processing method of the present invention is proposed. In this embodiment, after the adjustment processing is performed on the to-be-adjusted bins and the to-be-adjusted feature data blocks and the adjustment processing results are output, the method further includes:

[0120] Step a, calculating the information value of each feature bin in the statistical adjustment processing result;

[0121] The information value is the IV value, the full name of which is Information Value, which can measure the variable prediction ability. The larger the IV value, the better the binning processing effect it represents. In this embodiment, the information value of each feature bin in the statistical adjustment processing result is calculated.

[0122] Specifically, the information value of each feature bin in the target feature bin list is counted, including:

[0123] Step a1, statistically adjusting the event value and non-event value of each feature bin in the processing result to obtain the woe value;

[0124] Step a2, obtaining information value according to the event value, the non-event value and the woe value.

[0125] Specifically, refer to the following algorithm:

[0126] Features: X;

[0127] Number of bins: n, which indicates the number of segments into which feature X is divided after sorting;

[0128] Box: X i, 1<=i<=n, represents a segment of data of feature X after sorting;

[0129] Binning point: S, containing n-1 different values, S in turn i , 1<=i<=n-1, and S i i+1 ;

[0130] Number of events in each bin: T event , contains n values, in order 1<=i<=n;

[0131] Number of non-events in each bin: T non-event , contains n values, in order 1<=i<=n;

[0132] Total number of events: N event ,

[0133] Total number of non-events: N non-event ,

[0134] Woe value of each bin,

[0135] The IV value of each bin,

[0136] The final binning effect evaluation index IV value,

[0137] According to the event value and non-event value in the above algorithm, the woe value corresponding to the target feature bin can be calculated, and the IV value of the target feature bin can be obtained based on the woe value. The IV value is the information value.

[0138] Step b, if the information value is greater than or equal to the preset value, confirm that the adjustment processing effect is qualified;

[0139] Step c: if the information value is less than the preset value, it is confirmed that the adjustment processing effect is unqualified.

[0140] ​In this embodiment, the reference standard of information value is preset as a preset value, which can be set according to actual business needs. If the information value is greater than or equal to the preset value, it proves that the binning processing effect of the current adjustment processing result is qualified. If the information value is less than the preset value, it proves that the binning processing effect of the current adjustment processing result is unqualified. For example, assuming that the preset value is a and the information value is b, if b is greater than a, it proves that the current binning adjustment processing has an obvious trend effect, and the system device will confirm that the binning processing effect of the adjustment processing result is qualified; if b is less than a, it proves that the trend effect of the current binning adjustment processing is not obvious, and the system device will confirm that the binning processing effect of the adjustment processing result is unqualified.

[0141] In addition, an embodiment of the present invention further provides a data processing device, the data processing device comprising:

[0142] A relationship module, used for obtaining a binning quantile of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0143] A determination module, configured to determine, if a bin adjustment instruction is detected, a bin to be adjusted from each feature bin, and a feature data block to be adjusted of the bin to be adjusted according to the bin adjustment instruction and the corresponding relationship;

[0144] The adjustment module is used to perform adjustment processing on the to-be-adjusted bins and the to-be-adjusted characteristic data blocks, and output the adjustment processing results.

[0145] Optionally, the relationship module includes:

[0146] A cache processing unit, used for performing cache processing on the feature data blocks of each feature bin, and grouping the feature data blocks of each feature bin according to bin quantile points, so as to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin;

[0147] The adjustment module comprises:

[0148] The cache adjustment unit is used to perform adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache, and output the adjustment processing result.

[0149] Optionally, the cache adjustment unit includes:

[0150] An instruction type subunit, used for obtaining the to-be-adjusted quantile of the to-be-adjusted feature data block in the cache, and obtaining the instruction type of the bin adjustment instruction;

[0151] The cache adjustment subunit is used to perform cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile.

[0152] Optionally, the cache adjustment subunit is used to:

[0153] If the instruction type is a bin splitting type, the bin to be adjusted and the feature data block to be adjusted are split according to the quantile to be adjusted and the bin quantile to obtain a plurality of target split bins and target split data blocks corresponding to each target split bin;

[0154] First statistical information of each target split data is obtained, and a cache adjustment result is generated according to each target split bin, the target split data corresponding to each target split bin, and the first statistical information corresponding to each target split data.

[0155] Optionally, the cache adjustment subunit is used to:

[0156] If the instruction type is a bin merging type, then according to the quantile to be adjusted and the bin quantile, the bin to be adjusted and the feature data block to be adjusted are merged to obtain a target merged bin and a target merged data block corresponding to the target merged bin;

[0157] Obtaining second statistical information of the bin to be adjusted, and summarizing the second statistical information to generate target statistical information;

[0158] A cache adjustment result is generated according to the target merged data block and the target statistical information.

[0159] Optionally, the data processing device further includes:

[0160] The statistical module is used to calculate the information value of each feature bin in the statistical adjustment processing results;

[0161] The qualified module is used to confirm that the adjustment processing effect is qualified if the information value is greater than or equal to the preset value;

[0162] The unqualified module is used to confirm that the adjustment processing effect is unqualified if the information value is less than a preset value.

[0163] Optionally, the statistics module includes:

[0164] The statistical unit is used to statistically adjust the event value and non-event value of each feature bin in the processing result to obtain the woe value;

[0165] The information value unit is used to obtain the information value according to the event value, the non-event value and the woe value.

[0166] In addition, an embodiment of the present invention also proposes a device, which includes: a memory 109, a processor 110, and a data processing program stored in the memory 109 and executable on the processor 110, wherein the data processing program implements the steps of each embodiment of the above-mentioned data processing method when executed by the processor 110.

[0167] In addition, the present invention also provides a computer storage medium, which stores one or more programs. The one or more programs can also be executed by one or more processors to implement the steps of each embodiment of the above-mentioned data processing method.

[0168] The expanded contents of the specific implementation methods of the device and storage medium (ie, computer storage medium) of the present invention are basically the same as those of the above-mentioned data processing method embodiments, and will not be elaborated here.

[0169] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0170] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0172] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Obtaining a binning quantile of each feature bin, performing cache processing on the feature data blocks of each feature bin, and grouping the feature data blocks of each feature bin according to the binning quantile, so as to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin; If a bin adjustment instruction is detected, determining a bin to be adjusted from each feature bin and a feature data block to be adjusted of the bin to be adjusted according to the bin adjustment instruction and the corresponding relationship; Performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache, and outputting the adjustment processing result; The step of performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks in the cache and outputting the adjustment processing results includes: Obtaining the to-be-adjusted quantile of the to-be-adjusted feature data block in the cache, and obtaining the instruction type of the bin adjustment instruction; According to the instruction type, the quantile to be adjusted and the bin quantile, cache adjustment processing is performed on the bin to be adjusted and the feature data block to be adjusted.

2. The data processing method according to claim 1, characterized in that: The step of performing cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, comprises: If the instruction type is a bin splitting type, the bin to be adjusted and the feature data block to be adjusted are split according to the quantile to be adjusted and the bin quantile to obtain a plurality of target split bins and target split data blocks corresponding to each target split bin; First statistical information of each target split data is obtained, and a cache adjustment result is generated according to each target split bin, the target split data corresponding to each target split bin, and the first statistical information corresponding to each target split data.

3. The data processing method according to claim 1, characterized in that: The step of performing cache adjustment processing on the bin to be adjusted and the feature data block to be adjusted according to the instruction type, the quantile to be adjusted and the bin quantile, comprises: If the instruction type is a bin merging type, then according to the quantile to be adjusted and the bin quantile, the bin to be adjusted and the feature data block to be adjusted are merged to obtain a target merged bin and a target merged data block corresponding to the target merged bin; Obtaining second statistical information of the bin to be adjusted, and summarizing the second statistical information to generate target statistical information; A cache adjustment result is generated according to the target merged data block and the target statistical information.

4. The data processing method according to claim 1, characterized in that: After the adjustment processing is performed on the to-be-adjusted bins and the to-be-adjusted characteristic data blocks, and the adjustment processing results are output, the method further includes: The information value of each feature bin in the statistical adjustment processing results; If the information value is greater than or equal to the preset value, the adjustment processing effect is confirmed to be qualified; If the information value is less than the preset value, it is confirmed that the adjustment processing effect is unsatisfactory.

5. The data processing method according to claim 4, characterized in that: The information value of each feature bin in the statistical adjustment processing result includes: The event value and non-event value of each feature bin in the statistical adjustment processing result are calculated to obtain the woe value; The information value is obtained according to the event value, the non-event value and the woe value.

6. A data processing device, characterized in that: The data processing device comprises: A relationship module, used to obtain a binning quantile of each feature bin, cache the feature data blocks of each feature bin, and group the feature data blocks of each feature bin according to the binning quantile to generate a corresponding relationship between each feature bin and the feature data blocks of each feature bin; A determination module, configured to determine, if a bin adjustment instruction is detected, a bin to be adjusted from each feature bin, and a feature data block to be adjusted of the bin to be adjusted according to the bin adjustment instruction and the corresponding relationship; An adjustment module, used for performing adjustment processing on the to-be-adjusted bins and the to-be-adjusted feature data blocks in a cache, and outputting an adjustment processing result; The adjustment module is further used to obtain the to-be-adjusted quantile of the to-be-adjusted characteristic data block in the cache, and obtain the instruction type of the bin adjustment instruction; and perform cache adjustment processing on the to-be-adjusted bin and the to-be-adjusted characteristic data block according to the instruction type, the to-be-adjusted quantile and the bin quantile.

7. A device, characterized in that: The device comprises: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program implements the steps of the data processing method according to any one of claims 1 to 5 when executed by the processor.

8. A storage medium, characterized in that: The storage medium stores a data processing program, and when the data processing program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 5 are implemented.

Images