Data screening method, device, equipment, storage medium and product

Abstract

The application discloses a data screening method and device, equipment, storage medium and product, and relates to the technical field of big data. The method comprises the following steps: configuring a fitting threshold value associated with a target function of initial business sample data; classifying the initial business sample data according to the distance between the fitting threshold value and the sample of the initial business sample data, obtaining a corresponding first candidate sample set and a second candidate sample set; performing set replacement on a sample subset in the first candidate sample set and a sample subset in the second candidate sample set based on a set replacement condition, obtaining a corresponding target sample set, and using sample data in the target sample set to perform business prediction. The technical scheme of the application can achieve the purpose of minimizing the target sample set through the set replacement condition under the condition of setting the fitting threshold value, effectively reduce the calculation complexity and the calculation cost, and improve the accuracy of data sample screening and the effect of subsequent business prediction.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular to a data filtering method, apparatus, device, storage medium and product. Background Technology

[0002] Training parameters using data samples typically requires a large number of data samples. However, an excessive number of data samples leads to a significant increase in computational costs and also increases the generalization error. Current technologies for filtering large datasets typically employ techniques such as principal component analysis and cluster analysis to select key indicator data. However, this approach is computationally complex and has low filtering accuracy, failing to measure key information related to the objective function. Furthermore, it cannot confirm whether the information content of the objective function still meets expectations after filtering the data samples. Therefore, there is an urgent need for a data filtering method that can effectively reduce computational costs and generalization error levels without compromising the accuracy of the training parameters or within an acceptable range. Summary of the Invention

[0003] This invention provides a data filtering method, apparatus, device, storage medium, and product, which aim to minimize the target sample set through set permutation conditions, effectively reducing computational complexity and cost, improving the accuracy of data sample filtering, and effectively improving the effect of subsequent business prediction under the same conditions.

[0004] In a first aspect, embodiments of the present invention provide a data filtering method, including:

[0005] Configure fitting threshold values ​​that are associated with the objective function of the initial business sample data according to the preset business screening requirements;

[0006] The initial business sample data is classified according to the fitting threshold value and the inter-sample distance of the initial business sample data to obtain the corresponding first candidate sample set and second candidate sample set;

[0007] Based on pre-configured set permutation conditions, a sample subset in the first candidate sample set and a corresponding sample subset in the second candidate sample set are permuted to obtain a corresponding target sample set, so as to use the sample data in the target sample set for business prediction; wherein, the pre-configured set permutation conditions satisfy: the number of first sample points in each sample subset in the first candidate sample set is greater than the number of second sample points in the corresponding sample subset in the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function.

[0008] Secondly, embodiments of the present invention also provide a data filtering device, the device comprising:

[0009] The configuration module is used to configure the fitting threshold value associated with the objective function of the initial business sample data according to the preset business screening requirements;

[0010] The classification module is used to classify the initial business sample data according to the sample distance between the fitting threshold value and the initial business sample data, so as to obtain the corresponding first candidate sample set and second candidate sample set.

[0011] The permutation module is used to perform set permutations on a sample subset in the first candidate sample set and a corresponding sample subset in the second candidate sample set based on pre-configured set permutation conditions to obtain a corresponding target sample set, so as to use the sample data in the target sample set for business prediction; wherein, the pre-configured set permutation conditions satisfy: the number of first sample points in each sample subset in the first candidate sample set is greater than the number of second sample points in the corresponding sample subset in the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function.

[0012] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data filtering method as described in any of the embodiments of the present invention.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data filtering method as described in any of the embodiments of the present invention.

[0014] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the data filtering method as described in any of the embodiments of the present invention.

[0015] In this embodiment of the invention, when a fitting threshold associated with the objective function is set, the initial business sample data is classified by the sample distance and the fitting threshold. Based on a pre-configured set permutation condition, the classified sets are permuted to obtain the target sample set. This method can minimize the target sample set by using set permutation conditions while setting the fitting threshold, effectively reducing computational complexity and cost, improving the accuracy of data sample selection, and effectively improving the performance of subsequent business predictions under the same conditions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a data filtering method provided in an embodiment of the present invention;

[0018] Figure 2 A flowchart illustrating yet another data filtering method provided in an embodiment of the present invention;

[0019] Figure 3 This is a structural block diagram of a data filtering device provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0022] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0024] In one embodiment, Figure 1 This is a flowchart of a data filtering method provided in an embodiment of the present invention. This embodiment is applicable to situations where a large amount of business sample data is filtered to reduce the amount of business sample data. The method can be executed by a data filtering device, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0025] like Figure 1 As shown, the data filtering method in this implementation includes the following specific steps:

[0026] S110. Configure a fitting threshold value associated with the objective function of the initial business sample data according to the preset business screening requirements.

[0027] Among them, the preset business screening requirements can be understood as the requirements for screening relevant business sample data. Of course, the preset business screening requirements can include various business screening requirements, including but not limited to business management screening requirements and financial industry business transaction screening requirements.

[0028] In this embodiment, the initial business sample data refers to the original business sample data related to business screening requirements. This original business sample data consists of business sample data collected at different times, and may include, but is not limited to, interest rate business sample data, financial exchange rate business sample data, and customer behavior-related business sample data. In this embodiment, different original business sample data correspond to different business requirements. For example, when the business screening requirement is to screen interest rate business sample data, the initial business sample data is the interest rate business-related business sample data collected within a preset time period; when the business screening requirement is to screen financial exchange rate business sample data, the initial business sample data is the financial exchange rate business-related business sample data collected within a preset time period. This embodiment does not impose any limitations on this.

[0029] In this embodiment, the objective function is the objective function configured for the initial business sample data under the preset business screening requirements. The objective function in this embodiment can be in the form of a likelihood function or other functions; this embodiment does not impose any restrictions. In this embodiment, the objective function configured for the initial business sample data corresponds to a corresponding fitting threshold value. This fitting threshold value can also be understood as the minimum goodness-of-fit index of the Akaike Information Criterion (AIC) corresponding to the objective function, which can balance the complexity of the estimated objective function and the goodness of the data fitted by this criterion.

[0030] In this embodiment, since different preset business screening requirements correspond to different initial business sample data, initial business sample data within a preset time period can be selected. A corresponding objective function is constructed for the initial business sample data based on the preset business screening requirements. Based on this objective function, an AIC matching the objective function is selected, and a fitting threshold value associated with the AIC is obtained. This fitting threshold value ensures that, while maintaining the initial business sample data screening accuracy, a sufficiently small amount of target sample data is obtained from the initial business sample data. It should be noted that, since business sample data for different business requirements will vary at different times, in this embodiment, when screening the original business sample data, the time variable can be used as a feature variable of the original business sample data to make the screening results more accurate and, to a certain extent, improve the accuracy of subsequent business predictions.

[0031] S120. Classify the initial business sample data according to the sample distance between the fitted threshold value and the initial business sample data to obtain the corresponding first candidate sample set and second candidate sample set.

[0032] Here, the inter-sample distance refers to the distance between any two samples in the initial business sample data. This inter-sample distance can be calculated using one of the following distance formulas: Euclidean distance, Manhattan distance, Chebyshev distance, and Minkowski distance. This embodiment does not impose any restrictions on this.

[0033] In this embodiment, the first candidate sample set can also be called the minimum involved set, which can be understood as the set of candidate samples that meet the fitting threshold condition after classification based on the inter-sample distance of the initial business sample data. The second candidate sample set can also be called the non-minimum involved set, which can be understood as the set of candidate samples that do not meet the fitting threshold condition after classification based on the inter-sample distance of the initial business sample data. Of course, the first candidate sample set may include one or more sample subsets, each of which includes one or more business sample data; similarly, the second candidate sample set may also include one or more sample subsets, each of which includes one or more business sample data.

[0034] In this embodiment, the initial sample data can be processed in different spatial dimensions. The inter-sample distance of the processed initial business sample data is obtained by using a preset first distance formula. The initial sample data is then subjected to corresponding clustering analysis based on the inter-sample distance and a preset clustering algorithm to obtain the corresponding initial sample set. Based on this, the target inter-sample distance from the processed initial business sample data to the centroid of each sample set is determined according to the centroid of the sample set corresponding to each sample subset in the initial sample set and a preset second distance formula. The sample subsets are then removed from the initial sample set according to the target inter-sample distance and the fitting threshold value. The sample set whose ACI value of the unremoved sample subsets in the candidate sample set meets the fitting threshold value is formed into a first candidate sample set, and the removed sample set is formed into a second candidate sample set.

[0035] S130. Based on the pre-configured set permutation conditions, perform set permutation between the sample subsets in the first candidate sample set and the corresponding sample subsets in the second candidate sample set to obtain the corresponding target sample set, so as to use the sample data in the target sample set for business prediction.

[0036] Here, the set permutation condition refers to the pre-configured conditions that must be met for the set permutation to proceed. This condition ensures that while the sample data decreases during the permutation process, the resulting target sample set satisfies the fitting threshold associated with the objective function. The target sample set can be understood as the final sample set obtained, which can be used for subsequent business predictions.

[0037] In some embodiments, the pre-configured set permutation conditions satisfy the following: the number of first sample points in each sample subset of the first candidate sample set is greater than the number of second sample points in the corresponding sample subset of the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function. This can be understood as follows: when performing set permutation, sample subsets in the first candidate sample set and their corresponding sample subsets in the second candidate sample set that satisfy the set permutation conditions are permuted; conversely, if the set permutation conditions are not satisfied, no sample subset permutation is performed.

[0038] In this embodiment, each sample subset in the first candidate sample set and each corresponding sample subset in the second candidate sample set can be sorted according to a pre-set sorting method to obtain the corresponding sorted first target sample set and second target sample set. Based on the sorting, the sample subsets in the first target sample set and the sample subsets in the second target sample set are subjected to sample set permutation based on pre-configured set permutation conditions. Specifically, each sample point in the farthest sample set in the first target sample set whose distance from the centroid of the farthest sample set is greater than a preset first distance threshold can be identified as an edge sample point, and each sample point in the nearest sample set in the second target sample set whose distance from the centroid of the nearest sample set is less than a preset second distance threshold can be identified as a centroid vicinity point. The sample sets corresponding to the edge sample points and the sample sets corresponding to the centroid vicinity points are subjected to sample set permutation. The above sample set permutation method is repeated until the first target sample set and the second target sample set cannot be permuted, thus obtaining the target sample set.

[0039] In this embodiment, the sample data in the obtained target sample set can be used for business forecasting. The scenarios for business forecasting may include, but are not limited to, interest rate forecasting, market exchange rate forecasting, and customer behavior forecasting. Of course, the scenario for business forecasting is related to the initial business sample data. For example, if the business forecasting scenario is for interest rate forecasting, the initial business sample data is interest rate business sample data within a preset time period. High-precision interest rate business data is selected from the interest rate business sample data and subsequently used for interest rate forecasting. If the business forecasting scenario is for customer behavior forecasting, the initial business sample data is business sample data corresponding to customer behavior within a preset time period. High-precision business sample data is selected from the customer behavior business sample data and subsequently used for customer behavior forecasting. This embodiment does not impose any limitations on this.

[0040] The above-described technical solution in this embodiment, when setting a fitting threshold value associated with the objective function, classifies the initial business sample data by the sample distance and fitting threshold value of the initial business sample data, and then permutes the classified sets based on pre-configured set permutation conditions to obtain the target sample set. This can minimize the target sample set by setting the fitting threshold value and using set permutation conditions, effectively reducing computational complexity and cost, improving the accuracy of data sample selection, and effectively improving the effect of subsequent business prediction under the same conditions.

[0041] In one embodiment, Figure 2This is a flowchart of another data filtering method provided in an embodiment of the present invention. Based on the above embodiments, this embodiment configures a fitting threshold value associated with the objective function of the initial business sample data according to preset business filtering requirements; classifies the initial business sample data according to the sample distance between the fitting threshold value and the initial business sample data to obtain the corresponding first candidate sample set and second candidate sample set; and further refines the target sample set by performing set permutation on the sample subset in the first candidate sample set and the corresponding sample subset in the second candidate sample set based on the pre-configured set permutation conditions.

[0042] like Figure 2 As shown, the data filtering method in this embodiment may specifically include the following steps:

[0043] S210, Configure the target function for the initial business sample data.

[0044] In this embodiment, an objective function for the initial business sample data can be manually constructed. This objective function includes the total number of initial business sample data, the target business sample data to be achieved, and conditional parameters. To ensure the accuracy of the screening, this invention also introduces a time variable.

[0045] For example, to facilitate a better understanding of the construction of the objective function, in this embodiment, the objective function can be expressed by the formula: Objective function Y = f(X|θ), where Y represents the total number of initial business sample data, X represents the target business sample data to be achieved, and θ represents the condition parameter. Without loss of generality, the sample set A at time t can be set as follows: t ={(X 1t ,Y 1t ,t),…,(X Ntt ,Y Ntt ,t)}, where N represents the dimension.

[0046] S220. Determine the likelihood value of the objective function based on the preset business screening requirements.

[0047] The likelihood value can characterize the likelihood level of the objective function. Generally speaking, when the preset business screening requirements involve financial transactions, the likelihood level of the objective function is relatively high; when the preset business screening requirements involve business management, the likelihood level of the objective function is relatively low. It can be understood that the likelihood level of the objective function is related to the preset business screening requirements and can be set according to the preset business screening requirements. This embodiment does not impose any restrictions on this.

[0048] S230. Determine the corresponding fitting threshold value based on the likelihood value and the AIC that matches the objective function.

[0049] AIC can be understood as a standard for measuring the goodness of fit of the objective function. It can weigh the complexity of the objective function and the goodness of fit of the AIC data. It can be expressed by the formula AIC = 2·k-2·ln(Y), where Y represents the objective function, which is the likelihood function.

[0050] In this embodiment, the AIC that matches the objective function can be selected. The corresponding fitting threshold value can be determined based on the likelihood value of the objective function and the AIC that matches the objective function. For example, if the likelihood value of the objective function is C, then the fitting threshold value can be expressed as C multiplied by AIC. This fitting threshold value is the lowest fitting threshold value when screening the initial business sample data.

[0051] S240. Classify the initial business sample data according to the distance between the samples to obtain the corresponding initial sample set.

[0052] The initial sample set can be understood as the sample set obtained by initially classifying the initial business sample data. In this embodiment, the initial sample set may include at least two sample subsets.

[0053] In this embodiment, the distance between samples of the initial business sample data can be determined by a preset distance formula. The initial business sample data is then classified according to the distance between samples and a preset clustering algorithm to obtain the corresponding initial sample set. Specifically, the initial business sample data is normalized in different dimensions to obtain the first normalized business sample data. Based on the calculation of the distance between samples, the initial business sample data is classified according to the distance between samples and a preset clustering algorithm to obtain the corresponding initial sample set.

[0054] In one embodiment, S240 includes S2401-S2403:

[0055] S2401. Normalize the initial business sample data in the first dimension to obtain the first normalized business sample data.

[0056] The first dimension includes the first variable, objective function, and time variable corresponding to the initial business sample data. The first normalized business sample data refers to the normalized sample data obtained by modifying the initial business sample data in the first dimension.

[0057] In this embodiment, the initial business sample data can be normalized in the first dimension. This can be understood as modifying the initial business sample data in the first dimension to obtain the first normalized business sample data. Specifically, the normalization of the initial business sample data can be performed manually to ensure that the modified initial business sample data does not exhibit large fluctuations.

[0058] In one embodiment, the initial business sample data is normalized along a first dimension to obtain first normalized business sample data, including:

[0059] The first variable of the initial business sample data is transformed into a second variable that is related to the first variable and the first volatility coefficient;

[0060] Transform the time variables of the initial business sample data into new time variables that are related to the time variables and the second oscillation coefficient;

[0061] The second variable, the objective function, and the new time variable are used to obtain the corresponding first normalized business sample data.

[0062] In this embodiment, the normalization process includes: converting the first variable of the initial business sample data into a first ratio of the first variable and the first fluctuation coefficient, and using the first ratio as the converted second variable; converting the time variable of the initial business sample data into a second ratio of the time variable and the second fluctuation coefficient, and using the second ratio as the new time variable; and combining the second variable, the objective function, and the new time variable to form the first normalized business sample data corresponding to the first dimension.

[0063] For example, to better understand the transformation method of the initial business sample data in the first dimension, this embodiment uses a formula to further explain the processing of the initial business sample data in the first dimension. Specifically, the first dimension is expressed by the formula: (X, Y, t), where X represents the first variable, Y represents the objective function, and t represents the time variable. After processing and transformation in the first dimension, the initial business sample data can be transformed into... Where, Δf X σ represents the partial derivative of the objective function, used to normalize X to minimize fluctuations; σ represents the standard deviation, reflecting the dispersion of the initial business sample data, indicating a stable state. For the new first-time variable after the modification, This indicates the second variable.

[0064] S2402. Determine the first sample distance corresponding to the first normalized business sample data in the first dimension according to the preset first distance formula.

[0065] In this embodiment, the preset first distance formula can be understood as the formula for calculating the distance between samples in the first dimension of the initial business sample data. For example, the distance calculation formula between two sample data points in the initial business sample data can be expressed as: Where X is the first variable in n dimensions, Y is the objective function in m dimensions, and t represents the time variable. σ represents the partial derivative of the objective function in the first dimension, and σ represents the fluctuation coefficient; the inter-sample distance of the initial business sample data in the first dimension can be calculated using this inter-sample distance formula.

[0066] S2403. Classify the first normalized business sample data according to the preset clustering algorithm and the distance between the first samples to obtain the corresponding initial sample set.

[0067] The preset clustering algorithm can be partition-based k-means and its derivatives, or density-based DBSCAN algorithm. This embodiment does not impose any restrictions on this.

[0068] In this embodiment, the first normalized business sample data can be classified according to a preset clustering algorithm and the distance between the first samples to obtain the corresponding initial sample set. Specifically, cluster centers can be selected from the first normalized business sample data, and multiple clustering analyses can be performed based on the cluster centers and the distance between the first samples to obtain the corresponding initial sample set.

[0069] In one embodiment, the first normalized business sample data is classified according to a preset clustering algorithm and the distance between first samples to obtain a corresponding initial sample set, including:

[0070] At least one business sample data point is randomly selected from the first normalized business sample data as a cluster center point, and the cluster center point is used as the current cluster center point.

[0071] Based on the distance between each first sample, other business sample data points are sequentially clustered to the nearest current cluster center point to form at least one cluster set;

[0072] Select the mean of the business sample data in each cluster set, use the mean as the next cluster center, and use the next cluster center as the current cluster center. Return to the steps of sequentially clustering other business sample data points to the nearest current cluster center according to the distance between each first sample, until the movement range of the next cluster center is within a preset range, or the number of clusterings of the next cluster center reaches a preset number of clusterings, and obtain each initial sample set.

[0073] In this embodiment, at least one business sample data point is randomly selected from the first normalized business sample data as a cluster center point, and this cluster center point is used as the current cluster center point. Other business sample data points are then sequentially clustered to the nearest current cluster center point according to the distance between each first sample, forming at least one cluster set. The mean of the business sample data in each cluster set is selected, and this mean is used as the next cluster center point, which is then used as the current cluster center point. The process of sequentially clustering other business sample data points to the nearest current cluster center point according to the distance between each first sample is repeated until the movement range of the next cluster center point is within a preset range, or the number of clustering operations for the next cluster center point reaches a preset number of clustering operations, thus obtaining each initial sample set. For example, based on the K-means clustering algorithm and the distance between the first samples, s sample classification sets are formed, resulting in A. 1 ,…,A s Simultaneously, each sample set corresponds to the centroid of a sample classification set, i.e., A. 1 ,…,A s The centroid of the corresponding sample classification set is denoted as a. 1 ,…,a s .

[0074] S250. Classify the samples based on the AIC value and fitting threshold value of a portion of the initial sample set to obtain the corresponding first candidate sample set and second candidate sample set.

[0075] In this embodiment, when classifying the initial sample set based on the initial sample set, the classification can be based on the AIC value and fitting threshold value of some sample sets in the initial sample set to obtain the corresponding first candidate sample set and second candidate sample set. Specifically, the distance between target samples can be obtained based on the centroid of the sample set corresponding to each sample set in the initial sample set and the preset second distance formula, and the first candidate sample set and second candidate sample set can be obtained according to the preset sorting rules and the distance between target samples.

[0076] In one embodiment, S250 includes S2501-S2505:

[0077] S2501. Determine the centroid of each sample set in the initial sample set.

[0078] In this embodiment, the centroid of each sample set in the initial sample set can be determined by means calculation or centroid calculation. It can be understood that each sample set in the initial sample set corresponds to a centroid of the corresponding sample set.

[0079] S2502. Normalization processing of the initial business sample data in the second dimension yields the second normalized business sample data.

[0080] The second normalized business sample data refers to the normalized sample data obtained by modifying the initial business sample data in the second dimension. The second dimension includes the third variable and the time variable corresponding to the initial business sample data.

[0081] In this embodiment, the normalization processing of the initial business sample data in the second dimension yields the second normalized business sample data. This can be understood as modifying the initial business sample data in the second dimension to obtain the second normalized business sample data.

[0082] In one embodiment, normalization processing of the initial business sample data in the second dimension yields second normalized business sample data, including:

[0083] The third variable of the initial business sample data is transformed into a fourth variable that is related to the third variable and the first volatility coefficient;

[0084] Transform the time variable of the initial business sample data into a new second time variable that is related to the time variable and the second oscillation coefficient;

[0085] The fourth variable and the new second time variable are used to obtain the corresponding second normalized business sample data.

[0086] In this embodiment, the third variable of the initial business sample data is converted into the ratio of the third variable to the first fluctuation coefficient, and this ratio is used as the fourth variable. The time variable of the initial business sample data is converted into the ratio of the time variable and the second fluctuation coefficient, and this ratio is used as the new second time variable. Then, the fourth variable and the new second time variable are combined to form the corresponding second normalized business sample data.

[0087] For example, to facilitate a better understanding of the transformation method of the initial business sample data in the second dimension, this embodiment uses formulas to further explain the processing of the initial business sample data in the second dimension. That is, the initial business sample data in the second dimension is expressed by the formula (X,t), and the transformation process is as follows: Where, Δf X σ represents the partial derivative of the objective function, used to normalize X to minimize fluctuations; σ represents the standard deviation, reflecting the dispersion of the initial business sample data, indicating a stationary state. Represented as the third variable, This is represented as a new second time variable.

[0088] S2503. Determine the target sample distance from the second normalized business sample data to the centroid of each sample set according to the preset second distance formula.

[0089] In this embodiment, the preset second distance formula refers to the sample distance between the defined second normalized business sample data and the centroid of each sample set, which can be expressed by the formula:

[0090] Where X is the third variable in n dimensions, and t represents the time variable. σ represents the partial derivative of the objective function in the second dimension, and σ represents the fluctuation coefficient. The inter-sample distance of the initial business sample data in the second dimension can be calculated using this inter-sample distance formula, which is the target inter-sample distance from the second normalized business sample data to the centroid of each sample set.

[0091] S2504. Sort each subset of samples in the initial sample set according to the distance between the target samples to obtain the sorted candidate sample set.

[0092] In this embodiment, each subset of samples in the initial sample set is sorted according to the distance between the target samples to obtain a sorted candidate sample set. This can be understood as sorting the target samples from near to far or from far to near according to the distance between them to obtain a sorted candidate sample set.

[0093] S2505. Based on the fitting threshold value and AIC value, a subset of samples is removed from the candidate sample set from farthest to nearest. The subset of samples whose ACI values ​​meet the fitting threshold value is formed into the first candidate sample set, and the subset of samples removed is formed into the second candidate sample set.

[0094] The conditions for the ACI value to meet the fitting threshold are as follows: the ACI values ​​corresponding to the first initial business sample data to the z-th initial business sample data are greater than or equal to the fitting threshold value, and the ACI values ​​corresponding to the first initial business sample data to the z-th initial business sample data are less than the fitting threshold value.

[0095] In this embodiment, for the sorted candidate sample set, a subset of samples is removed from the candidate sample set from farthest to near according to the fitting threshold value and ACI value. The subset of samples whose ACI values ​​meet the fitting threshold value is formed into a first candidate sample set, and the removed sample subsets are formed into a second candidate sample set.

[0096] For example, to facilitate a better understanding of the classification method of the first candidate sample set and the second candidate sample set, this embodiment provides a detailed explanation using an example. In this embodiment, the initial business sample data in the second dimension is represented as (X,t), and the transformation process is as follows: The distance between the initial sample set and the second normalized business sample data is defined as: the set mass a corresponding to each subset of samples in the initial sample set. s to and Euclidean distance l s In this embodiment, the method for locating the first candidate sample set, which involves the minimum number of sets, is as follows: without loss of generality, the first candidate sample set is sorted from nearest to farthest as A. 1 ,…,A s The samples are removed one by one from the initial sample set, from farthest to nearest, until a subset A exists. z This makes it possible to have and Then A 1 ,…,A z The set of samples that is least involved, i.e. the set of the first candidate samples, is denoted as G = {A}. 1 ,…,A z}; then the remainder set A z+1 ,…,A s The set formed is a non-minimum involved set, i.e., the second candidate sample set, denoted as G. c ={A z+1 ,…,A s}

[0097] S260. Sort each sample subset in the first candidate sample set according to the first sorting method to obtain the first target sample set.

[0098] In this embodiment, the first target sample set is obtained by sorting each sample subset in the first candidate sample set according to the first sorting method. It should be noted that the first sorting method can be sorting from far to near according to the distance between the centroid of each sample subset in the first candidate sample set and each sample data in the corresponding sample subset, or sorting from near to far.

[0099] S270. Sort each subset of samples in the second candidate sample set according to the second sorting method to obtain the second target sample set; wherein, the sorting order of the first sorting method and the second sorting method is reversed.

[0100] In this embodiment, the second target sample set is obtained by sorting each sample subset in the second candidate sample set according to the second sorting method. In this embodiment, the first sorting method can be sorting from farthest to nearest, or from nearest to farthest, according to the distance between the centroid of each sample subset in the second candidate sample set and each sample data within the corresponding sample subset. It should be noted that the sorting methods of the corresponding sample subsets in the first and second candidate sample sets are opposite. It can be understood that if the first sorting method is sorting from farthest to nearest, then the second sorting method is sorting from nearest to farthest; similarly, if the first sorting method is sorting from nearest to farthest, then the second sorting method is sorting from farthest to nearth.

[0101] S280. Obtain the permuted target sample set based on the first target sample set, the second target sample set, and the pre-configured set permutation conditions.

[0102] In this embodiment, based on the sorted first and second target sample sets and the pre-configured set permutation conditions, a permuted target sample set can be obtained. Specifically, edge sample points of the farthest sample set are selected from the first target sample set, and points near the centroid of the nearest sample set are selected from the second target sample set. The edge sample points of the farthest sample set and the points near the centroid of the nearest sample set are permuted according to the pre-configured set permutation conditions. This operation is repeated until the permutation conditions can no longer be met to obtain the permuted target sample set.

[0103] In one embodiment, obtaining the permuted target sample set based on a first target sample set, a second target sample set, and pre-configured set permutation conditions includes:

[0104] Select edge sample points of the farthest sample set from the first target sample set, and use the edge sample points of the farthest sample set as the current farthest sample set; wherein, the edge sample points are sample points in the farthest sample set whose distance from the centroid of the farthest sample set is greater than a preset first distance threshold.

[0105] Select the centroid-nearest point of the nearest sample set from the second target sample set, and use the centroid-nearest point of the nearest sample set as the current nearest sample set; wherein, the centroid-nearest point is the sample point in the nearest sample set whose distance from the centroid of the nearest sample set is less than a preset second distance threshold;

[0106] If the number of first sample points in each sample subset of the first candidate sample set is greater than the number of second sample points in each sample subset of the second candidate sample set, and the fitting threshold value associated with the objective function is satisfied, the edge sample points of the farthest sample set and the centroid-near points of the nearest sample set are swapped to obtain the target sample subset. Then, the step of selecting the edge sample points of the farthest sample set from the first target sample set and using the edge sample points of the farthest sample set as the current farthest sample set is repeated until the edge sample points in the sample subset of the first target sample set and the centroid-near points in the sample subset of the second target sample set can no longer be swapped.

[0107] Each subset of target samples is combined into a target sample set, which is then used as the key information data after the initial business sample data has been filtered.

[0108] In this embodiment, from the first target sample set, each sample point in the farthest sample set whose distance from the centroid of the farthest sample set is greater than a preset first distance threshold is selected. This sample point is designated as an edge sample point of the farthest sample set, and the edge sample point of the farthest sample set is designated as the current farthest sample set. From the second target sample set, each sample point in the nearest sample set whose distance from the centroid of the nearest sample set is less than a preset second distance threshold is selected as a point near the centroid of the nearest sample set, and the point near the centroid of the nearest sample set is designated as the current nearest sample set. The number of first sample points in each sample subset of the first candidate sample set is greater than that in the second candidate sample set. If the number of second sample points in each sample subset is determined and the fitting threshold associated with the objective function is satisfied, the edge sample points of the farthest sample set and the centroid-near points of the nearest sample set are swapped to obtain the target sample subset. The process of selecting the edge sample points of the farthest sample set from the first target sample set and using the edge sample points of the farthest sample set as the current farthest sample set is repeated until the edge sample points in the sample subsets of the first target sample set and the centroid-near points in the sample subsets of the second target sample set can no longer be swapped. The target sample subsets are then combined to form the target sample set, which is used as the key information data after the initial business sample data is filtered.

[0109] For example, to facilitate a better understanding of set permutation methods, this embodiment illustrates the set permutation method using an example. First, take the farthest set A from the least involved set set set. z Edge sample points, which refer to set A z The sample point and its mass a z The distance (defined as d) y Select sample points with larger distances from the edge. Furthermore, the sample points have been sorted in descending order. Take the nearest set A that has not yet entered the least involved set. z+1 The point near the centroid, where the point near the centroid refers to the set A. z+1 The sample point and its centroid a z+1 The distance (defined as d) y Sample points with a small distance (in distance). Several points near this centroid are selected. Furthermore, the sample points have been sorted in ascending order. Sample permutation will yield set A. z edge sample points With the obtained set A z+1 points near the center of mass Perform the replacement. Replacement requirements: (1) The number of edge sample points is greater than the number of points near the centroid, that is, the number of sample points to be replaced is greater than the number of sample points to be replaced, m>n; (2) Test and ensure that the AIC of the sample model does not exceed c·AIC, otherwise do not replace. A z Continue the permutation: Following the steps described above, move the permutation to the next nearest set A that has not yet entered the least involved set. z+2 ,…,A s Perform sample permutations one by one until set A is reached. z Unable to remove. In the set involving the least number of sets, from farthest to nearest (i.e., from A...). z-1 To A 1 ) one by one to the set G that is not the least involved set. c ={A z+1 ,…,A s From near to far (i.e., from A) z+1 To A s The process of substitution continues until no set can be replaced. The final set containing the fewest remaining samples is the key information data.

[0110] The technical solution described in this embodiment determines the fitting threshold by using the likelihood value of the objective function and the AIC that matches the objective function. It then classifies the initial business sample data based on the inter-sample distance to obtain an initial sample set. Based on the AIC values ​​and fitting thresholds of a subset of samples in the initial sample set, it classifies them to obtain a first candidate sample set and a second candidate sample set. Each subset of samples in the first candidate sample set and each subset of samples in the second candidate sample set are sorted according to a preset sorting method to obtain a first target sample set and a second target sample set. Finally, based on the first target sample set, the second target sample set, and pre-configured set permutation conditions, a permuted target sample set is obtained. This approach, by setting a fitting threshold, minimizes the target sample set through set permutation conditions, effectively reducing computational complexity and cost, improving the accuracy of data sample selection, and enhancing the effectiveness of subsequent business prediction under the same conditions.

[0111] In one embodiment, Figure 3 This is a structural block diagram of a data filtering device according to an embodiment of the present invention. This device is suitable for filtering large amounts of business sample data to reduce the volume of business sample data. The device can be implemented in hardware or software. It can be configured in an electronic device to implement a data filtering method according to an embodiment of the present invention. Figure 3 As shown, the device includes: a configuration module 310, a classification module 320, and a replacement module 330.

[0112] The configuration module 310 is used to configure the fitting threshold value associated with the objective function of the initial business sample data according to the preset business screening requirements.

[0113] The classification module 320 is used to classify the initial business sample data according to the fitting threshold value and the inter-sample distance of the initial business sample data, so as to obtain the corresponding first candidate sample set and second candidate sample set.

[0114] The permutation module 330 is used to perform set permutation on a sample subset in the first candidate sample set and a corresponding sample subset in the second candidate sample set based on pre-configured set permutation conditions to obtain a corresponding target sample set, so as to use the sample data in the target sample set for business prediction; wherein, the pre-configured set permutation conditions satisfy: the number of first sample points in each sample subset in the first candidate sample set is greater than the number of second sample points in the corresponding sample subset in the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function.

[0115] In this embodiment of the invention, with a fitting threshold value associated with the objective function set, the classification module classifies the initial business sample data based on the sample distance and the fitting threshold value. The permutation module permutes the classified set based on pre-configured set permutation conditions to obtain the target sample set. This allows for minimizing the target sample set by using set permutation conditions while setting the fitting threshold value, effectively reducing computational complexity and cost, improving the accuracy of data sample selection, and effectively improving the performance of subsequent business predictions under the same conditions.

[0116] In one embodiment, the configuration module 310 includes:

[0117] The function configuration module is used to configure the target function for the initial business sample data;

[0118] The first determining module is used to determine the likelihood value of the objective function based on preset business screening requirements;

[0119] The second determining module is used to determine the corresponding fitting threshold value based on the likelihood value and the Akaike Information Content Criterion (AIC) that matches the objective function.

[0120] In one embodiment, the classification module 320 includes:

[0121] An initial set determination unit is used to classify the initial business sample data according to the inter-sample distance of the initial business sample data to obtain a corresponding initial sample set;

[0122] The candidate set determination unit is used to classify the samples based on the AIC value of a portion of the initial sample set and the fitting threshold value to obtain the corresponding first candidate sample set and second candidate sample set.

[0123] In one embodiment, the initial set determination unit includes:

[0124] The processing subunit is used to normalize the initial business sample data in the first dimension to obtain the first normalized business sample data.

[0125] The distance determination subunit is used to determine the first inter-sample distance corresponding to the first normalized business sample data on the first dimension according to a preset first distance formula; wherein, the first dimension includes: the first variable, the objective function, and the time variable corresponding to the initial business sample data;

[0126] The set determines the sub-unit, which is used to classify the first normalized business sample data according to the preset clustering algorithm and the distance between the first samples, so as to obtain the corresponding initial sample set.

[0127] In one embodiment, the processing subunit is further configured to:

[0128] The first variable of the initial business sample data is transformed into a second variable that is related to the first variable and the first fluctuation coefficient;

[0129] The time variable of the initial business sample data is transformed into a new time variable that is related to the time variable and the second oscillation coefficient;

[0130] The second variable, the objective function, and the new time variable are used to obtain the corresponding first normalized business sample data.

[0131] In one embodiment, the set of determining sub-units is further used for:

[0132] At least one business sample data point is randomly selected from the first normalized business sample data as a cluster center point, and the cluster center point is used as the current cluster center point;

[0133] The other business sample data points are sequentially clustered to the nearest current cluster center point according to the distance between each of the first samples, so as to form at least one cluster set;

[0134] The mean of the business sample data in each of the cluster sets is selected, and the mean is used as the next cluster center point. The next cluster center point is used as the current cluster center point. The step of sequentially clustering the other business sample data points to the nearest current cluster center point according to the distance between each of the first samples is returned until the movement range of the next cluster center point is within a preset range, or the number of clusterings of the next cluster center point reaches a preset number of clusterings, thus obtaining each of the initial sample sets.

[0135] In one embodiment, the candidate set determination unit further includes:

[0136] The centroid determination subunit is used to determine the centroid of the sample set corresponding to each sample subset in the initial sample set.

[0137] The processing subunit is used to perform normalization processing on the initial business sample data in the second dimension to obtain the second normalized business sample data.

[0138] The distance determination subunit is used to determine the target sample distance from the second normalized business sample data to the centroid of each sample set according to a preset second distance formula.

[0139] The sorting subunit is used to sort each subset of samples in the initial sample set according to the distance between the target samples, so as to obtain a sorted candidate sample set;

[0140] The sub-unit is used to remove sample subsets from the candidate sample set from far to near according to the fitting threshold value, and to form a first candidate sample set by the sample subsets in the candidate sample set whose ACI values ​​satisfy the fitting threshold value, and to form a second candidate sample set by the removed sample subsets.

[0141] The conditions under which the ACI value satisfies the fitting threshold include: the ACI values ​​corresponding to the first initial business sample data to the z-th initial business sample data are greater than or equal to the fitting threshold, and the ACI values ​​corresponding to the first initial business sample data to the z-th initial business sample data are less than the fitting threshold.

[0142] In one embodiment, the second dimension includes a third variable and a time variable corresponding to the initial business sample data; the processing subunit is further configured to:

[0143] The third variable of the initial business sample data is transformed into a fourth variable that is related to the third variable and the first fluctuation coefficient;

[0144] The time variable of the initial business sample data is transformed into a new time variable that is related to the time variable and the second oscillation coefficient;

[0145] The fourth variable and the new time variable are used to obtain the corresponding second normalized business sample data.

[0146] In one embodiment, the replacement module includes:

[0147] The first sorting unit is used to sort each sample subset in the first candidate sample set according to the first sorting method to obtain the first target sample set;

[0148] The second sorting unit is used to sort each sample subset in the second candidate sample set according to the second sorting method to obtain the second target sample set; wherein the sorting order of the first sorting method and the second sorting method is reversed.

[0149] The permutation unit is used to obtain the permuted target sample set based on the first target sample set, the second target sample set, and pre-configured set permutation conditions.

[0150] In one embodiment, the replacement unit includes:

[0151] The first set selection subunit is used to select the edge sample points of the farthest sample set from the first target sample set, and take the edge sample points of the farthest sample set as the current farthest sample set; wherein, the edge sample points are sample points in the farthest sample set whose distance from the centroid of the farthest sample set is greater than a preset first distance threshold.

[0152] The second set selection subunit is used to select the centroid vicinity point of the nearest sample set from the second target sample set, and take the centroid vicinity point of the nearest sample set as the current nearest sample set; wherein, the centroid vicinity point is the sample point in the nearest sample set whose distance from the centroid of the nearest sample set is less than a preset second distance threshold.

[0153] The permutation subunit is used to permutate the edge sample points of the farthest sample set and the centroid-near points of the nearest sample set when the number of first sample points in each sample subset of the first candidate sample set is greater than the number of second sample points in each sample subset of the second candidate sample set, and the fitting threshold value associated with the objective function is satisfied. This results in the target sample subset being obtained by permuting the sample sets of the edge sample points of the farthest sample set and the edge sample points of the farthest sample set being used as the current farthest sample set. This process continues until the edge sample points in the sample subset of the first target sample set and the centroid-near points in the sample subset of the second target sample set can no longer be permuted.

[0154] The sub-unit is used to combine the various target sample subsets into the target sample set, and to use the target sample set as the key information data after the initial business sample data is filtered.

[0155] The data filtering device provided in the embodiments of the present invention can execute the data filtering method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0156] In one embodiment, Figure 4 This is a schematic diagram of an electronic device provided for an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0157] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0158] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0159] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data filtering methods.

[0160] In some embodiments, the data filtering method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data filtering method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data filtering method by any other suitable means (e.g., by means of firmware).

[0161] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data filtering device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0165] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0166] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0167] In one embodiment, the present invention further includes a computer program product, the computer program product comprising a computer program that, when executed by a processor, implements the data filtering method described in any embodiment of the present invention.

[0168] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data filtering method provided in any embodiment of this application.

[0169] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A data filtering method, characterized in that, include: Configure fitting threshold values ​​that are associated with the objective function of the initial business sample data according to the preset business screening requirements; The initial business sample data is classified according to the fitting threshold value and the inter-sample distance of the initial business sample data to obtain the corresponding first candidate sample set and second candidate sample set; Based on pre-configured set permutation conditions, a subset of samples in the first candidate sample set and a subset of samples in the second candidate sample set are permuted to obtain a corresponding target sample set, so as to use the sample data in the target sample set for business prediction. Wherein, the pre-configured set permutation conditions satisfy: the number of first sample points in each sample subset in the first candidate sample set is greater than the number of second sample points in the corresponding sample subset in the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function; The step of classifying the initial business sample data based on the sample distance between the fitted threshold value and the initial business sample data to obtain the corresponding first candidate sample set and second candidate sample set includes: The initial business sample data is classified according to the inter-sample distance to obtain the corresponding initial sample set; Based on the AIC value of a portion of the sample set in the initial sample set and the fitting threshold value, classification is performed to obtain the corresponding first candidate sample set and second candidate sample set; The step of classifying samples based on the AIC values ​​of a subset of samples in the initial sample set and the fitting threshold value to obtain corresponding first candidate sample sets and second candidate sample sets includes: Determine the centroid of the sample set corresponding to each subset of samples in the initial sample set; The normalization process performed on the initial business sample data in the second dimension yields the second normalized business sample data. The distance between the target samples from the second normalized business sample data to the centroid of each sample set is determined according to the preset second distance formula; Sort each subset of samples in the initial sample set according to the distance between the target samples to obtain the sorted candidate sample set; According to the fitting threshold, a subset of samples is removed from the candidate sample set from far to near, and the subset of samples whose AIC values ​​satisfy the fitting threshold are formed into a first candidate sample set, and the removed sample set is formed into a second candidate sample set. The step of removing a subset of samples from the candidate sample set from farthest to nearest according to the fitted threshold value includes: The sample subsets are removed one by one from the candidate sample set in order from farthest to nearest, until the AIC value corresponding to the unremoved sample subset in the candidate sample set satisfies the fitting threshold value. The conditions under which the AIC value satisfies the fitting threshold include: the AIC value corresponding to the set consisting of the first sample subset to the z-th sample subset in the first candidate sample set is greater than or equal to the fitting threshold, and the AIC value corresponding to the set consisting of the first sample subset to the (z-1)-th sample subset in the first candidate sample set is less than the fitting threshold.

2. The method according to claim 1, characterized in that, The step of configuring a fitting threshold value associated with the objective function of the initial business sample data according to preset business screening requirements includes: Configure the objective function for the initial business sample data; The likelihood value of the objective function is determined based on the preset business screening requirements; The corresponding fitting threshold is determined based on the likelihood value and the Akaike Information Criterion (AIC) that matches the objective function.

3. The method according to claim 1, characterized in that, The step of classifying the initial business sample data according to the inter-sample distance to obtain the corresponding initial sample set includes: The initial business sample data is normalized along the first dimension to obtain the first normalized business sample data. The first inter-sample distance of the first normalized business sample data on the first dimension is determined according to a preset first distance formula; wherein, the first dimension includes: the first variable, the objective function, and the time variable corresponding to the initial business sample data; The first normalized business sample data is classified according to the preset clustering algorithm and the distance between the first samples to obtain the corresponding initial sample set.

4. The method according to claim 3, characterized in that, The normalization process performed on the initial business sample data in the first dimension to obtain the first normalized business sample data includes: The first variable of the initial business sample data is transformed into a second variable that is related to the first variable and the first fluctuation coefficient; The time variable of the initial business sample data is transformed into a new time variable that is related to the time variable and the second oscillation coefficient; The second variable, the objective function, and the new time variable are used to obtain the corresponding first normalized business sample data.

5. The method according to claim 3, characterized in that, The step of classifying the first normalized business sample data according to a preset clustering algorithm and the first sample distance to obtain a corresponding initial sample set includes: At least one business sample data point is randomly selected from the first normalized business sample data as a cluster center point, and the cluster center point is used as the current cluster center point; Based on the distance between each of the first samples, other business sample data points are sequentially clustered to the nearest current cluster center point to form at least one cluster set; The mean of the business sample data in each of the cluster sets is selected, and the mean is used as the next cluster center point. The next cluster center point is used as the current cluster center point. The step of sequentially clustering the other business sample data points to the nearest current cluster center point according to the distance between each of the first samples is returned until the movement range of the next cluster center point is within a preset range, or the number of clusterings of the next cluster center point reaches a preset number of clusterings, thus obtaining each of the initial sample sets.

6. The method according to claim 1, characterized in that, The second dimension includes a third variable and a time variable corresponding to the initial business sample data; the normalization process performed on the initial business sample data in the second dimension to obtain the second normalized business sample data includes: The third variable of the initial business sample data is transformed into a fourth variable that is related to the third variable and the first fluctuation coefficient; The time variable of the initial business sample data is transformed into a new time variable that is related to the time variable and the second oscillation coefficient; The fourth variable and the new time variable are used to obtain the corresponding second normalized business sample data.

7. The method according to claim 1, characterized in that, The step of performing set permutations on a subset of samples in the first candidate sample set and a corresponding subset of samples in the second candidate sample set based on pre-configured set permutation conditions to obtain the corresponding target sample set includes: The first target sample set is obtained by sorting each sample subset in the first candidate sample set according to the first sorting method. The second target sample set is obtained by sorting each subset of samples in the second candidate sample set according to the second sorting method; wherein the sorting order of the first sorting method is the reverse of the second sorting method. The permuted target sample set is obtained based on the first target sample set, the second target sample set, and the pre-configured set permutation conditions.

8. The method according to claim 7, characterized in that, The step of obtaining the permuted target sample set based on the first target sample set, the second target sample set, and pre-configured set permutation conditions includes: Select edge sample points of the farthest sample set from the first target sample set, and use the edge sample points of the farthest sample set as the current farthest sample set; wherein, the edge sample point is a sample point in the farthest sample set whose distance from the centroid of the farthest sample set is greater than a preset first distance threshold. Select the centroid-nearest point of the nearest sample set from the second target sample set, and use the centroid-nearest point of the nearest sample set as the current nearest sample set; wherein, the centroid-nearest point is the sample point in the nearest sample set whose distance from the centroid of the nearest sample set is less than a preset second distance threshold; If the number of first sample points in each sample subset of the first candidate sample set is greater than the number of second sample points in each sample subset of the second candidate sample set, and the fitting threshold value associated with the objective function is satisfied, the edge sample points of the farthest sample set and the centroid-near points of the nearest sample set are swapped to obtain the target sample subset. Then, the step of selecting the edge sample points of the farthest sample set from the first target sample set and using the edge sample points of the farthest sample set as the current farthest sample set is repeated until the edge sample points in the sample subset of the first target sample set and the centroid-near points in the sample subset of the second target sample set can no longer be swapped. The target sample set is formed by combining the various target sample subsets, and the target sample set is used as the key information data after the initial business sample data is filtered.

9. A data filtering device, characterized in that, include: The configuration module is used to configure the fitting threshold value associated with the objective function of the initial business sample data according to the preset business screening requirements; The classification module is used to classify the initial business sample data according to the sample distance between the fitting threshold value and the initial business sample data, so as to obtain the corresponding first candidate sample set and second candidate sample set. The permutation module is used to perform set permutation on the sample subset in the first candidate sample set and the corresponding sample subset in the second candidate sample set based on pre-configured set permutation conditions to obtain the corresponding target sample set, so as to use the sample data in the target sample set for business prediction; Wherein, the pre-configured set permutation conditions satisfy: the number of first sample points in each sample subset in the first candidate sample set is greater than the number of second sample points in the corresponding sample subset in the second candidate sample set, and the target sample set satisfies the fitting threshold value associated with the objective function; The classification module includes: An initial set determination unit is used to classify the initial business sample data according to the inter-sample distance of the initial business sample data to obtain a corresponding initial sample set; The candidate set determination unit is used to classify the samples based on the AIC value of a portion of the sample set in the initial sample set and the fitting threshold value to obtain the corresponding first candidate sample set and second candidate sample set. The candidate set determination unit further includes: The centroid determination subunit is used to determine the centroid of the sample set corresponding to each sample subset in the initial sample set. The processing subunit is used to perform normalization processing on the initial business sample data in the second dimension to obtain the second normalized business sample data. The distance determination subunit is used to determine the target sample distance from the second normalized business sample data to the centroid of each sample set according to a preset second distance formula. The sorting subunit is used to sort each subset of samples in the initial sample set according to the distance between the target samples, so as to obtain a sorted candidate sample set; The sub-unit is used to remove sample subsets from the candidate sample set from far to near according to the fitting threshold value, and to form a first candidate sample set by the sample subsets in the candidate sample set whose AIC value satisfies the fitting threshold value, and to form a second candidate sample set by the removed sample subsets. The step of removing a subset of samples from the candidate sample set from farthest to nearest according to the fitted threshold value includes: The sample subsets are removed one by one from the candidate sample set in order from farthest to nearest, until the AIC value corresponding to the unremoved sample subset in the candidate sample set satisfies the fitting threshold value. The conditions under which the AIC value satisfies the fitting threshold include: the AIC value corresponding to the set consisting of the first sample subset to the z-th sample subset in the first candidate sample set is greater than or equal to the fitting threshold, and the AIC value corresponding to the set consisting of the first sample subset to the (z-1)-th sample subset in the first candidate sample set is less than the fitting threshold.

10. The apparatus according to claim 9, characterized in that, The configuration module includes: The target configuration unit is used to configure the target function for the initial business sample data. The likelihood value determination unit is used to determine the likelihood value of the objective function based on preset business screening requirements; The threshold value determination unit is used to determine the corresponding fitting threshold value based on the likelihood value and the Akaike Information Content Criterion (AIC) that matches the objective function.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data filtering method according to any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data filtering method according to any one of claims 1-8.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data filtering method according to any one of claims 1-8.

Images