Feature data processing method, terminal, device and readable storage medium

Abstract

The application discloses a feature data processing method, comprising the following steps: carrying out block processing on a to-be-verified data set, using a distributed thread, a preset distributed calculation formula and a preset likelihood function to respectively calculate the distribution value and the likelihood value corresponding to each data block, determining the overall distribution value and the overall likelihood value corresponding to the to-be-verified data set based on the distribution value and the likelihood value corresponding to each data block, inputting the overall distribution value and the overall likelihood value into a preset score chi-square formula to obtain a chi-square value, and determining the evaluation result corresponding to the feature data in the to-be-verified data set based on the chi-square value. The application further discloses a device, a terminal and a readable storage medium. When the chi-square value of the feature value is calculated by using the chi-square formula and the feature data is evaluated, the distributed calculation mode of the application improves the calculation speed compared with single machine processing, and especially when the sample set is large, the evaluation efficiency of the feature data is further improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a feature data processing method, terminal, device, and readable storage medium. Background Technology

[0002] Machine learning technology is increasingly being used in more and more fields. The accuracy and robustness of machine learning models are crucial, which requires a large amount of high-quality datasets for training.

[0003] Because logistic regression models offer good interpretability, they are commonly used in scenarios requiring interpretability, such as financial risk control and medical models. The training dataset includes feature values ​​and target values. The chi-square score of the feature values ​​is an important quantitative indicator for evaluating features that did not enter the logistic regression model. Currently, chi-square scores are primarily processed using single-machine data processing. However, single-machine memory is limited. If the sample set is large, determining the chi-square score using a single machine will result in slow processing speed, leading to low efficiency in feature data processing. Summary of the Invention

[0004] The main objective of this invention is to provide a feature data processing method, terminal, device, and readable storage medium, aiming to solve the technical problem of low evaluation efficiency when using chi-square scores to evaluate features that have not entered the logistic regression model when the number of samples is large.

[0005] To achieve the above objectives, the present invention provides a feature data processing method, which includes the following steps:

[0006] The dataset to be validated is divided into blocks to obtain a preset number of data blocks;

[0007] Using distributed threads, a pre-defined distributed computation formula, and a pre-defined likelihood function, the distribution value and likelihood value corresponding to each data block are calculated respectively.

[0008] The overall distribution value and overall likelihood value of the dataset to be verified are determined based on the distribution value and likelihood value corresponding to each data block.

[0009] The overall distribution value and the overall likelihood value are input into a preset scoring chi-square formula to obtain the chi-square value, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the chi-square value.

[0010] Furthermore, in one embodiment, before the step of dividing the dataset to be verified into blocks to obtain a preset number of data blocks, the method further includes:

[0011] Input a preset sample set into the prediction model to generate the prediction probability corresponding to the sample set;

[0012] The sample set and the predicted probability are used as the dataset to be verified.

[0013] Furthermore, in one embodiment, the step of calculating the distribution value and likelihood value corresponding to each data block using distributed threads, a preset distributed computing formula, and a preset likelihood function includes:

[0014] The distributed thread is used to obtain data pairs from each data block, and the data pairs are input into the distributed calculation formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block. The data pair is a combination of feature data, target data and prediction probability in the dataset to be verified.

[0015] Further, in one embodiment, the data block comprises multiple sets of data pairs, and the step of inputting the data pairs into the distributed computing formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block includes:

[0016] Input all data pairs of the data block into the distributed computing formula to obtain the sub-distribution value corresponding to each data pair, and then sum the sub-distribution values ​​corresponding to all data pairs in each data block to obtain the distribution value corresponding to each data block.

[0017] Input all data pairs of the data block into the likelihood function to obtain the sub-likelihood value corresponding to each data pair, and then sum the sub-likelihood values ​​corresponding to all data pairs in each data block to obtain the likelihood value corresponding to each data block.

[0018] Further, in one embodiment, the step of inputting the overall distribution value and the overall likelihood value into a preset chi-square formula to obtain the predicted score includes:

[0019] The total distribution value and the total likelihood value are input into the preset chi-square formula to calculate the chi-square value by squaring the total likelihood value and then calculating the chi-square value by combining it with the total distribution value.

[0020] Further, in one embodiment, the step of determining the evaluation result corresponding to the feature data in the dataset to be verified based on the predicted score includes:

[0021] Obtain the degrees of freedom corresponding to the sample set, and determine the chi-square distribution function based on the degrees of freedom;

[0022] The output value corresponding to the chi-square value is obtained based on the chi-square distribution function, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the output value and the threshold.

[0023] Further, in one embodiment, the step of determining the overall distribution value and overall likelihood value corresponding to the dataset to be verified based on the distribution value and likelihood value corresponding to each data block includes:

[0024] The overall distribution value is obtained by summing the distribution values ​​corresponding to each data block.

[0025] The overall likelihood value is obtained by summing the likelihood values ​​corresponding to each data block.

[0026] Further, in one embodiment, the feature data processing apparatus includes:

[0027] The chunking module is used to divide the dataset to be validated into chunks to obtain a preset number of data chunks.

[0028] The calculation module is used to calculate the distribution value and likelihood value of each data block by using distributed threads, preset distributed calculation formulas and preset likelihood functions.

[0029] The determination module is used to determine the overall distribution value and overall likelihood value of the dataset to be verified based on the distribution value and likelihood value corresponding to each data block;

[0030] The evaluation module is used to input the overall distribution value and the overall likelihood value into a preset scoring chi-square formula to obtain the chi-square value, and to determine the evaluation result corresponding to the feature data in the dataset to be verified based on the chi-square value.

[0031] In addition, to achieve the above objectives, the present invention also provides a terminal, the terminal comprising: a memory, a processor, and a feature data processing program stored in the memory and executable on the processor, wherein the feature data processing program, when executed by the processor, implements the steps of the feature data processing method described in any of the above claims.

[0032] In addition, to achieve the above objectives, the present invention also provides a readable storage medium storing a feature data processing program, which, when executed by a processor, implements the steps of the feature data processing method described in any of the above claims.

[0033] This invention divides the dataset to be validated into blocks, obtaining a predetermined number of data blocks. Then, using distributed threads, a predetermined distributed computation formula, and a predetermined likelihood function, the distribution value and likelihood value corresponding to each data block are calculated. Next, based on the distribution value and likelihood value of each data block, the overall distribution value and overall likelihood value of the dataset to be validated are determined. Then, the overall distribution value and overall likelihood value are input into a predetermined scoring chi-square formula to obtain the chi-square value. Based on the chi-square value, the evaluation result corresponding to the feature data in the dataset to be validated is determined. When calculating the predicted score of feature values ​​using the chi-square formula, compared with single-machine processing, the distributed computation method of this application improves the computation speed, especially for larger sample sets, thus improving the evaluation efficiency of feature data. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the terminal structure in the hardware operating environment involved in the embodiments of the present invention;

[0035] Figure 2 This is a flowchart illustrating the first embodiment of the feature data processing method of the present invention;

[0036] Figure 3 This is a flowchart illustrating the second embodiment of the feature data processing method of the present invention;

[0037] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the feature data processing device of the present invention.

[0038] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0040] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal structure in the hardware operating environment involved in the embodiments of the present invention.

[0041] like Figure 1As shown, the terminal may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0042] Optionally, the terminal may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. These sensors may include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display screen according to the ambient light level, while the proximity sensor can turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, an attitude sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's attitude (such as landscape / portrait switching, related games, magnetometer attitude calibration), vibration recognition functions (such as pedometers, taps), etc. Of course, the terminal may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, which will not be elaborated here.

[0043] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0044] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a feature data processing program.

[0045] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the user terminal and communicate with the user terminal; and the processor 1001 can be used to call the feature data processing program stored in the memory 1005.

[0046] In this embodiment, the terminal includes: a memory 1005, a processor 1001, and a feature data processing program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the feature data processing program stored in the memory 1005, it executes the steps of the feature data processing methods provided in the various embodiments of this application.

[0047] The present invention also provides a feature data processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the feature data processing method of the present invention.

[0048] The embodiments of the present invention provide an embodiment of a feature data processing method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0049] In this embodiment, the feature data processing method includes:

[0050] Step S10: Divide the dataset to be verified into blocks to obtain a preset number of data blocks;

[0051] In this embodiment, the chi-square value is a statistic in nonparametric tests, mainly used in nonparametric statistical analysis. It is a key test indicator in the chi-square test, a widely used hypothesis testing method for count data. It falls under the category of nonparametric tests, primarily comparing the correlation between two or more sample rates (proportions) and two categorical variables. Its fundamental idea is to compare the degree of agreement or goodness of fit between theoretical and actual frequencies. Specifically, in this invention, the prediction model is a logistic regression model trained on samples. The sample set includes the original feature dataset and the target dataset corresponding to the original feature dataset. Further, before segmenting the dataset to be validated, new feature datasets are added to the original feature dataset based on a preset format to generate the feature dataset to be tested. For ease of description, the variables are defined as follows:

[0052] Original feature dataset: X′, containing n data points {x′1, x′2, ..., x′} n Each piece of data Contains kt eigenvalues ​​{x′ i1 ,x′ i2 ,...x′ i(k-t)};

[0053] A new feature dataset, X″, contains n data points {x″1, x″2, ..., x″}. n Each piece of data Contains t feature values ​​{x″ i1 ,x″ i2 ,...x″ it)};

[0054] The feature dataset X in the dataset to be verified is the result of concatenating each data point from the original feature dataset X′ and the newly added feature dataset X″, including n data points {x1, x2, ... x″}. n Each piece of data Includes k eigenvalues ​​{x} i1 ,x i2 ,...x ik}, where the first kt feature values ​​come from X′, and the last t feature values ​​come from X″.

[0055] Target dataset: Y, containing n data points {y1, y2, ... y n};

[0056] Predicted probability set: P, containing n data points {p1, p2, ... pn} n};

[0057] The parameters θ0 of the prediction model are: a kt-dimensional vector {θ1, θ2, ... θ} k-t}

[0058] The parameters θ of the prediction model are k-dimensional vectors {θ1, θ2, ... θ}. k-t ,0,...0}

[0059] The chi-square formula S for the prediction model is shown below:

[0060]

[0061] Among them, H θ and U θ The values ​​are respectively:

[0062]

[0063]

[0064] As can be seen from the chi-square calculation formula of the target model, the main computational bottleneck lies in calculating the population distribution value H. θ and overall likelihood value U θ H θ and U θ All calculations can be performed using matrix decomposition on a row-by-row basis. The matrix decomposition of the chi-square formula corresponding to the target model is as follows:

[0065]

[0066]

[0067] Based on the overall distribution value H θ and overall likelihood value U θ The key feature is that the calculations between rows are completely independent during the multiplication phase, allowing for independent distributed computation. The accumulation phase then aggregates the results for each row, making it very convenient to employ distributed computing principles. Specifically, the dataset to be validated is divided into blocks to obtain a predetermined number of data blocks, and then the distribution value and likelihood value corresponding to each data block are calculated separately.

[0068] Step S20: Using distributed threads, a preset distributed calculation formula, and a preset likelihood function, calculate the distribution value and likelihood value corresponding to each data block respectively.

[0069] In this embodiment, a distributed computing approach is adopted. The preset distributed computing formula and the preset likelihood function are respectively performed on each data block for distributed computing, and then the corresponding distribution value and likelihood value are obtained.

[0070] Specifically, step S20 includes:

[0071] Step S21 includes: using the distributed thread to obtain data pairs in each data block, and inputting the data pairs into the distributed calculation formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block, wherein the data pair is a combination of feature data, target data and prediction probability in the dataset to be verified.

[0072] Specifically, step S21 includes:

[0073] Step a: Input all data pairs of the data block into the distributed computing formula to obtain the sub-distribution value corresponding to each data pair, and sum the sub-distribution values ​​corresponding to all data pairs in each data block to obtain the distribution value corresponding to each data block.

[0074] In this embodiment, a data pair is a combination of feature data, target data, and predicted probability in the dataset to be verified. There are multiple data pairs in a data block. The data pairs of the data block are input into the distributed computing formula to obtain the sub-distribution value corresponding to each data pair. The sub-distribution values ​​corresponding to all data pairs in each data block are accumulated to obtain the distribution value corresponding to each data block.

[0075] For example, the sub-distribution value is Substitute the feature dataset and the predicted probability Perform calculations to obtain the specific value of value1, and then accumulate value1 from a data block to obtain the distribution value corresponding to that data block.

[0076] Step b: Input all data pairs of the data block into the likelihood function to obtain the sub-likelihood value corresponding to each data pair, and accumulate the sub-likelihood values ​​corresponding to all data pairs in each data block to obtain the likelihood value corresponding to each data block.

[0077] In this embodiment, a data pair is a combination of feature data, target data, and predicted probability in the dataset to be verified. There are multiple data pairs in a data block. The data pairs of the data block are input into the likelihood function to obtain the sub-likelihood value corresponding to each data pair. The sub-likelihood values ​​corresponding to all data pairs in each data block are accumulated to obtain the likelihood value corresponding to each data block.

[0078] For example, the sub-distribution value is Substitute the feature dataset and target data Perform calculations to obtain the specific value of value2, and then accumulate value2 from a data block to obtain the likelihood value corresponding to that data block.

[0079] Step S30: Determine the overall distribution value and overall likelihood value of the dataset to be verified based on the distribution value and likelihood value corresponding to each data block;

[0080] Specifically, step S30 includes:

[0081] Step c: Sum the distribution values ​​corresponding to each data block to obtain the overall distribution value;

[0082] Step d: Accumulate the likelihood values ​​corresponding to each data block to obtain the overall likelihood value.

[0083] In this embodiment, a distributed computing approach is adopted. The preset distributed computing formula and the preset likelihood function are performed in a distributed manner on each data block to obtain their respective distribution values ​​and likelihood values. Then, based on the distribution values ​​and likelihood values ​​corresponding to each data block, the overall distribution value and overall likelihood value corresponding to the dataset to be verified are determined for chi-square value calculation.

[0084] Specifically, the distribution values ​​corresponding to each data block are summed to obtain the overall distribution value. Similarly, the likelihood values ​​corresponding to each data block are summed to obtain the overall likelihood value.

[0085] Step S40: Input the overall distribution value and the overall likelihood value into the preset scoring chi-square formula to obtain the chi-square value, and determine the evaluation result corresponding to the feature data in the dataset to be verified based on the chi-square value.

[0086] In this embodiment, the overall distribution value and the overall likelihood value are input into a preset scoring chi-square formula to obtain the chi-square value. Then, the evaluation result corresponding to the feature data in the dataset to be verified is further determined. The evaluation result includes two types: qualified evaluation result and unqualified evaluation result.

[0087] Specifically, step S40 includes:

[0088] Step S41: Input the total distribution value and the total likelihood value into the preset chi-square formula so that the total likelihood value is squared and then the chi-square value is calculated by combining it with the total distribution value.

[0089] In this embodiment, the overall distribution value and the overall likelihood value are input into the chi-square formula. According to the chi-square formula, relevant calculations are performed. First, the overall likelihood value is squared to obtain the squared result. Then, the result is combined with the overall distribution value to calculate the chi-square value.

[0090] Step S42: Obtain the degrees of freedom corresponding to the sample set, and determine the chi-square distribution function based on the degrees of freedom;

[0091] Step S43: Obtain the output value corresponding to the chi-square value based on the chi-square distribution function, and determine the evaluation result corresponding to the feature data in the dataset to be verified based on the output value and the threshold.

[0092] In this embodiment, the evaluation result cannot be determined directly by the chi-square value. It is necessary to further obtain the significance of the chi-square value based on the chi-square distribution function, and determine the test result based on the significance of the chi-square value and the threshold.

[0093] The chi-square distribution is the distribution of the sum of squares of n independent random variables that follow a standard normal distribution. Chi-square values ​​are non-negative; a larger chi-square value indicates greater significance. If the significance (SIG.) of the chi-square value is less than 0.05, it indicates that the two variables are significantly correlated. Specifically, the degrees of freedom of the sample set are obtained, the chi-square distribution function is determined based on the degrees of freedom, the output value corresponding to the chi-square value is obtained from the chi-square distribution function, and the output value is compared with a threshold, which can be set to 0.05. When the output value is less than 0.05, the evaluation result of the feature data in the dataset to be validated is determined to be satisfactory; when the output value is greater than or equal to 0.05, the evaluation result of the feature data in the dataset to be validated is determined to be unsatisfactory.

[0094] The feature data processing method proposed in this embodiment divides the dataset to be verified into blocks to obtain a preset number of data blocks. Then, using distributed threads, a preset distributed computing formula, and a preset likelihood function, the distribution value and likelihood value corresponding to each data block are calculated respectively. Next, based on the distribution value and likelihood value corresponding to each data block, the overall distribution value and overall likelihood value corresponding to the dataset to be verified are determined. Then, the overall distribution value and overall likelihood value are input into a preset scoring chi-square formula to obtain the chi-square value, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the chi-square value. When calculating the predicted score of feature values ​​using the chi-square formula, compared with single-machine processing, the distributed computing method of this application improves the computing speed, especially for large sample sets, thus improving the evaluation efficiency of feature data.

[0095] Based on the first embodiment, referring to Figure 3 The second embodiment of the feature data processing method of the present invention is proposed. In this embodiment, before step S10, the method further includes:

[0096] Step S50: Input the preset sample set into the prediction model to generate the prediction probability corresponding to the sample set;

[0097] Step S60: Use the sample set and the predicted probability as the dataset to be verified.

[0098] In this embodiment, the sample set is used as input to the prediction model. The model outputs the predicted value along with its corresponding prediction probability. The prediction probability is generally a value between 0 and 1 that accompanies the predicted value when the model makes its predictions on the input data. A larger value indicates a higher confidence level; generally, a prediction probability greater than or equal to 0.5 indicates a high degree of confidence (more than 50% probability). Therefore, this is a numerical information inherent to the logistic regression model itself. In practical applications, for a given data point, the probability of different predicted outcomes by the logistic regression model varies. The outcome with the highest prediction probability is selected as the final predicted value for that data. Therefore, the predicted value and prediction probability are interdependent in the model. Specifically, a preset sample set is input into the prediction model to generate the prediction probability corresponding to the sample set. The sample set and prediction probability are then used as the dataset to be validated.

[0099] The feature data processing method proposed in this embodiment inputs a preset sample set into the prediction model, generates the prediction probability corresponding to the sample set, and uses the sample set and prediction probability as a dataset to be verified, providing data preparation before distributive computation of chi-square value.

[0100] The present invention further provides a feature data processing apparatus, with reference to Figure 4 , Figure 4This is a schematic diagram of the functional modules of an embodiment of the feature data processing device of the present invention.

[0101] Blocking module 10 is used to divide the dataset to be verified into blocks to obtain a preset number of data blocks.

[0102] The calculation module 20 is used to calculate the distribution value and likelihood value of each data block respectively by using distributed threads, preset distributed calculation formulas and preset likelihood functions.

[0103] The determination module 30 is used to determine the overall distribution value and overall likelihood value of the dataset to be verified based on the distribution value and likelihood value corresponding to each data block;

[0104] The evaluation module 40 is used to input the overall distribution value and the overall likelihood value into a preset scoring chi-square formula to obtain the chi-square value, and to determine the evaluation result corresponding to the feature data in the dataset to be verified based on the chi-square value.

[0105] Furthermore, the feature data processing device further includes:

[0106] Input a preset sample set into the prediction model to generate the prediction probability corresponding to the sample set;

[0107] The sample set and the predicted probability are used as the dataset to be verified.

[0108] Furthermore, the computing module 20 is also used for:

[0109] The distributed thread is used to obtain data pairs from each data block, and the data pairs are input into the distributed calculation formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block. The data pair is a combination of feature data, target data and prediction probability in the dataset to be verified.

[0110] Furthermore, the computing module 20 is also used for:

[0111] Input all data pairs of the data block into the distributed computing formula to obtain the sub-distribution value corresponding to each data pair, and then sum the sub-distribution values ​​corresponding to all data pairs in each data block to obtain the distribution value corresponding to each data block.

[0112] Input all data pairs of the data block into the likelihood function to obtain the sub-likelihood value corresponding to each data pair, and then sum the sub-likelihood values ​​corresponding to all data pairs in each data block to obtain the likelihood value corresponding to each data block.

[0113] Furthermore, the induction module 30 is also used for:

[0114] The total distribution value and the total likelihood value are input into the preset chi-square formula to calculate the chi-square value by squaring the total likelihood value and then calculating the chi-square value by combining it with the total distribution value.

[0115] Furthermore, the evaluation module 40 is also used for:

[0116] Obtain the degrees of freedom corresponding to the sample set, and determine the chi-square distribution function based on the degrees of freedom;

[0117] The output value corresponding to the chi-square value is obtained based on the chi-square distribution function, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the output value and the threshold.

[0118] Furthermore, the device for verifying the chi-square value also includes:

[0119] The first aggregation module is used to accumulate the distribution values ​​corresponding to each data block to obtain the overall distribution value;

[0120] The second aggregation module is used to accumulate the likelihood values ​​corresponding to each data block to obtain the overall likelihood value.

[0121] Furthermore, embodiments of the present invention also propose a readable storage medium storing a feature data processing program, which, when executed by a processor, implements the steps of the feature data processing methods described in the above embodiments.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0123] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a system device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0125] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A feature data processing method, characterized in that, The feature data processing method includes the following steps: A preset sample set is input into the prediction model to generate the prediction probability corresponding to the sample set, wherein the prediction model is a logistic regression model trained on the sample. The sample set and the predicted probability are used as the dataset to be verified, wherein the sample set includes the original feature dataset, the target dataset corresponding to the original feature dataset, and the newly added feature dataset added to the original feature dataset; The dataset to be validated is divided into blocks to obtain a preset number of data blocks; Using distributed threads, a pre-defined distributed computation formula, and a pre-defined likelihood function, the distribution value and likelihood value corresponding to each data block are calculated respectively. The overall distribution value and overall likelihood value of the dataset to be verified are determined based on the distribution value and likelihood value corresponding to each data block. The overall distribution value and the overall likelihood value are input into a preset scoring chi-square formula to obtain the chi-square value, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the chi-square value. The step of calculating the distribution value and likelihood value of each data block using distributed threads, a preset distributed computing formula, and a preset likelihood function includes: The distributed thread is used to obtain data pairs in each data block, and the data pairs are input into the distributed calculation formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block. The data pair is a combination of feature data, target data and prediction probability in the dataset to be verified. The step of determining the overall distribution value and overall likelihood value of the dataset to be verified based on the distribution value and likelihood value corresponding to each data block includes: The overall distribution value is obtained by summing the distribution values ​​corresponding to each data block. The overall likelihood value is obtained by summing the likelihood values ​​corresponding to each data block.

2. The feature data processing method as described in claim 1, characterized in that, The data block comprises multiple sets of data pairs. The step of inputting the data pairs into the distributed computing formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block includes: Input all data pairs of the data block into the distributed computing formula to obtain the sub-distribution value corresponding to each data pair, and then sum the sub-distribution values ​​corresponding to all data pairs in each data block to obtain the distribution value corresponding to each data block. Input all data pairs of the data block into the likelihood function to obtain the sub-likelihood value corresponding to each data pair, and then sum the sub-likelihood values ​​corresponding to all data pairs in each data block to obtain the likelihood value corresponding to each data block.

3. The feature data processing method as described in claim 1, characterized in that, The step of inputting the population distribution value and the population likelihood value into a preset chi-square formula to obtain the chi-square value includes: The total distribution value and the total likelihood value are input into the preset chi-square formula to calculate the chi-square value by squaring the total likelihood value and then calculating the chi-square value by combining it with the total distribution value.

4. The feature data processing method as described in claim 1, characterized in that, The step of determining the evaluation result corresponding to the feature data in the dataset to be verified based on the chi-square value includes: Obtain the degrees of freedom corresponding to the sample set, and determine the chi-square distribution function based on the degrees of freedom; The output value corresponding to the chi-square value is obtained based on the chi-square distribution function, and the evaluation result corresponding to the feature data in the dataset to be verified is determined based on the output value and the threshold.

5. A feature data processing apparatus, characterized in that, The feature data processing device includes: The chunking module is used to divide the dataset to be validated into chunks to obtain a preset number of data chunks. The calculation module is used to calculate the distribution value and likelihood value of each data block by using distributed threads, preset distributed calculation formulas and preset likelihood functions. The determination module is used to determine the overall distribution value and overall likelihood value of the dataset to be verified based on the distribution value and likelihood value corresponding to each data block; The evaluation module is used to input the overall distribution value and the overall likelihood value into a preset scoring chi-square formula to obtain the chi-square value, and to determine the evaluation result corresponding to the feature data in the dataset to be verified based on the chi-square value; The calculation module is further used for: The distributed thread is used to obtain data pairs in each data block, and the data pairs are input into the distributed calculation formula and the likelihood function to obtain the distribution value and likelihood value corresponding to each data block. The data pair is a combination of feature data, target data and prediction probability in the dataset to be verified. The device for testing the chi-square value also includes: The first aggregation module is used to accumulate the distribution values ​​corresponding to each data block to obtain the overall distribution value; The second aggregation module is used to accumulate the likelihood values ​​corresponding to each data block to obtain the overall likelihood value; The feature data processing device is also used for: A preset sample set is input into the prediction model to generate the prediction probability corresponding to the sample set, wherein the prediction model is a logistic regression model trained on the sample. The sample set and the predicted probability are used as the dataset to be verified, wherein the sample set includes the original feature dataset, the target dataset corresponding to the original feature dataset, and the newly added feature dataset added to the original feature dataset.

6. A terminal, characterized in that, The terminal includes: a memory, a processor, and a feature data processing program stored in the memory and executable on the processor, wherein when the feature data processing program is executed by the processor, it implements the steps of the feature data processing method as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a feature data processing program, which, when executed by a processor, implements the steps of the feature data processing method as described in any one of claims 1 to 4.

Images