Data processing method, storage medium and computer terminal

Abstract

The application discloses a data processing method, a storage medium and a computer terminal. The method comprises the following steps: obtaining an original data set and a target variable, wherein the original data set comprises at least original features; performing screening processing on the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; performing screening processing on the original features to obtain a target feature set, wherein the target feature set is used to represent features in the original features that are used to explain the target variable alone and meet a preset statistical distribution; and explaining the target variable based on the target data set and the target feature set to obtain an explanation result, wherein the explanation result is used to represent the influence degree of the features on the target variable. The application solves the technical problem of low accuracy of data explanation in the process of algorithm governance in the related art.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, a storage medium and a computer terminal. Background Art

[0002] Currently, algorithmic governance primarily focuses on effectively explaining models. Depending on the interpretable object, current interpretability learning can be roughly divided into three categories: pre-modeling interpretability methods, model self-interpretation, and explanation of model results. However, current interpretation methods suffer from numerous issues, such as unstable interpretations, non-independent effects of features on variables, and high computational costs for large-scale data. Furthermore, when applying interpretable methods to large-scale datasets, the inherent complexity of the scenarios makes the relationships between multiple variables complex and unknown. Therefore, in practical applications, the accuracy of interpretations of sample data remains to be verified.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present invention provide a data processing method, a storage medium, and a computer terminal to at least solve the technical problem of low accuracy in interpreting data during algorithm governance in related technologies.

[0005] According to one aspect of an embodiment of the present invention, a data processing method is provided, comprising: obtaining an original data set and a target variable, wherein the original data set includes at least original features; performing screening processing on the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; performing screening processing on the original features to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to a preset statistical distribution; interpreting the target variable based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

[0006] According to one aspect of an embodiment of the present invention, a data processing method is provided, comprising: displaying an original data set and a target variable on an interactive interface, wherein the original data set includes at least original features; and displaying an interpretation result corresponding to the target variable on the interactive interface when a target touch operation is sensed in the interactive interface, wherein the interpretation result is generated by interpreting the target variable based on the target data set and a target feature set, the interpretation result is used to represent the degree of influence of the feature on the target variable, the target feature set is obtained by screening the original features, the target feature set is used to represent the features of the original features that are used alone to interpret the target variable and conform to a preset statistical distribution, the target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

[0007] According to another aspect of an embodiment of the present invention, a data processing device is provided, including: an acquisition module, used to acquire an original data set and a target variable, wherein the original data set includes at least original features; a first screening module, used to screen the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; a second screening module, used to screen the original features to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to a preset statistical distribution; an interpretation module, used to interpret the target variable based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

[0008] According to another aspect of an embodiment of the present invention, a data processing device is provided, including: a display module, for displaying an original data set and a target variable on an interactive interface, wherein the original data set includes at least original features; a sensing module, for displaying an interpretation result corresponding to the target variable on the interactive interface when a target touch operation is sensed on the interactive interface, wherein the interpretation result is generated by interpreting the target variable based on the target data set and the target feature set, and the interpretation result is used to indicate the degree of influence of the feature on the target variable, and the target feature set is obtained by screening the original features, and the target feature set is used to indicate the features of the original features that are used alone to interpret the target variable and conform to a preset statistical distribution, and the target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

[0009] According to another aspect of an embodiment of the present invention, a storage medium is further provided, wherein the storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute any one of the above-mentioned data processing methods.

[0010] According to another aspect of an embodiment of the present invention, a computer terminal is provided, comprising: a processor and a memory, wherein the processor is configured to run a program stored in the memory, wherein the program executes any one of the above-mentioned data processing methods when running.

[0011] In an embodiment of the present invention, an original data set and a target variable are first obtained, wherein the original data set includes at least original features, the original data set is screened to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value, the original features are screened to obtain a target feature set, wherein the target feature set and the target feature set are used to represent the features of the original features that are used to explain the target variable alone and conform to the preset statistical distribution; based on the target data set and the target feature set, the target variable is interpreted to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the features on the target variable, thereby improving the accuracy of the interpretation result. It is easy to notice that by screening the original data set, a target data set that can represent the original data set is obtained to improve the calculation speed, and by screening the original features in the original data set, the features that can explain the target variable alone can be removed from the original features, so as to exclude the features that still contribute significantly to the interpretation result after excluding the significance caused by chance, so as to focus on the subsequent interpretation direction, thereby improving the accuracy of the interpretation, thereby solving the technical problem of low accuracy of data interpretation in the process of algorithm governance in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present invention;

[0014] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0015] Figure 3 is an empirical distribution diagram of a target variable according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of a visual display according to an embodiment of the present invention;

[0017] Figure 5 is a flow chart of another data processing method according to an embodiment of the present invention;

[0018] Figure 6 is a flow chart of another data processing method according to an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of a data processing device according to an embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of another data processing device according to an embodiment of the present invention;

[0021] Figure 9 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0025] Interpretable learning: refers to the data, models, and results in the machine learning process that can be understood by humans;

[0026] Model self-explanation: The model itself is explanatory and highly transparent, and the decision logic can be directly understood;

[0027] Post-hoc explanation methods: independent methods that aim to explain the trained target model;

[0028] Prototype: a data instance that is representative of the dataset;

[0029] Test statistic: It is a random variable calculated based on sample data. It is used in hypothesis testing. You can use the test statistic to determine whether to reject the null hypothesis.

[0030] Cumulative local effect: used to describe the average impact of feature variables on the predicted target.

[0031] At present, the prediction effect of simple interpretable models is far inferior to that of complex deep models, and complex models are difficult to obtain and sort out. This has led to the emergence of post-explanation methods for model results, such as partial dependence plots (PDP), local interpretable model-agnostic explanations (LIME), and AHapley additive exPlanations (SHAP). These methods can be applied to the result data of any model and are more flexible.

[0032] Existing post-model explanation methods for explaining the effects of predictors on the target variable vary widely. For example, in LIME, perturbed data points are sampled from a Gaussian distribution, ignoring inter-feature correlations. This can result in unlikely data points being used to learn local explanatory models. Another significant issue is the instability of explanations: in simulations and in practice, the explanations for two very close points can differ significantly. Furthermore, repeated sampling can produce different explanations, making such instability difficult to trust. While this approach holds promise as a local surrogate model for explanations, it remains in its developmental stages for security applications. The recently proposed SHAP method, based on cooperative game theory to explain individual predictions, can largely circumvent unstable explanations. However, it ignores inter-variable dependencies and suffers from slow computational speed, making it impractical for large-scale data applications. In the field of global interpretable methods, Accumulated Local Effects was proposed. It quantifies the effect of the predictor variable on the target variable by calculating the cumulative effect, and can also provide a visualization of the effect of variable interaction on the target variable. However, its effect estimation method is more suitable for variables with standard distributions (such as normal distribution), and will cause deviations in the case of extreme variable distributions. In addition, in large-scale data, non-representative samples may dilute the characteristic effects in the interpretation results, and understanding the results of the ALE first-order and second-order graphs of all variables is aimless and time-consuming.

[0033] However, research and application have shown that these interpretable methods suffer from issues such as explanation stability, variable non-independence, and high computational costs when working with large-scale data. In real-world internet scenarios, interpretable methods are often applied to large datasets to provide insights and conclusions for scenarios and models. Furthermore, the inherent complexity of these scenarios leads to complex and unknown relationships between multiple variables. Consequently, practical applications are susceptible to limitations in explanation stability, the impact of complex variable relationships, and high computational costs.

[0034] To address such problems, this application implements representative sample screening, key explanatory dimension identification, and stable feature effect calculation through sample prototype selection, key feature identification, and cumulative local effect calculation, thereby building a more efficient and stable interpretable system. In this application, the ideas of sample prototype extraction, statistical testing, orthogonalization, sampling method with replacement (bootstrap), and Gaussian mixture distribution fitting can be integrated to refine a set of interpretable methods applicable to multiple scenarios. Overall, it consists of three core modules: "sample selection and filtering-significant feature identification-feature cumulative local effect calculation", which can effectively ensure efficient, accurate, and stable interpretation of data and model results in complex scenarios. Among them, sample selection and filtering can effectively improve calculation time and avoid sample dilution in effect calculation. Significant feature identification can adopt the ideas of orthogonalization and bootstrap to identify features that still make significant contributions to the interpretation of results after excluding the significance brought by luck, focusing on the subsequent interpretation and insight direction. Finally, the calculation of feature cumulative local effects can be achieved by introducing Gaussian mixture distribution to fit the distribution of predictor variables, and the results of distribution fitting are used in the weight of effect calculation to avoid effect calculation bias on extreme distribution variables, thereby ensuring the efficiency and accuracy of interpretable methods under large-scale complex data.

[0035] Example 1

[0036] According to an embodiment of the present invention, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors (the processor may include but is not limited to a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0038] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0042] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the aforementioned computer device (or mobile device).

[0043] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 is a flowchart of a data processing method according to embodiment 1 of the present invention.

[0044] Step S202: Obtain the original data set and target variable.

[0045] The original data set at least includes original features.

[0046] The aforementioned original data set may be a data set used to explain the target variable, and the aforementioned original features may be feature attributes in the original data set.

[0047] The target variable mentioned above may be a variable that needs to be explained, wherein the target variable may be the output result of the model, the target variable may also be a variable existing in the model itself, or the target variable may also be a pre-set variable.

[0048] In an optional embodiment, an original dataset and a target variable can be obtained so that the target variable can be interpreted through the original dataset. In an advertising application scenario, the original dataset can be feature data corresponding to multiple users, where the feature data can be the user's interests and gender, and the target variable can be the probability of pushing an ad to the user. In a financial application scenario, the original dataset can be feature data corresponding to multiple users, where the feature data can be the user's age and gender, and the target variable can be the probability that the user will successfully obtain a loan. In a search application scenario, the original dataset can be categories and application scenarios corresponding to multiple search contents, and the target variable can be the probability of pushing an ad to the target user.

[0049] Step S204: screening the original data set to obtain a target data set.

[0050] The difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

[0051] The target dataset mentioned above may be a prototype dataset, wherein the prototype dataset is a data instance that is representative of the dataset.

[0052] In an optional embodiment, the original dataset may be screened to obtain a target dataset having a data distribution difference with the original dataset that is less than a preset value, wherein the target dataset is a dataset that can represent the original dataset.

[0053] In another optional embodiment, a loss function can be constructed based on the maximum average difference, so as to measure the difference between the data distribution of the original data set and the data set through the loss function. By updating the loss function, a representative target data set can be obtained.

[0054] Step S206: Screen the original features to obtain a target feature set.

[0055] The target feature set is used to represent the features in the original features that are used alone to explain the target variable and conform to the preset statistical distribution.

[0056] In an optional embodiment, in order to eliminate the correlation between the original features, the original features can be screened to obtain features that can be used by individual users to interpret the target variable and conform to the preset statistical distribution, that is, the above-mentioned target feature set.

[0057] In another optional embodiment, the user can select candidate features that can explain the target variable from the original features according to needs, and then orthogonalize the candidate features to obtain the orthogonalized features. The orthogonalized features can be used to perform regression analysis on the target variable. The original data set is sampled by bootstrap and a large number of experiments are performed to obtain a significant empirical distribution caused by chance alone. The regression statistics of the original features on the target variable can be obtained. The regression statistics are compared with the significant empirical distribution to obtain the significance of the candidate features' explanation of the target variable, thereby obtaining the key feature set, that is, the above-mentioned target feature set.

[0058] Step S208: interpret the target variable based on the target data set and the target feature set to obtain an interpretation result.

[0059] Among them, the explanation result is used to indicate the degree of influence of the feature on the target variable.

[0060] In an optional embodiment, because the target dataset is representative, computational failure can be effectively reduced and sample dilution of effect calculations can be avoided. During the screening process of the original features, features that still contribute significantly to the interpretation of the results after excluding the significance caused by luck can be identified. In other words, the aforementioned target feature set can focus subsequent interpretations and insights, avoiding biased interpretations. During the interpretation of the target variable based on the target dataset and target feature set, an effective method for solving the optimization problem with implicit variables (Expectation Maximization, abbreviated as EM) can be used to fit the target dataset and target feature set to obtain a fitting result, which is then incorporated into the estimation of the cumulative local effect to obtain the aforementioned interpretation result.

[0061] In another optional embodiment, the target data set and the target feature set may be fitted using a Gaussian mixture distribution model to obtain a fitting result.

[0062] In yet another optional embodiment, when interpreting the target variable based on the target data set and the target feature set, the calculation of the cumulative effect can be achieved based on the call of the model.

[0063] By the above steps, first, the original data set and the target variable are obtained, wherein the original data set at least includes original features, the original data set is filtered to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value, the original features are filtered to obtain a target feature set, wherein the target feature set and the features used to represent the original features are individually used to explain the target variable, and meet the preset statistical distribution; the target variable is explained based on the target data set and the target feature set to obtain an explanation result, wherein the explanation result is used to represent the influence degree of the features on the target variable, which improves the accuracy of the explanation result. It is easy to note that by filtering the original data set, a target data set that can represent the original data set is obtained to improve the speed of calculation, and by filtering the original features in the original data set, features that can be individually explained to the target variable can be removed to exclude features that still have a significant contribution to the explanation result after excluding the significance caused by chance, so as to focus on the subsequent explanation direction, thereby improving the accuracy of the explanation, and further solving the technical problem of low accuracy of data explanation in the process of algorithm governance in the related art.

[0064] In the above embodiments of the present application, the target variable is explained based on the target data set to obtain an explanation result, including: filtering the target data set based on the target feature set to obtain a target data set; fitting at least one feature of the target data set using a Gaussian mixture model to obtain a fitting result, wherein the fitting result is used to represent the target probability of the at least one feature corresponding to the target value; estimating the at least one feature based on the fitting result and the cumulative local effect estimation formula to obtain an estimation result; explaining the target variable based on the estimation result to obtain the explanation result.

[0065] In an optional embodiment, the target data set can be filtered based on the target feature set to further filter the target data set and obtain a more representative target data set. After obtaining the target data set, at least one feature of the target data set can be fitted using a Gaussian mixture model to obtain a fitting result. Since the accuracy and representativeness of the estimated value are affected by different data distributions, most estimation methods are based on the assumption that the data distribution is normally distributed. A Gaussian mixture distribution model can be used to replace the assumption of normal distribution to fit the original data distribution and obtain the fitting result. Optionally, the fitting of at least one feature of the target data set can be performed by a Gaussian distribution density function.

[0066] In another optional embodiment, the target probability of at least one feature in the fitting result corresponding to the target value can be used as a weight to perform differential weighting on the data points and substituted into the estimation of the cumulative local effect within the observation point field to obtain an estimated result, wherein the estimated result can be the influence trend of the feature on the target variable, and the target variable can be interpreted based on the estimated result to obtain the degree of influence of the feature on the target variable.

[0067] It should be noted that the method of calculating target probabilities is different for different types of target values. For example, the method of calculating target probabilities is different for target values ​​of discrete variables and continuous variables.

[0068] In the above embodiment of the present application, the original data set is screened to obtain a target sample set, including: screening the original data set based on a target distance function to obtain a first data set; constructing a target loss function based on the original data set and the first data set; and updating the first data set based on the target loss function to obtain a target data set.

[0069] The target distance function mentioned above may be maximum mean discrepancy (MMD).

[0070] In an optional embodiment, the original data set can be screened according to the MMD to obtain a first data set, wherein the first data set can be a data set selected from the original data set for the first time, and a target loss function can be constructed based on the difference between the original data set and the first data set, and the first data set can be updated according to the target loss function, so that the data distribution of the obtained target data set gradually approaches the data distribution of the original data set, so that the target data set can become a representative data set of the original data set, so that when the target variable is subsequently explained by the target data set, the efficiency of the explanation can be improved.

[0071] In the above embodiment of the present application, the original features are screened to obtain a target feature set, including: orthogonalizing the original features to obtain a target residual set, wherein the target residual set includes at least one residual, and the at least one residual is used to represent the features of the original features that cannot explain the target variable; determining a target empirical distribution corresponding to the target residual set, wherein the target empirical distribution is used to represent the distribution relationship between the target variable and the at least one residual; determining a target test statistic corresponding to the original features; and determining the target feature set based on the target empirical distribution and the target test statistic.

[0072] The target test statistic mentioned above may be a test statistic obtained by sampling the original features with replacement.

[0073] In an optional embodiment, the user can screen the original features according to needs to obtain candidate features, and the candidate features can be orthogonalized to construct the null hypothesis, wherein the null hypothesis can be that the residuals after the features are orthogonalized are difficult to explain the target variable, that is, to obtain the above-mentioned target residual set, and the original data set can be sampled by bootstrap with replacement to obtain sampled data, and the orthogonalized features can be used to perform regression analysis on the target variable to obtain the maximum test statistic, so as to determine the target empirical distribution corresponding to the target residual set based on the maximum test statistic, and the sampled data can be used to conduct a large number of experiments to obtain the target test statistic, and the significant empirical distribution can be obtained based on the target empirical distribution and the target test statistic, so as to obtain the above-mentioned target feature set based on the significant empirical distribution.

[0074] like Figure 3 The figure shows the empirical distribution diagram of the target variable, where the horizontal axis X is the target variable and the vertical axis is the cumulative probability, which can represent the target empirical distribution of the target variable.

[0075] In the above embodiment of the present application, determining the target empirical distribution corresponding to the target residual set includes: sampling the target residual set to obtain at least one data set; determining the maximum test statistic corresponding to the at least one data set; and performing distribution fitting on the maximum test statistic corresponding to the at least one data set to obtain the target empirical distribution.

[0076] In an optional embodiment, the target residual set can be sampled with replacement to obtain at least one data set, and the maximum test statistic corresponding to the at least one data set can be determined. The maximum test statistic corresponding to the at least one data set can be fitted with a Gaussian distribution model to obtain the above-mentioned target empirical distribution, so as to obtain the influence trend of the feature on the target variable.

[0077] In the above embodiment of the present application, the target feature set is determined based on the target empirical distribution and the target test statistic, including: obtaining a confidence threshold corresponding to the target empirical distribution; comparing the target test statistic and the confidence threshold to obtain a comparison result, wherein the comparison result is used to indicate whether the confidence corresponding to the original feature in the target test statistic is within the confidence threshold; and determining the target feature set based on the comparison result and the original feature.

[0078] The above confidence threshold is the rejection region of the target empirical distribution at a given significance level, wherein the comparison result is also used to determine whether the target statistic falls into the rejection region.

[0079] In an optional embodiment, a confidence threshold corresponding to the target empirical distribution can be obtained. By comparing the target test statistic with the confidence threshold, if the confidence corresponding to the original feature in the target test statistic is greater than the confidence threshold, the original feature can be determined to be a key feature, wherein the key feature is a feature that can independently explain the target variable, and the above-mentioned target feature set can be obtained based on multiple key features. If the confidence corresponding to the original feature is less than the confidence threshold, it means that the original feature needs to rely on other features to explain the target variable, it is not a key feature, and this feature does not need to be added to the original feature set.

[0080] In the above embodiment of the present application, determining the maximum test statistic corresponding to at least one data set includes: performing regression analysis on the at least one data set to obtain multiple initial test statistics corresponding to the at least one data set; sorting the multiple initial test statistics to obtain a sorting result; and determining the maximum test statistic corresponding to the at least one data set based on the sorting result, wherein the maximum test statistic is used to represent the largest test statistic among the multiple initial test statistics.

[0081] In an optional embodiment, regression analysis can be performed on at least one data set to obtain multiple initial test statistics corresponding to the at least one data set. After obtaining the multiple initial test statistics, the multiple initial test statistics can be sorted to obtain a sorting result, so as to determine the maximum test statistic from the multiple initial test statistics based on the sorting result.

[0082] In another optional embodiment, multiple initial test statistics can be obtained by regressing each residual variable of at least one data set on the target variable.

[0083] In the above embodiment of the present application, the method further includes: outputting the interpretation result according to a preset display mode; receiving an adjustment instruction, and adjusting the interpretation result based on the adjustment instruction to obtain an adjustment result.

[0084] The above-mentioned preset display method may be to connect the explanation results of each feature on the target variable.

[0085] In an optional embodiment, the influence degree of each feature can be determined by a point set, and the point set can be connected and visualized. Optionally, the connected curve can be displayed on the client. If the user needs to adjust the curve, the adjustment result can be fed back in the form of an adjustment instruction, so that the interpretation result can be adjusted according to the adjustment instruction to obtain the above adjustment result. Figure 4 The figure shows a schematic diagram of a visual display, in which the horizontal axis represents the feature, the vertical axis represents the degree of influence of the feature on the target variable, and the set of connected points can be the degree of influence of the feature on the target variable.

[0086] The following combination Figure 5 An embodiment of the present application is described in detail. Figure 5 As shown, the method may include the following steps:

[0087] Step S501, constructing a target loss function based on the maximum average difference;

[0088] Step S502, updating the first data set based on the target loss function to obtain a target data set;

[0089] Step S503, performing orthogonalization processing on the original features to obtain a target residual set;

[0090] Step S504: sampling the target residual set to obtain at least one data set;

[0091] Step S505 , performing distribution fitting on the maximum test statistic of at least one data set to obtain a target empirical distribution;

[0092] Step S506, performing feature significance judgment based on the target empirical distribution and the target test statistic corresponding to the original feature to obtain a target feature set;

[0093] Step S507, fitting the target feature set and the target data set based on the Gaussian mixture distribution to obtain a fitting result;

[0094] Step S508: estimating at least one feature according to the fitting result and the cumulative local effect formula to obtain an estimation result, and interpreting the target variable based on the estimation result to obtain an interpretation result;

[0095] The above fitting results are used to represent the target probability of at least one feature corresponding to the target value.

[0096] Step S509: Visualize the interpretation result according to a preset display method.

[0097] In this application, a combination of Gaussian mixture model and cumulative local effect estimation is proposed, which can effectively avoid the deviation caused by extreme distribution of data in real scenarios, thereby achieving the accuracy and universality of effect estimation. In this application, a new feature selection perspective is proposed, which introduces the ideas and methods of orthogonalization and bootstrap, and integrates statistical tests to extract features that are still significant after removing accidental factors as key explanatory dimensions. In this application, by screening representative samples, identifying key explanatory dimensions and calculating stable feature effects, it can not only help explain the focus of data sets and explanatory dimensions, but also greatly reduce computing overhead and improve processing efficiency, and also build a more efficient and stable interpretable system.

[0098] In summary, the interpretable method and system based on the cumulative local effects of variables under large-scale data proposed in this solution are suitable for e-commerce platforms, media platforms, financial platforms, etc., and can effectively ensure the provision of more accurate, efficient and stable model interpretation in large-scale data situations.

[0099] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0100] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0101] Example 2

[0102] According to an embodiment of the present application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] Figure 6 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 7 As shown, the method may include the following steps:

[0104] Step S602: displaying the original data set and the target variable on the interactive interface, wherein the original data set includes at least the original features;

[0105] In step S604, when a target touch operation is sensed in the interactive interface, an explanation result corresponding to the target variable is displayed in the interactive interface, wherein the explanation result is generated based on the target data set and the target feature set to interpret the target variable, and the explanation result is used to indicate the degree of influence of the feature on the target variable. The target feature set is obtained by screening the original features, and the target feature set is used to indicate the features of the original features that are used alone to interpret the target variable and conform to the preset statistical distribution. The target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

[0106] The target touch operation mentioned above may be an operation in which the user touches the interactive interface, wherein the target touch operation may be used to interpret the target variable, obtain the interpretation result, and display the interpretation result in the interactive interface for the user to view.

[0107] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0108] Example 3

[0109] According to an embodiment of the present invention, a data processing device for implementing the above data processing method is also provided. Figure 7 As shown, the device includes: an acquisition module 702 , a first screening module 704 , a second screening module 706 , and an interpretation module 708 .

[0110] Among them, the acquisition module is used to obtain the original data set and the target variable, wherein the original data set includes at least the original features; the first screening module is used to screen the original data set to obtain the target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; the second screening module is used to screen the original features to obtain the target feature set, wherein the target feature set is used to represent the features in the original features that are used alone to explain the target variable and conform to the preset statistical distribution; the interpretation module is used to interpret the target variable based on the target data set and the target feature set to obtain the interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

[0111] It should be noted here that the acquisition module 702, the first screening module 704, the second screening module 706, and the interpretation module 708 correspond to steps S202 to S208 in Example 1. The instances and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules can be run in the computer terminal 10 provided in Example 1 as part of the tool.

[0112] In the above embodiments of the present application, the interpretation module includes: a fitting unit, an estimation unit, and an interpretation unit.

[0113] Among them, the fitting unit is used to fit at least one feature of the target data set using a Gaussian mixture model to obtain a fitting result, wherein the fitting result is used to represent the target probability of at least one feature corresponding to the target value; the estimation unit is used to estimate at least one feature based on the fitting result and the cumulative local effect estimation formula to obtain an estimation result; the interpretation unit is used to interpret the target variable based on the estimation result to obtain an interpretation result.

[0114] In the above embodiment of the present application, the first screening module includes: a first screening unit, a construction unit, and an updating unit.

[0115] Among them, the first screening unit is used to screen the original data set based on the target distance function to obtain the first data set; the construction unit is used to construct the target loss function based on the original data set and the first data set; the updating unit is used to update the first data set based on the target loss function to obtain the target data set.

[0116] In the above embodiment of the present application, the second screening unit includes: an orthogonalization processing unit and a determination unit.

[0117] Among them, the orthogonalization processing unit is used to perform orthogonalization processing on the original features to obtain a target residual set, wherein the target residual set includes at least one residual, and the at least one residual is used to represent the features of the original features that cannot explain the target variable; the determination unit is used to determine the target empirical distribution corresponding to the target residual set, wherein the target empirical distribution is used to represent the distribution relationship between the target variable and the at least one residual; the determination unit is also used to determine the target test statistic corresponding to the original features; the determination unit is also used to determine the target feature set based on the target empirical distribution and the target test statistic.

[0118] In the above embodiment of the present application, the determination unit includes: a sampling subunit, a determination subunit, and a fitting subunit.

[0119] Among them, the sampling subunit is used to sample the target residual set to obtain at least one data set; the determination subunit is used to determine the maximum test statistic corresponding to at least one data set; the fitting subunit is used to perform distribution fitting on the maximum test statistic corresponding to at least one data set to obtain the target empirical distribution.

[0120] In the above embodiment of the present application, the determination unit includes: an acquisition subunit and a comparison subunit.

[0121] Among them, the acquisition subunit is used to obtain the confidence threshold corresponding to the target empirical distribution; the comparison subunit is used to compare the target test statistic and the confidence threshold to obtain a comparison result, wherein the comparison result is used to indicate whether the confidence corresponding to the original feature in the target test statistic is within the confidence threshold; the determination subunit is also used to determine the target feature set based on the comparison result and the original feature.

[0122] In the above embodiment of the present application, the determination subunit is further used to perform regression analysis on at least one data set to obtain multiple initial test statistics corresponding to the at least one data set; the determination subunit is further used to sort the multiple initial test statistics to obtain a sorting result; the determination subunit is further used to determine the maximum test statistic corresponding to the at least one data set based on the sorting result, wherein the maximum test statistic is used to represent the largest test statistic among the multiple initial test statistics.

[0123] In the above embodiments of the present application, the device further includes: an output module and a receiving module.

[0124] Among them, the output module is used to output the interpretation result according to a preset display mode; the receiving module is used to receive an adjustment instruction, and adjust the interpretation result based on the adjustment instruction to obtain an adjustment result.

[0125] It should be noted that the preferred implementation scheme involved in the above embodiments of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0126] Example 4

[0127] According to an embodiment of the present invention, a data processing device for implementing the above data processing method is also provided. Figure 8 As shown, the device includes: a display module 802 and a sensing module 804.

[0128] A display module 802 is configured to display the original data set and the target variable on an interactive interface, wherein the original data set includes at least the original features;

[0129] The sensing module 804 is used to display the interpretation result corresponding to the target variable in the interactive interface when a target touch operation is sensed in the interactive interface, wherein the interpretation result is generated based on the target data set and the target feature set to interpret the target variable, and the interpretation result is used to indicate the degree of influence of the feature on the target variable. The target feature set is obtained by screening the original features, and the target feature set is used to indicate the features of the original features that are used alone to interpret the target variable and conform to the preset statistical distribution. The target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

[0130] It should be noted that the display module 802 and the sensing module 804 described above correspond to steps S702 to S704 in Example 2. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the above modules, as part of the tool, can be run in the computer terminal 10 provided in Example 1.

[0131] It should be noted that the preferred implementation scheme involved in the above-mentioned Example 1 of this application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0132] Example 5

[0133] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0134] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0135] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: obtaining an original data set and a target variable, wherein the original data set includes at least original features; screening the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; screening the original features to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to the preset statistical distribution; interpreting the target variable based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to indicate the degree of influence of the feature on the target variable.

[0136] Optionally, Figure 9 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 9 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors and a memory.

[0137] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned data processing method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain an original data set and a target variable, wherein the original data set includes at least original features; screen the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; screen the original features to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to the preset statistical distribution; interpret the target variable based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

[0139] Optionally, the processor may also execute the program code of the following steps: fitting at least one feature of the target data set using a Gaussian mixture model to obtain a fitting result, wherein the fitting result is used to represent the target probability of the at least one feature corresponding to the target value; estimating at least one feature based on the fitting result and the cumulative local effect estimation formula to obtain an estimation result; and interpreting the target variable based on the estimation result to obtain an interpretation result.

[0140] Optionally, the processor may also execute the program code of the following steps: screening the original data set based on the target distance function to obtain a first data set; constructing a target loss function based on the original data set and the first data set; and updating the first data set based on the target loss function to obtain a target data set.

[0141] Optionally, the processor may also execute the program code of the following steps: performing orthogonalization on the original features to obtain a target residual set, wherein the target residual set includes at least one residual, and the at least one residual is used to represent the features of the original features that cannot explain the target variable; determining a target empirical distribution corresponding to the target residual set, wherein the target empirical distribution is used to represent the distribution relationship between the target variable and the at least one residual; determining a target test statistic corresponding to the original features; and determining a target feature set based on the target empirical distribution and the target test statistic.

[0142] Optionally, the processor may also execute program code for the following steps: sampling the target residual set to obtain at least one data set; determining the maximum test statistic corresponding to at least one data set; and performing distribution fitting on the maximum test statistic corresponding to at least one data set to obtain a target empirical distribution.

[0143] Optionally, the processor may also execute the program code of the following steps: obtaining a confidence threshold corresponding to the target empirical distribution; comparing the target test statistic and the confidence threshold to obtain a comparison result, wherein the comparison result is used to indicate whether the confidence corresponding to the original feature in the target test statistic is within the confidence threshold; and determining the target feature set based on the comparison result and the original feature.

[0144] Optionally, the processor may also execute program code for the following steps: performing regression analysis on at least one data set to obtain multiple initial test statistics corresponding to the at least one data set; sorting the multiple initial test statistics to obtain a sorting result; and determining a maximum test statistic corresponding to the at least one data set based on the sorting result, wherein the maximum test statistic is used to represent the largest test statistic among the multiple initial test statistics.

[0145] Optionally, the processor may further execute program codes of the following steps: outputting the interpretation result in a preset display mode; receiving an adjustment instruction, and adjusting the interpretation result based on the adjustment instruction to obtain an adjustment result.

[0146] According to an embodiment of the present invention, an original data set and a target variable are obtained, wherein the original data set includes at least original features; the original data set is screened to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; the original features are screened to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to a preset statistical distribution; the target variable is interpreted based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable, thereby solving the technical problem of low accuracy in interpreting data in the process of algorithm governance in related technologies.

[0147] It can be understood by those skilled in the art that Figure 9 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 9 It does not limit the structure of the above electronic device. For example, the computer terminal 10 may also include Figure 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 9 Different configurations shown.

[0148] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0149] Example 6

[0150] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0151] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0152] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an original data set and a target variable, wherein the original data set includes at least original features; screening the original data set to obtain a target data set, wherein the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value; screening the original features to obtain a target feature set, wherein the target feature set is used to represent the features of the original features that are used alone to explain the target variable and conform to a preset statistical distribution; interpreting the target variable based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

[0153] Optionally, the storage medium is further configured to store program code for executing the following steps: fitting at least one feature of the target data set using a Gaussian mixture model to obtain a fitting result, wherein the fitting result is used to represent the target probability of the at least one feature corresponding to the target value; estimating at least one feature based on the fitting result and the cumulative local effect estimation formula to obtain an estimation result; and interpreting the target variable based on the estimation result to obtain an interpretation result.

[0154] Optionally, the above-mentioned storage medium is also configured to store program code for performing the following steps: screening the original data set based on the target distance function to obtain a first data set; constructing a target loss function based on the original data set and the first data set; updating the first data set based on the target loss function to obtain a target data set.

[0155] Optionally, the storage medium is further configured to store program code for executing the following steps: performing orthogonalization processing on the original features to obtain a target residual set, wherein the target residual set includes at least one residual, and the at least one residual is used to represent the features of the original features that cannot explain the target variable; determining a target empirical distribution corresponding to the target residual set, wherein the target empirical distribution is used to represent the distribution relationship between the target variable and the at least one residual; determining a target test statistic corresponding to the original features; and determining a target feature set based on the target empirical distribution and the target test statistic.

[0156] Optionally, the storage medium is further configured to store program code for executing the following steps: sampling a target residual set to obtain at least one data set; determining a maximum test statistic corresponding to at least one data set; and performing distribution fitting on the maximum test statistic corresponding to at least one data set to obtain a target empirical distribution.

[0157] Optionally, the above-mentioned storage medium is also configured to store program code for executing the following steps: obtaining a confidence threshold corresponding to the target empirical distribution; comparing the target test statistic and the confidence threshold to obtain a comparison result, wherein the comparison result is used to indicate whether the confidence corresponding to the original feature in the target test statistic is within the confidence threshold; determining the target feature set based on the comparison result and the original feature.

[0158] Optionally, the storage medium is further configured to store program code for executing the following steps: performing regression analysis on at least one data set to obtain multiple initial test statistics corresponding to the at least one data set; sorting the multiple initial test statistics to obtain a sorting result; and determining a maximum test statistic corresponding to the at least one data set based on the sorting result, wherein the maximum test statistic is used to represent the largest test statistic among the multiple initial test statistics.

[0159] Optionally, the storage medium is further configured to store program code for performing the following steps: outputting the explanation result according to a preset display mode; receiving an adjustment instruction, and adjusting the explanation result based on the adjustment instruction to obtain an adjusted result.

[0160] By adopting the embodiment of the present application, the original data set and the target variable are acquired, wherein the original data set at least includes original features; the original data set is filtered to obtain a target data set, wherein a difference between a data distribution of the target data set and a data distribution of the original data set is less than a preset value; the original features are filtered to obtain a target feature set, wherein the target feature set is used to represent features in the original features that are used to explain the target variable alone and that conform to a preset statistical distribution; and the target variable is explained based on the target data set and the target feature set to obtain an explanation result, wherein the explanation result is used to represent an influence degree of the features on the target variable, thereby solving the technical problem of low accuracy of data explanation in the process of algorithm governance in the related art.

[0161] The serial numbers of the above embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0162] In the above embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0163] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0164] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be integrated into multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0165] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0166] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0167] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining an original data set and a target variable, wherein the original data set includes at least original features, the original features being feature data of users in an application scenario, and the target variable being a probability of pushing a scenario object corresponding to the application scenario to the user; Performing screening processing on the original data set to obtain a target data set, wherein a difference between a data distribution of the target data set and a data distribution of the original data set is less than a preset value; Screening the original features to obtain a target feature set, wherein the target feature set is used to represent features of the original features that are solely used to explain the target variable and conform to a preset statistical distribution; The target variable is interpreted based on the target data set and the target feature set to obtain an interpretation result, wherein the interpretation result is used to represent the degree of influence of the feature on the target variable.

2. The method according to claim 1, characterized in that The target variable is interpreted based on the target data set to obtain an interpretation result, including: Fitting at least one feature of the target data set using a Gaussian mixture model to obtain a fitting result, wherein the fitting result is used to represent a target probability of the at least one feature corresponding to a target value; estimating the at least one feature based on the fitting result and a cumulative local effect estimation formula to obtain an estimation result; The target variable is interpreted based on the estimation result to obtain the interpretation result.

3. The method according to claim 1, characterized in that The original data set is screened to obtain a target data set, including: Performing screening processing on the original data set based on the target distance function to obtain a first data set; Constructing a target loss function based on the original data set and the first data set; The first data set is updated based on the target loss function to obtain the target data set.

4. The method according to claim 1, wherein The original features are screened to obtain a target feature set, including: Orthogonalizing the original features to obtain a target residual set, wherein the target residual set includes at least one residual, and the at least one residual is used to represent a feature of the original features that cannot explain the target variable; Determining a target empirical distribution corresponding to the target residual set, wherein the target empirical distribution is used to represent a distribution relationship between the target variable and the at least one residual; Determining a target test statistic corresponding to the original feature; The target feature set is determined based on the target empirical distribution and the target test statistic.

5. The method according to claim 4, characterized in that Determining a target empirical distribution corresponding to the target residual set includes: Sampling the target residual set to obtain at least one data set; determining a maximum test statistic corresponding to the at least one data set; Distribution fitting is performed on the maximum test statistic corresponding to the at least one data set to obtain the target empirical distribution.

6. The method according to claim 4, characterized in that Determining a target feature set based on the target empirical distribution and the target test statistic includes: Obtaining a confidence threshold corresponding to the target empirical distribution; Comparing the target test statistic with the confidence threshold to obtain a comparison result, wherein the comparison result is used to indicate whether the confidence corresponding to the original feature in the target test statistic is within the confidence threshold; The target feature set is determined based on the comparison result and the original features.

7. The method according to claim 5, characterized in that Determining a maximum test statistic corresponding to the at least one data set includes: performing regression analysis on the at least one data set to obtain a plurality of initial test statistics corresponding to the at least one data set; sorting the multiple initial test statistics to obtain a sorting result; A maximum test statistic corresponding to the at least one data set is determined based on the sorting result, wherein the maximum test statistic is used to represent the largest test statistic among the multiple initial test statistics.

8. The method according to claim 1, characterized in that The method further comprises: Outputting the interpretation result in a preset display mode; An adjustment instruction is received, and the interpretation result is adjusted based on the adjustment instruction to obtain an adjustment result.

9. A data processing method, characterized in that: include: Displaying an original data set and a target variable on an interactive interface, wherein the original data set includes at least original features, the original features being feature data of a user in an application scenario, and the target variable being a probability of pushing a scenario object corresponding to the application scenario to the user; When a target touch operation is sensed in the interactive interface, an interpretation result corresponding to the target variable is displayed in the interactive interface, wherein the interpretation result is generated by interpreting the target variable based on a target data set and a target feature set, and the interpretation result is used to indicate the degree of influence of the feature on the target variable. The target feature set is obtained by screening the original features, and the target feature set is used to indicate the features of the original features that are used alone to interpret the target variable and conform to a preset statistical distribution. The target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

10. A data processing device, characterized in that: include: An acquisition module is configured to acquire an original data set and a target variable, wherein the original data set includes at least original features, the original features being feature data of a user in an application scenario, and the target variable being a probability of pushing a scenario object corresponding to the application scenario to the user; a first screening module, configured to screen the original data set to obtain a target data set, wherein a difference between a data distribution of the target data set and a data distribution of the original data set is less than a preset value; A second screening module is configured to screen the original features to obtain a target feature set, wherein the target feature set is used to represent features of the original features that are solely used to explain the target variable and conform to a preset statistical distribution; The explanation module is used to explain the target variable based on the target data set and the target feature set to obtain an explanation result, wherein the explanation result is used to represent the degree of influence of the feature on the target variable.

11. A data processing device, characterized in that: include: A display module is configured to display an original data set and a target variable on an interactive interface, wherein the original data set includes at least original features, which are characteristic data of users in an application scenario, and the target variable is the probability of pushing a scenario object corresponding to the application scenario to the user; A sensing module is used to display the interpretation result corresponding to the target variable in the interactive interface when a target touch operation is sensed in the interactive interface, wherein the interpretation result is generated by interpreting the target variable based on a target data set and a target feature set, and the interpretation result is used to indicate the degree of influence of the feature on the target variable, the target feature set is obtained by screening the original features, and the target feature set is used to indicate the features of the original features that are used alone to interpret the target variable and conform to a preset statistical distribution, the target data set is obtained by screening the original data set, and the difference between the data distribution of the target data set and the data distribution of the original data set is less than a preset value.

12. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 9.

13. A computer terminal, characterized in that: include: A processor and a memory, wherein the processor is used to run a program stored in the memory, wherein the program executes the data processing method according to any one of claims 1 to 9 when running.

Images