Data evaluation method and device

Abstract

Embodiments of the present application provide a data evaluation method and device. The method comprises: obtaining external data to be evaluated; performing data cleaning and standardization on the external data, and performing data dimension reduction on the standardized external data to obtain reduced external data; inputting the reduced external data into a naive Bayes model, and determining the quality of the external data to be evaluated according to the naive Bayes model; wherein the naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples comprise reduced external data obtained by performing data dimension reduction on standardized external data used for model training. Thus, the quality of the external data can be accurately analyzed, and reliable reference can be provided for decision-making.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data evaluation method and apparatus. Background Technology

[0002] Currently, banks typically use user transaction-related data, known as internal data. In contrast, there is external data, which refers to customer data that cannot be obtained during the banking process, such as customer educational background, location, and registered mobile phone number. Banks generally use external data by directly accessing external data sources for decision-making, which inherently involves some subjectivity. External data, as a supplement to customer characteristic data, must have a strong correlation with a particular business activity to be considered. However, there is currently no process for determining the degree of relevance of external data to business operations; for example, the lack of registered mobile phone number registration may be correlated with customer risk. Furthermore, there are currently no clear evaluation standards or methods for assessing the quality of external data.

[0003] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0004] Currently, external data plays an increasingly important role in banking operations. However, when banks introduce external data, it is difficult to define and evaluate its quality. If the quality of external data is poor, it may lead to incorrect decisions and increase business risks.

[0005] To address at least one of the aforementioned problems, embodiments of this application provide a data evaluation method and apparatus. This enables accurate analysis of the quality of external data and provides a reliable reference for decision-making.

[0006] According to one aspect of the embodiments of this application, a data evaluation method is provided, the method comprising:

[0007] Obtain the external data to be evaluated;

[0008] The external data is cleaned and standardized, and then the standardized external data is dimensionality reduced to obtain dimensionality-reduced external data.

[0009] The dimensionality-reduced external data is input into the Naive Bayes model, and the quality of the external data to be evaluated is determined based on the Naive Bayes model; wherein, the Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by performing data dimensionality reduction on the standardized external data used for model training.

[0010] In some implementations, the external data is cleaned and standardized, including: discarding data with an effective data rate lower than a first threshold to obtain cleaned external data; and standardizing the cleaned external data using standard deviation.

[0011] In some implementations, the dimensionality reduction of the standardized external data to obtain dimensionality-reduced external data includes: sampling the external data to obtain a signal, and performing independent component analysis on the signal; wherein the number of iterations or the iteration target of the independent component analysis is specified by gradient descent optimization.

[0012] In some embodiments, the method further includes: acquiring the training samples; training a preset machine learning model using the training samples to obtain the Naive Bayes model; wherein obtaining the Naive Bayes model includes: determining the probability distribution model of the Naive Bayes model based on the sample attributes of the dimensionality-reduced external data in the training samples.

[0013] In some implementations, determining the probability distribution model of the Naive Bayes model based on the sample attributes of the dimensionality-reduced external data includes: calculating the probability distribution in the discrete space when the sample attributes are discrete; and estimating the probability model parameters using maximum likelihood estimation when the sample attributes are continuous.

[0014] In some implementations, determining the quality of the external data to be evaluated based on the Naive Bayes model includes: determining the quality of the external data based on the score value of the external data to be evaluated in the Naive Bayes model, wherein the quality of the external data is proportional to the score value.

[0015] In some implementations, the score of the external data to be evaluated in the Naive Bayes model includes: the maximum value of n probability values ​​of the external data in the discrete target space of the Naive Bayes model is the score of the data.

[0016] According to one aspect of the embodiments of this application, a data evaluation apparatus is provided, the apparatus comprising:

[0017] The acquisition unit acquires the external data to be evaluated;

[0018] The data processing unit performs data cleaning and standardization on the external data, and performs dimensionality reduction on the standardized external data to obtain dimensionality-reduced external data.

[0019] The evaluation unit inputs the dimensionality-reduced external data into the Naive Bayes model and determines the quality of the external data to be evaluated based on the Naive Bayes model. The Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include dimensionality-reduced external data obtained by reducing the dimensionality of the standardized external data used for model training.

[0020] According to one aspect of the embodiments of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned data evaluation method.

[0021] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing an execution data evaluation method.

[0022] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, implements the aforementioned data evaluation method.

[0023] One of the beneficial effects of the embodiments of this application is that it can accurately analyze the quality of external data and provide reliable reference for decision-making.

[0024] Specific embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of this application can be adopted. It should be understood that the embodiments of this application are not limited in scope. Within the spirit and scope of the appended claims, embodiments of this application include many changes, modifications, and equivalents.

[0025] The feature information described and illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with feature information in other embodiments, or substituted for feature information in other embodiments.

[0026] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, whole, step, or component, but does not exclude the presence or addition of one or more other features, wholes, steps, or components. Attached Figure Description

[0027] Many aspects of this application can be better understood by referring to the following accompanying drawings. The components in the drawings are not drawn to scale, but are only intended to illustrate the principles of this application. Corresponding portions in the drawings may be enlarged or reduced for ease of illustration and description of certain parts of this application. Elements and features described in one drawing or embodiment of this application may be combined with elements and features shown in one or more other drawings or embodiments. Furthermore, similar reference numerals in the drawings denote corresponding components in several drawings and can be used to indicate corresponding components used in more than one embodiment.

[0028] In the attached diagram:

[0029] Figure 1 This is a configuration diagram of a data evaluation device according to an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of a Naive Bayes model according to an embodiment of this application;

[0031] Figure 3 This is a flowchart of the data evaluation process according to an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of a data evaluation method according to an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of computer device 500 in an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0039] It should be noted that the data evaluation method and apparatus of the present invention can be used in the financial field, or in any field other than the financial field. The application field of the data evaluation method and apparatus of the present invention is not limited.

[0040] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of this application may be employed. It should be understood that this application is not limited to the described embodiments; rather, it includes all modifications, variations, and equivalents falling within the scope of the appended claims.

[0041] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in association and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0042] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0043] To address the aforementioned problems, various embodiments of this application will be described below with reference to the accompanying drawings. These embodiments are merely exemplary and are not intended to limit the scope of this application.

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0045] This application provides a data evaluation device.

[0046] Figure 1 This is a configuration diagram of a data evaluation apparatus according to an embodiment of this application. Figure 1 As shown, the data evaluation device 100 includes:

[0047] Acquisition unit 110 acquires the external data to be evaluated;

[0048] The data processing unit 120 performs data cleaning and standardization on the external data, and performs data dimensionality reduction on the standardized external data to obtain dimensionality-reduced external data.

[0049] Evaluation unit 130 inputs the dimensionality-reduced external data into a Naive Bayes model and determines the quality of the external data to be evaluated based on the Naive Bayes model; wherein the Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by dimensionality reduction of the standardized external data used for model training.

[0050] This enables accurate analysis of the quality of external data and provides reliable references for decision-making.

[0051] In some implementations, the data processing unit 120 performs data cleaning and standardization on the external data by: discarding data with an effective data rate lower than a first threshold to obtain the cleaned external data; and standardizing the cleaned external data using the standard deviation.

[0052] For example, high-quality data is the first step in building an accurate model. In order to provide higher-quality data for model training, the original external data needs to be cleaned and filtered. Data cleaning involves discarding data that does not meet the criteria, excluding it from model training, and directly classifying it as low-quality data. Therefore, the conditions for data cleaning are definite and objective.

[0053] In some implementations, data cleaning involves discarding data with an effective data rate below a first threshold and performing subsequent standardization on data with an effective data rate greater than or equal to the first threshold. For example, the effective data rate refers to the ratio of the number of indicators m with effective values ​​for a given external data to the total number of indicators p; for example, the first threshold can be 60%. This application does not limit the specific values ​​of the number of indicators with effective values, the total number of indicators, or the first threshold, and those skilled in the art can refer to the prior art.

[0054] Therefore, cleaned data can improve the performance of the model.

[0055] For example, to accelerate model training and improve model performance, data standardization is necessary after data cleaning. Data standardization transforms data to make the values ​​more standardized and reasonable. For instance, if the upper and lower bounds of features differ significantly, but their influence is unrelated to these bounds, standardized data can be obtained by calculating the relative values ​​of the data within a given interval, thus accelerating model convergence. This method uses standard deviation standardization, and the specific calculation formula is as follows:

[0056]

[0057] in, Here, x represents the data after standardization, and 'x' represents the data before standardization. S is the mean of the data, and S is the standard deviation of the data. The mean and standard deviation mentioned above are based on the training data samples. In subsequent use of the model, the mean and standard deviation will not change with the introduction of evaluation data.

[0058] Therefore, standardized data can further accelerate model training and improve model performance.

[0059] In some implementations, dimensionality reduction is performed on the standardized external data to obtain the dimensionality-reduced data, including: sampling the external data to obtain a signal, and performing independent component analysis on the signal; wherein the number of iterations or the iteration target of the independent component analysis is specified by gradient descent optimization.

[0060] For example, the target space is the feature space of the business objective. If the business objective is customer credit evaluation, then the target space is the credit score, and the result data is a one-dimensional data space. For example, data can be divided into discrete data and continuous data, corresponding to discrete space and continuous space. When the business objective data is continuous, the objective data resides in the continuous data space, called the continuous target space; when the business data is discrete, the objective data resides in the discrete data space, called the discrete target space. When the discrete target space is not infinitely large, it is called the finite discrete target space.

[0061] For example, the Naive Bayes model is a machine learning model based on Bayes' theorem and the assumption of feature conditional independence. Figure 2 This is a schematic diagram of a Naive Bayes model according to an embodiment of this application. Figure 2 As shown, for example, the Naive Bayes algorithm assumes that the attributes of the dataset are independent of each other. For instance, the dataset has attribute 1, attribute 2, attribute w, etc. Therefore, the algorithm's logic is very simple, and it is relatively stable. When the data exhibits different characteristics, the classification performance of Naive Bayes does not vary significantly. In other words, the Naive Bayes algorithm is robust and does not show significant differences for different types of datasets. When the relationships between the attributes of the dataset are relatively independent, the Naive Bayes classification algorithm will perform well.

[0062] For example, the Naive Bayes model requires that features be independent of each other. However, source data from external banking data sources are usually not independent. Therefore, dimensionality reduction is needed to map the n external data features into an x-dimensional independent vector space. For example, Independent Component Analysis (ICA) can be used for dimensionality reduction.

[0063] ICA performs independent component analysis based on the sampled signal. For the signal X = {X1, X2, X3, ..., X...} n} T X is a vector composed of random variables. i It is a random variable, and is the vector of results obtained by sampling a certain random variable multiple times. Assume there exist mutually independent random variables S. i The vector S = {S1, S2, S3, ..., S} is composed of... n} T Since each random variable in vector S is independent, it can be determined through the random variable S. i probability density function Obtain the probability density function of S.

[0064]

[0065] P S (S) is the probability density function of S, also known as the joint probability density function.

[0066] The probability distribution function of the combined vectors S and X can be expressed as F S (S), F X Since each random variable in the combination vector X is not independent, the probability density function of X can be obtained by differentiating the probability distribution, i.e.

[0067] P X(X)=F′ X (X)

[0068] =(P(X<x))′

[0069] Expanding the standard ICA formula S = AX, since A is just a coefficient matrix and has no practical meaning, for ease of understanding, A is represented by W, resulting in...

[0070] S = WX

[0071]

[0072] The formula (P(X<x))′ can be further transformed to obtain

[0073]

[0074] As can be seen, W in the above formula is a coefficient matrix, and the probability density function should be a scalar value. Therefore, it is necessary to take the determinant of W in the internal probability density expression and then differentiate it:

[0075]

[0076] According to the rules of determinant operations, we can further obtain:

[0077]

[0078] In the above formula, the derivative of the probability distribution function with respect to S can be expressed by the probability density function, i.e., P X (X)=||P S (Wx)W||

[0079] Because the probability distribution function involves differentiation with respect to x, the probability density function of S in the above equation needs to be multiplied by the coefficient matrix. In the above equation, only the coefficient matrix is ​​expressed as a vector, resulting in:

[0080] P X (X)=P S (Wx)||W||

[0081] Combined again:

[0082]

[0083] We can obtain:

[0084]

[0085] For a series of samples of X, the probability of that sample occurring can be obtained:

[0086]

[0087] In the above formula, m represents the number of samples, and d j This means X is R. n×m The j-th column of the matrix. It can be seen that ||W|| is an unknown in the above equation, so the probability L is also unknown. The optimal solution is to estimate W so that the probability L reaches its maximum value. There are several ways to find the optimal solution, such as Newton's iteration method to maximize negative entropy, or using gradient descent to approximate the optimal solution. Here, we use gradient descent, so we need to calculate the gradient of probability L with respect to ||W||. For ease of calculation, we first convert the product to addition, and take the natural logarithm of both sides of the equation to obtain:

[0088]

[0089] Since W is a coefficient matrix, we need to calculate the partial derivative of L with respect to each scalar in the coefficient matrix. Therefore, the partial derivative with respect to the coefficient in the u-th row and v-th column is:

[0090]

[0091] In the above formula, (-1) u+v M uv It is W u，v The algebraic cofactor, as shown in the above equation, indicates that to calculate the partial derivative of L with respect to the elements in ||W||, the probability density function of S needs to be obtained. Clearly, estimating the probability density function of S without prior data is difficult. Theoretically, some error can be tolerated; a convenient probability density function can be estimated, and the partial derivatives can be calculated. Alternatively, the probability density can be analyzed using prior data. After calculating the partial derivatives, the parameters can be updated using the following formula:

[0092] W = W + ηL

[0093] In the above formula, η represents the learning rate for parameter updates. When using gradient descent optimization methods for prediction, the number of iterations or the iteration objective of the ICA analysis can be specified.

[0094] Therefore, it is possible to perform data dimensionality reduction on the standardized external data.

[0095] In some embodiments, the acquisition unit 110 further acquires the training samples; the training samples are used to train a preset machine learning model to obtain the Naive Bayes model; wherein obtaining the Naive Bayes model includes: determining the probability distribution model of the Naive Bayes model based on the sample attributes of the dimensionality-reduced external data in the training samples. In some embodiments, determining the probability distribution model of the Naive Bayes model based on the sample attributes of the dimensionality-reduced external data includes: calculating the probability distribution in the discrete space when the sample attributes are discrete; and estimating the probability model parameters using maximum likelihood estimation when the sample attributes are continuous.

[0096] For example, Naive Bayes is a classic generative model, and the Naive Bayes formula is:

[0097]

[0098] In the above formula, c k For the k-th class in the finite discrete target space C, x i This corresponds to the value of the i-th attribute of a training sample point y. For each attribute value, since estimation based on a probability model is required, the probability model of the attribute needs to be validated first. Typically, it is only necessary to distinguish between discrete and continuous attribute values. For discrete attributes, the probability P(x...) is... i |c k ) equals the attribute value x in the sample i And belongs to c k The number of classes divided by the total number of c in the sample k The number of samples in each class. For continuous attributes, it is assumed by default that their distribution follows a normal distribution, which leads to a probability distribution model based on the normal distribution:

[0099]

[0100] μ and σ are the mean and standard deviation of the normal distribution model, respectively. According to the probability distribution model, we can obtain:

[0101]

[0102] For continuous models, the mean and standard deviation of the distribution of continuous attributes in the sample are needed to calculate the conditional probability P(x). i |c k It is easy to estimate the parameters of a probabilistic model on known samples using existing methods, usually by employing maximum likelihood estimation, which can be found in existing techniques.

[0103] Therefore, the Naive Bayes model generated above is a probability distribution model of sample attributes, while model training is based on the probability distribution model to estimate the required parameters or calculate the probability distribution in the discrete space.

[0104] In some embodiments, determining the quality of the external data to be evaluated based on the Naive Bayes model includes: determining the quality of the external data based on the score value of the external data to be evaluated in the Naive Bayes model, wherein the quality of the external data is proportional to the score value. In some embodiments, the score value of the external data to be evaluated in the Naive Bayes model includes: the maximum value of n probability values ​​of the external data in the discrete target space of the Naive Bayes model is the score value of the data.

[0105] For example, after generating a Naive Bayes model, it can predict the probability of each discrete value in a finite discrete target space for external data to be evaluated for a customer. For instance, the probability distribution in the target discrete space evaluated by the above model is the quality score of the data. The maximum value of the n probability values ​​of the data in the discrete target space is the score of the data; that is, the higher the probability that the data belongs to a certain class, the higher the confidence in classifying the sample, and the higher the score. The formula is expressed as:

[0106] score=argmax{P(y|c1),P(y|c2),…,P(y|c n )}

[0107] Therefore, the quality of external data can be accurately evaluated.

[0108] Figure 3 This is a flowchart illustrating the data evaluation process in an embodiment of this application. Figure 3 As shown:

[0109] Step 301: Obtain external data;

[0110] Step 302: Perform data cleaning and standardization on external data;

[0111] Step 303: Perform dimensionality reduction on the standardized external data to obtain dimensionality-reduced external data;

[0112] Step 304: Input the dimensionality-reduced external data into the Naive Bayes model; wherein, the Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by performing dimensionality reduction on the standardized external data used for model training.

[0113] Step 305: Determine the quality of the external data to be evaluated based on the Naive Bayes model.

[0114] This enables accurate analysis of the quality of external data and provides reliable references for decision-making.

[0115] It should be noted that the hardware structure in the above example may also include devices not shown in the figure. For details, please refer to the prior art. This application embodiment is not intended to limit the hardware structure. Alternatively, the hardware structure may not necessarily include all the components shown in the figure. These will not be listed here.

[0116] For simplicity, the figures only exemplify the connection relationships or signal flows between various components or modules. However, those skilled in the art should understand that such connections can employ various related technologies, such as electrical connections. This application does not limit the scope of the embodiments in this regard.

[0117] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0118] This application is able to accurately analyze the quality of external data and provide reliable reference for decision-making.

[0119] This application also provides a data evaluation method.

[0120] This method corresponds to the data evaluation device in the foregoing embodiments. Embodiments of this method can be found in the foregoing embodiments; repeated details will not be described again.

[0121] Figure 4 This is a schematic diagram of a data evaluation method according to an embodiment of this application; as shown Figure 4 As shown, the method includes:

[0122] Step 401: Obtain the external data to be evaluated;

[0123] Step 402: Perform data cleaning and standardization on the external data, and perform dimensionality reduction on the standardized external data to obtain dimensionality-reduced external data;

[0124] Step 403: Input the dimensionality-reduced external data into the Naive Bayes model, and determine the quality of the external data to be evaluated based on the Naive Bayes model; wherein, the Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by performing dimensionality reduction on the standardized external data used for model training.

[0125] The execution of each of the above steps and their specific details can be found in the descriptions of the functions and structures of the relevant components in the foregoing embodiments, and will not be repeated here.

[0126] This enables accurate analysis of the quality of external data and provides reliable references for decision-making.

[0127] Embodiments of this application also provide a computer device. Figure 5 This is a schematic diagram of the computer device 500 in an embodiment of this application. The computer device 500 is capable of implementing all the steps in the data evaluation method in the above embodiments. The computer device 500 specifically includes the following:

[0128] Processor 501, memory 502, communications interface 503, and communication bus 504;

[0129] The processor 501, memory 502, and communication interface 503 communicate with each other through the communication bus 504; the communication interface 503 is used to realize information transmission between server-side devices, detection devices, and user-side devices and other related devices.

[0130] The processor 501 is used to call the computer program in the memory 502. When the processor executes the computer program, it implements all the steps in the data evaluation method in the above embodiments.

[0131] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the data evaluation method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the data evaluation method in the above embodiments.

[0132] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the data evaluation method described in the above embodiments.

[0133] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations. The user information in the embodiments of this application was obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the users.

[0134] While this invention provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual device or user terminal product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on its spirit and principles, and these modifications and variations are also within the scope of the present application.

Claims

1. A data evaluation method, characterized in that, The method includes: Obtain the external data to be evaluated; The external data is cleaned and standardized, and then the standardized external data is dimensionality reduced to obtain dimensionality-reduced external data. The dimensionality-reduced external data is input into the Naive Bayes model, and the quality of the external data to be evaluated is determined based on the Naive Bayes model; wherein, the Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by reducing the dimensionality of the standardized external data used for model training. The process of determining the quality of the external data to be evaluated based on the Naive Bayes model includes: The quality of the external data is determined based on the score value of the external data to be evaluated in the Naive Bayes model, wherein the quality of the external data is proportional to the score value; The score values ​​of the external data to be evaluated in the Naive Bayes model include: The maximum value of the n probability values ​​of the external data in the discrete target space of the Naive Bayes model is the score value of the data.

2. The method according to claim 1, characterized in that, The external data is cleaned and standardized, including: Data with an effective data rate below a first threshold is discarded to obtain the cleaned external data; and The cleaned external data is standardized using standard deviation.

3. The method according to claim 1, characterized in that, The standardized external data is subjected to dimensionality reduction to obtain dimensionality-reduced external data, including: The signal is obtained by sampling the external data, and independent component analysis is performed on the signal; wherein, the number of iterations or the iteration target of the independent component analysis is specified by gradient descent optimization.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the training samples; The training samples are used to train a preset machine learning model to obtain the Naive Bayes model; wherein, obtaining the Naive Bayes model includes: determining the probability distribution model of the Naive Bayes model based on the sample attributes of the dimensionality-reduced external data in the training samples.

5. The method according to claim 4, characterized in that, The probability distribution model of the Naive Bayes model is determined based on the sample attributes of the dimensionality-reduced external data, including: When the sample attributes are discrete, calculate the probability distribution in the discrete space; When the sample attributes are continuous, maximum likelihood estimation is used to estimate the parameters of the probability model.

6. A data evaluation device, characterized in that, The device includes: The acquisition unit acquires the external data to be evaluated; The data processing unit performs data cleaning and standardization on the external data, and performs dimensionality reduction on the standardized external data to obtain dimensionality-reduced external data. The evaluation unit inputs the dimensionality-reduced external data into the Naive Bayes model and determines the quality of the external data to be evaluated based on the Naive Bayes model. The Naive Bayes model is obtained by training a preset machine learning model using training samples, and the training samples include: dimensionality-reduced external data obtained by reducing the dimensionality of the standardized external data used for model training. The evaluation unit is used for: The quality of the external data is determined based on the score value of the external data to be evaluated in the Naive Bayes model, wherein the quality of the external data is proportional to the score value; The score values ​​of the external data to be evaluated in the Naive Bayes model include: The maximum value of the n probability values ​​of the external data in the discrete objective space of the Naive Bayes model is the score value of the data.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1 to 5.

Images