Automatic data standard construction method and device based on machine learning

Abstract

The embodiment of the invention provides an automatic data standard construction method and device based on machine learning, and the method comprises the steps: obtaining original data features in historical business data of an enterprise, determining the index correlation of each original data feature according to the linear correlation degree between the original data features and a target business index, and obtaining the index correlation of each original data feature; determining feature importance of each original data feature according to Gini exponential gain generated when the original data features are split at nodes of a preset decision tree, and determining principal component contribution degree of each original data feature according to feature values and feature vectors of covariance matrixes corresponding to the original data features; performing business abstraction on the target data through the index classification model to obtain atomic indexes; receiving a data standard construction request sent by a user, and automatically combining to generate a corresponding business report; according to the method, multi-dimensional feature evaluation can be comprehensively considered, business semantics can be understood, and the accuracy of data standard construction is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method and device for automatically constructing data standards based on machine learning. Background Art

[0002] In the current era of data-driven decision-making, building high-quality data standards is crucial for enterprises. However, existing data standard construction methods have many limitations and are difficult to meet the needs of large-scale and complex business scenarios.

[0003] First, the traditional data standard construction method mainly relies on manual experience. This method is not only time-consuming and labor-intensive, but also easily affected by personal subjective judgment, making it difficult to ensure the consistency and scientificity of the standards. When faced with a large amount of complex business data, the efficiency of manual methods is even more difficult to meet the needs.

[0004] Second, some existing automated methods are often too simplistic and fail to fully consider the complex relationship between data features and business goals. For example, some methods select features based only on simple statistical indicators (such as correlation coefficients), ignoring the nonlinear relationships and interactions that may exist in the data.

[0005] Finally, existing methods often lack close integration with actual business needs. They may generate a set of indicators that seem statistically reasonable, but these indicators may be difficult to directly apply to actual business decisions and report generation.

[0006] These limitations highlight the need to develop more intelligent, comprehensive, and business-insightful approaches to automated data standardization. Summary of the invention

[0007] In response to the problems in the prior art, the present application provides a method and device for automatic data standard construction based on machine learning, which can comprehensively consider multi-dimensional feature evaluation, understand business semantics, and improve the accuracy of data standard construction.

[0008] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a method for automatically constructing data standards based on machine learning, comprising:

[0010] Obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0011] According to the indicator correlation, the feature importance and the principal component contribution, the key factor features in the original data features are determined, the original data features and the key factor features are input into the input layer of the multi-layer perceptron model for model training, and the weights are tuned by calculating the gradient mean and the gradient uncentered variance in the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0012] The target data is business abstracted through the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators; a data standard construction request sent by a user is received, and the corresponding type indicators and measurement indicators are determined according to the target business indicators and target business dimensions in the data standard construction request, and a corresponding business report is generated according to the combination of the type indicators and the measurement indicators.

[0013] Furthermore, the acquiring of original data features in the enterprise's historical business data and determining the indicator correlation of each original data feature according to the linear correlation between the original data features and the target business indicator include:

[0014] Obtain the company's historical business data and perform feature engineering to obtain the original data features;

[0015] The Pearson correlation coefficient between the original data features and the target business indicators is calculated to determine the indicator correlation of each of the original data features.

[0016] Furthermore, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node includes:

[0017] Building a decision tree based on the original data features, evaluating all possible feature splits at each node of the decision tree, and calculating the Gini impurity before and after the split to obtain the Gini index gain generated during the split;

[0018] The feature importance of each of the original data features is determined according to the accumulated value of the Gini index gain generated during the splitting.

[0019] Furthermore, determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features includes:

[0020] Standardizing the original data features to obtain corresponding standardized features, calculating the covariance matrix between the standardized features and performing eigenvalue decomposition on the covariance matrix to obtain corresponding eigenvalues ​​and eigenvectors;

[0021] Calculate the proportion of each eigenvalue to the total eigenvalues ​​to obtain the variance proportion explained by each principal component, calculate the load corresponding to the eigenvector, square the load and multiply it by the variance proportion of each principal component to obtain the weighted square load, and for each original feature, calculate the sum of its weighted square loads on all principal components to determine the principal component contribution of each original data feature.

[0022] Furthermore, determining the key factor features in the original data features according to the indicator correlation, the feature importance and the principal component contribution includes:

[0023] After normalizing the index correlation, the feature importance and the principal component contribution, corresponding weights are assigned through a cross-validation method, and a comprehensive score of each of the original data features is calculated;

[0024] The original data features are sorted in descending order according to the comprehensive scores to determine key factor features in the original data features.

[0025] Furthermore, the target data is business abstracted by the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators, including:

[0026] Input each indicator in the target data into the indicator classification model to obtain a classification result output by the indicator classification model, wherein the classification result includes a type indicator and a measurement indicator;

[0027] Perform semantic analysis on the type indicators and measurement indicators to extract the core business meaning and calculation logic of the indicators. For atomic indicators classified as type indicators, determine their possible value ranges and business rules; for atomic indicators classified as measurement indicators, determine their measurement units, aggregation methods and calculation formulas.

[0028] Further, the receiving a data standard construction request sent by a user, determining a corresponding type indicator and a metric indicator according to a target business indicator and a target business dimension in the data standard construction request, and generating a corresponding business report according to a combination of the type indicator and the metric indicator, includes:

[0029] Receive and parse the data standard construction request sent by the user, extract the target business indicators and target business dimension information, match the target business indicators with the measurement indicators, and match the target business dimensions with the type indicators according to the pre-established indicator library;

[0030] Determine the report's hierarchy and layout based on the logical relationship between indicators of different types. Add calculation logic to the report based on the calculation formula, aggregation method, and display format of the metrics to generate the corresponding business report.

[0031] In a second aspect, the present application provides an automatic data standard construction device based on machine learning, comprising:

[0032] A data feature analysis module is used to obtain the original data features in the enterprise's historical business data, determine the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determine the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at the preset decision tree nodes, and determine the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0033] An indicator model building module is used to determine the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, input the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and perform weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is met, thereby obtaining an indicator classification model;

[0034] The data standard construction module is used to perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report according to the combination of the type indicators and the measurement indicators.

[0035] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for constructing automatic data standards based on machine learning when executing the program.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the automatic data standard construction method based on machine learning.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the automatic data standard construction method based on machine learning.

[0038] It can be seen from the above technical scheme that the present application provides a method and device for automatic data standard construction based on machine learning, which obtains the original data features in the historical business data of the enterprise, determines the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determines the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determines the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; performs business abstraction on the target data through the indicator classification model to obtain atomic indicators; receives the data standard construction request sent by the user, and automatically combines and generates the corresponding business report, thereby comprehensively considering multi-dimensional feature evaluation, understanding business semantics, and improving the accuracy of data standard construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is one of the flowcharts of the automatic data standard construction method based on machine learning in the embodiment of the present application;

[0041] Figure 2 This is a second flow chart of the automatic data standard construction method based on machine learning in an embodiment of the present application;

[0042] Figure 3 The third flowchart of the automatic data standard construction method based on machine learning in the embodiment of the present application;

[0043] Figure 4 This is a fourth flowchart of the method for automatically constructing data standards based on machine learning in an embodiment of the present application;

[0044] Figure 5This is a fifth flow chart of the automatic data standard construction method based on machine learning in an embodiment of the present application;

[0045] Figure 6 This is a sixth flow chart of the automatic data standard construction method based on machine learning in an embodiment of the present application;

[0046] Figure 7 FIG7 is a flowchart of the automatic data standard construction method based on machine learning in an embodiment of the present application;

[0047] Figure 8 A structural diagram of an automatic data standard construction device based on machine learning in an embodiment of the present application;

[0048] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0049] Reference numerals:

[0050] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0053] In view of the problems existing in the prior art, the present application provides a method and device for automatically constructing data standards based on machine learning, which obtains the original data features in the historical business data of the enterprise, determines the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determines the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determines the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; performs business abstraction on the target data through the indicator classification model to obtain atomic indicators; receives the data standard construction request sent by the user, and automatically combines and generates the corresponding business report, thereby comprehensively considering multi-dimensional feature evaluation, understanding business semantics, and improving the accuracy of data standard construction.

[0054] In order to comprehensively consider multi-dimensional feature evaluation, understand business semantics, and improve the accuracy of data standard construction, this application provides an embodiment of an automatic data standard construction method based on machine learning, see Figure 1 , the automatic data standard construction method based on machine learning specifically includes the following contents:

[0055] Step S101: obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0056] Optionally, in this embodiment, step S101 is a key step in the automatic data standard construction method based on machine learning, which mainly involves the acquisition and evaluation of data features. The core goal of this step is to extract valuable information from the company's historical business data and evaluate this information through multiple dimensions, laying the foundation for subsequent model construction and data standard formulation.

[0057] First, this step requires obtaining the original data features from the company's historical business data. This process usually involves technologies such as data cleaning and feature engineering. Data cleaning may include processing missing values, outliers, and unifying data formats. Feature engineering may involve operations such as feature selection, feature combination, and feature conversion, with the goal of extracting the most valuable information from the original data. For example, for an e-commerce company, the original data features may include users' purchase history, browsing history, search keywords, user demographic characteristics, etc.

[0058] Next, this step evaluates the importance of these features from three different perspectives: indicator correlation, feature importance, and principal component contribution. These three dimensions evaluate the importance of features from the perspectives of linear correlation, nonlinear relationship, and data structure, and can comprehensively capture the various attributes of features.

[0059] The Pearson correlation coefficient is used to calculate the correlation between indicators. This is a common statistical method used to measure the degree of linear correlation between two variables. By calculating the Pearson correlation coefficient between each raw data feature and the target business indicator, we can determine which features are most correlated with the target indicator. For example, we may find that the user's purchase frequency has a strong positive correlation with their probability of purchase in the next 30 days.

[0060] The calculation of feature importance utilizes the decision tree model. A decision tree is a nonlinear model that can capture complex interactions between features. By constructing a decision tree and calculating the Gini index gain of each feature when the node is split, we can evaluate the importance of each feature in the classification task. The Gini index is an indicator of the purity of a dataset, and the Gini index gain reflects the contribution of a feature to improving the purity of a dataset. For example, we may find that a user's search keywords are of high importance in predicting user interests.

[0061] The calculation of the principal component contribution uses the principal component analysis (PCA) method. PCA is a dimensionality reduction technique that can find the most important direction of change in the data. By calculating the covariance matrix of the features and performing eigenvalue decomposition, we can get the contribution of each feature in the principal component. This method is particularly suitable for processing high-dimensional data and can effectively capture the overall structure of the data. For example, we may find that the two features of the user's purchase category and purchase amount have a great contribution to the overall variation of the data.

[0062] This step solves several key technical problems. First, it solves the problem of how to extract valuable information from complex corporate historical data. Through feature engineering, we can transform raw data into features that can be used for machine learning. Second, it solves the problem of how to comprehensively evaluate the importance of features. By combining linear and nonlinear methods, and considering the overall structure of the data, we can evaluate the importance of features from multiple perspectives. Finally, it also solves the problem of how to deal with high-dimensional data. Through principal component analysis, we can effectively reduce the dimensionality of the data while retaining the most important information.

[0063] The technical effect achieved by this step is significant. First, it realizes the deep mining of the enterprise's historical data and extracts the most valuable information from the massive data. Second, through multi-dimensional feature evaluation, it provides a reliable feature selection basis for subsequent model construction, which helps to improve the performance and interpretability of the model. Third, this method improves the efficiency and accuracy of data analysis, enabling enterprises to better understand and utilize their data assets.

[0064] Step S102: determining the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, inputting the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and performing weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0065] Optionally, in this embodiment, step S102 is a key link in the automatic data standard construction method based on machine learning, which mainly involves the determination of key factor features and the construction of an indicator classification model. The core goal of this step is to filter out the most important features from the original data features and use these features to build a model that can accurately classify indicators.

[0066] First, this step needs to determine the key factor features in the original data features based on the previously calculated indicator correlation, feature importance, and principal component contribution. This process usually involves feature selection techniques. A common method is to standardize the three indicators, and then assign weights to each indicator through a cross-validation method to calculate a comprehensive score for each original data feature. Based on this comprehensive score, we can sort the original data features in descending order and select the top-ranked features as the key factor features. This method can comprehensively consider the linear correlation, nonlinear importance, and contribution to the overall structure of the data of the features, thereby selecting the most representative and predictive features.

[0067] Next, this step inputs the original data features and key factor features into the input layer of the multilayer perceptron model for model training. The multilayer perceptron is a feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer. Each layer is composed of multiple neurons, and the neurons between adjacent layers are fully connected. This structure enables the multilayer perceptron to learn complex nonlinear relationships and is suitable for processing high-dimensional data and complex classification problems. In this method, the original data features and key factor features together constitute the input of the model. The purpose of this is to retain all possible useful information and highlight the most important features, which helps the model learn effective feature combinations more quickly.

[0068] During model training, this method performs weight tuning by calculating the mean gradient and the uncentered variance of the gradient. The mean gradient reflects the overall direction of parameter updates, while the uncentered variance of the gradient reflects the uncertainty of parameter updates. By considering these two indicators at the same time, we can better control the learning process of the model. For example, when the mean gradient is large and the variance is small, we can use a larger learning rate to speed up model convergence; conversely, when the mean gradient is small and the variance is large, we may need to reduce the learning rate to avoid the model falling into a local optimal solution. This method of dynamically adjusting the learning rate helps improve the training efficiency and generalization ability of the model.

[0069] Model training will continue until the preset convergence conditions are reached. Common convergence conditions include reaching the maximum number of iterations, the loss function value is lower than a certain threshold, and the performance on the validation set no longer improves. When the model converges, we get a model that can classify indicators.

[0070] This step solves several key technical problems. First, it solves the problem of how to select the most important features from a large number of original features. By comprehensively considering the importance of features in multiple dimensions, we can effectively reduce the dimensionality of the data while retaining the most critical information. Second, it solves the problem of how to build a model that can accurately classify indicators. By using a multi-layer perceptron model, we can capture the complex nonlinear relationship between features, thereby improving the accuracy of classification. Finally, it also solves the problem of how to effectively train neural network models. By dynamically adjusting the learning rate, we can improve the efficiency of training while ensuring model convergence.

[0071] The technical effect achieved by this step is significant. First, it realizes the effective screening of the original data features, greatly reduces the amount of data to be processed later, and improves the training speed and efficiency of the model. Second, by building a multi-layer perceptron model, it can accurately classify indicators, providing a reliable foundation for the subsequent construction of data standards. Third, by dynamically adjusting the learning rate, it improves the training efficiency and generalization ability of the model, allowing the model to better adapt to different data distributions.

[0072] Step S103: Perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, where the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report based on the combination of the type indicators and the measurement indicators.

[0073] Optionally, in this embodiment, step S103 is a key step in the automatic data standard construction method based on machine learning, which mainly involves business abstraction, atomic indicator generation, and business report construction. The core goal of this step is to apply the previously trained indicator classification model to actual business data, thereby achieving data standardization and automatic generation of business reports.

[0074] First, this step uses the indicator classification model to perform business abstraction on the target data and obtain atomic indicators. This process involves the inference application of the model. Specifically, we input the target data into the previously trained multi-layer perceptron model, and the model will classify each data item as a type indicator or a metric indicator. Type indicators are usually descriptive, discrete data, such as product categories, customer levels, etc.; metric indicators are usually quantifiable, continuous data, such as sales, order quantity, etc. This classification method can effectively structure complex business data and lay the foundation for subsequent data analysis and report generation.

[0075] Next, this step needs to receive the data standard construction request sent by the user. This request usually contains the target business indicator and target business dimension that the user is interested in. For example, the user may want to know "monthly sales of different product categories", where "monthly sales" is the target business indicator and "product category" is the target business dimension. The system needs to parse this request and find the corresponding type indicator and measurement indicator in the previously generated atomic indicators.

[0076] After determining the corresponding type indicators and metric indicators, the system will generate the corresponding business report based on the combination of these indicators. This process involves multiple steps such as data aggregation, calculation, and visualization. First, the system groups the data according to the type indicators (such as product categories), and then aggregates and calculates the metric indicators (such as sales) within each group (such as sum, average, etc.). Next, the system will select the appropriate chart type (such as bar charts, line charts, etc.) to visualize the data. Finally, the system will generate a business report containing a data table and a visualization chart.

[0077] This step solves several key technical problems. First, it solves the problem of how to standardize and structure complex business data. By using the indicator classification model, we can automatically classify various business data into type indicators or measurement indicators, which greatly simplifies the complexity of data processing. Secondly, it solves the problem of how to flexibly generate business reports according to user needs. By parsing user requests and matching corresponding atomic indicators, the system can realize automatic generation of reports and improve the efficiency of data analysis. Finally, it also solves the problem of how to present complex data to users in an intuitive and easy-to-understand way. By choosing the appropriate visualization method, the system can help users better understand and use data.

[0078] The technical effect achieved by this step is significant. First, it realizes the automated standardization of business data, greatly reduces the workload of manual data processing, and improves the efficiency and consistency of data processing. Second, by automatically generating business reports, it greatly improves the efficiency of data analysis, allowing enterprises to make data-driven decisions more quickly. Third, by providing intuitive data visualization, it improves the readability and comprehensibility of data, allowing users with non-technical backgrounds to easily interpret complex business data.

[0079] From the above description, it can be seen that the automatic data standard construction method based on machine learning provided in the embodiment of the present application can obtain the original data features in the historical business data of the enterprise, determine the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determine the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determine the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; perform business abstraction on the target data through the indicator classification model to obtain atomic indicators; receive data standard construction requests sent by users, and automatically combine and generate corresponding business reports, thereby comprehensively considering multi-dimensional feature evaluation, understanding business semantics, and improving the accuracy of data standard construction.

[0080] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 2 , and can also include the following:

[0081] Step S201: Acquire the historical business data of the enterprise and perform feature engineering to obtain the original data features;

[0082] Step S202: Calculate the Pearson correlation coefficient between the original data features and the target business indicators to determine the indicator correlation of each of the original data features.

[0083] Optionally, in this embodiment, steps S201 and S202 are key steps in the data analysis and machine learning process, which together form the basis for data preparation and preliminary analysis. The main goal of these two steps is to extract valuable information from the original historical business data of the enterprise and perform preliminary correlation analysis on this information.

[0084] First, step S201 involves data acquisition and feature engineering. Data acquisition is the starting point of the entire analysis process, which usually involves extracting historical business data from various data sources of the enterprise (such as databases, log files, APIs, etc.). This process may require processing a large amount of raw data, so distributed computing frameworks such as Hadoop or Spark are usually used to improve processing efficiency. After acquiring the data, feature engineering is performed next, which is a process of converting raw data into features that can be used for machine learning. Feature engineering usually includes steps such as data cleaning, feature extraction, feature selection, and feature conversion.

[0085] Data cleaning is the first step of feature engineering, which mainly deals with problems such as missing values, outliers, and duplicate data. For example, you can use the mean, median, or machine learning-based methods to fill missing values; use statistical methods or domain knowledge to identify and handle outliers; delete or merge duplicate data, etc. The next step is feature extraction, which aims to create new and more informative features from the original data. For example, for time series data, we may extract time-related features, such as the day of the week, whether it is a holiday, etc.; for text data, we may use natural language processing technology to extract keywords or topics.

[0086] Feature selection is the process of selecting the most relevant and useful subset of features from existing features. This step can help reduce the dimensionality of the data and improve the efficiency and generalization ability of the model. Commonly used feature selection methods include correlation-based methods (such as correlation coefficients), model-based methods (such as feature importance using decision trees), etc. The last step is feature transformation, which may include standardization, normalization, logarithmic transformation, etc., in order to make the features meet certain statistical assumptions or improve the performance of the model.

[0087] Step S202 focuses on calculating the correlation between the original data features and the target business indicators. This step uses the Pearson correlation coefficient, which is a statistic that measures the degree of linear correlation between two variables. The value of the Pearson correlation coefficient is between -1 and 1, with 1 indicating a perfect positive correlation, -1 indicating a perfect negative correlation, and 0 indicating no linear correlation.

[0088] The process of calculating the Pearson correlation coefficient usually includes the following steps: First, each raw data feature and target business indicator is standardized to have a mean of 0 and a standard deviation of 1. Then, the covariance of each pair of features and target indicators is calculated. Finally, the covariance is divided by the product of the standard deviations of the two variables to obtain the correlation coefficient. This process needs to be repeated for all raw data features to obtain a correlation coefficient matrix.

[0089] These two steps solve several key technical problems. First, they solve the problem of how to extract useful information from complex and messy raw data. Through feature engineering, we can transform raw data into a more structured and information-rich form. Second, they solve the problem of how to preliminarily evaluate the importance of features. By calculating the correlation coefficient, we can quickly identify which features may have an important impact on the target indicator. Finally, they also lay the foundation for the subsequent modeling process and provide high-quality input data.

[0090] The technical effects achieved by these two steps are significant. First, they greatly improve the quality and availability of data. Through feature engineering, the original data is transformed into a form more suitable for machine learning, which not only improves the performance of subsequent models, but also increases the interpretability of the models. Second, through correlation analysis, we gain a preliminary understanding of the data, which can guide subsequent feature selection and model design. Third, this automated data processing and analysis method greatly improves the work efficiency of data scientists, allowing them to gain insights from data more quickly.

[0091] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 3 , and can also include the following:

[0092] Step S301: construct a decision tree based on the original data features, evaluate all possible feature splits at each node of the decision tree, and calculate the Gini impurity before and after the split to obtain the Gini index gain generated during the split;

[0093] Step S302: determining the feature importance of each of the original data features according to the accumulated value of the Gini index gain generated during the splitting.

[0094] Steps S301 and S302 are key steps for feature selection and evaluation in the decision tree algorithm. The main goal of these two steps is to evaluate the importance of each feature by building a decision tree, thereby providing a basis for subsequent model optimization and feature screening.

[0095] First, step S301 involves the process of building a decision tree. A decision tree is a non-parametric supervised learning method that makes decisions through a series of if-then rules. In this step, we use the original data features as input to build a decision tree. The process of building a decision tree is top-down, starting from the root node, by continuously selecting the best features and split points to split the data set.

[0096] At each node in the decision tree, we need to evaluate all possible feature splits. For each feature, we consider all possible split points and calculate the Gini impurity before and after the split. Gini impurity is a measure of the purity of a node. It reflects the probability that two samples randomly drawn from the node have inconsistent categories. The smaller the Gini impurity, the purer the node, that is, the samples in the node tend to belong to the same category.

[0097] For each possible split, we calculate the difference between the Gini impurity before and after the split, and this difference is the Gini index gain. The Gini index gain reflects the contribution of this split to improving the purity of the node. We select the feature and split point with the largest Gini index gain to perform the actual split. This process will be recursively performed until the stopping condition is met (for example, the maximum depth is reached, or the number of samples in the node is less than a certain threshold).

[0098] Step S302 focuses on evaluating the importance of features based on the decision tree construction process. In this step, we accumulate the Gini index gain generated by each feature on all nodes. This accumulated value reflects the total contribution of the feature to improving the purity of the node during the entire decision tree construction process. The larger the accumulated value of the feature, the more important the feature is.

[0099] Finally, we can normalize the accumulated values ​​of all features to get the relative importance of each feature. The value range of this relative importance is usually between 0 and 1, and the sum of the importance of all features is 1. In this way, we can intuitively compare the importance of different features.

[0100] These two steps solve several key technical problems. First, they solve the problem of how to identify important features in a high-dimensional feature space. Through the process of building a decision tree, we can automatically find which features are most helpful for predicting the target variable. Second, they solve the problem of how to deal with nonlinear relationships. Decision trees can capture nonlinear relationships between features and target variables, which simple linear models cannot do. Finally, they also solve the problem of how to deal with categorical features and continuous features. Decision trees can handle both types of features at the same time without complex preprocessing.

[0101] The technical effects achieved by these two steps are significant. First, they provide an objective and automated way to assess feature importance. This not only helps us understand which factors are most important to business goals, but also guides subsequent feature engineering and model optimization. Second, this method has good interpretability. We can understand how the model makes decisions by looking at the structure of the decision tree, which is very useful for scenarios where we need to explain the model results to non-technical people. Third, this method is computationally efficient and can quickly produce results even on large-scale datasets.

[0102] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 4 , and can also include the following:

[0103] Step S401: Standardizing the original data features to obtain corresponding standardized features, calculating the covariance matrix between the standardized features and performing eigenvalue decomposition on the covariance matrix to obtain corresponding eigenvalues ​​and eigenvectors;

[0104] Step S402: Calculate the proportion of each eigenvalue to the total eigenvalues ​​to obtain the variance proportion explained by each principal component, calculate the load corresponding to the eigenvector, square the load and multiply it by the variance proportion of each principal component to obtain the weighted square load, and for each original feature, calculate the sum of its weighted square loads on all principal components to determine the principal component contribution of each original data feature.

[0105] Optionally, in this embodiment, steps S401 and S402 constitute the core process of principal component analysis (PCA), which is a statistical method widely used in data dimensionality reduction and feature extraction. The main goal of these two steps is to convert the original features into a set of mutually orthogonal new features (principal components) through linear transformation, and evaluate the contribution of each original feature to these principal components.

[0106] First, step S401 involves data preprocessing and calculation and decomposition of the covariance matrix. Data standardization is an important pre-step of PCA, which ensures that features of different scales can be compared under the same standard. Standardization usually uses the Z-score method, that is, the value of each feature is subtracted from the mean of the feature and then divided by the standard deviation. After this processing, the mean of each feature is 0 and the standard deviation is 1.

[0107] After standardization, we calculate the covariance matrix between these standardized features. The covariance matrix reflects the linear relationship between features. Its diagonal elements are the variances of each feature, and the off-diagonal elements are the covariances between features. For standardized data, the covariance matrix is ​​actually the correlation coefficient matrix.

[0108] Next, we perform eigenvalue decomposition on the covariance matrix. This step is the core of PCA, and it can find the main directions (principal components) in the data. Eigenvalue decomposition decomposes the covariance matrix into eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the variance of the corresponding principal component, while the eigenvectors define the direction of the principal component. Usually, we sort by eigenvalue from largest to smallest, so that the first principal component is the direction with the largest variance, and so on.

[0109] Step S402 focuses on calculating the importance of each principal component and the contribution of the original features to the principal component. First, we calculate the proportion of each eigenvalue to the total eigenvalue sum, which is the proportion of variance explained by the corresponding principal component. This proportion reflects the importance of the principal component in the overall data variation.

[0110] Next, we calculate the load corresponding to the eigenvector. The load is actually the correlation coefficient between the original feature and the principal component, which reflects the contribution of the original feature to the principal component. The square of the load is the contribution ratio of the original feature to the variance of the principal component.

[0111] In order to fully assess the importance of each original feature, we need to consider its contribution to all principal components. Therefore, we multiply the square of the loading by the variance proportion of the corresponding principal component to obtain a weighted square loading. This weighting takes into account the importance of the principal component itself.

[0112] Finally, we calculate the sum of the weighted squared loadings of each original feature on all principal components. This sum is the principal component contribution of the original feature, which reflects the importance of the feature in the entire PCA process.

[0113] These two steps solve several key technical problems. First, they solve the problem of how to reduce the dimensionality of data while retaining the main information of the data. Through PCA, we can use fewer principal components to represent the original data, thereby achieving data dimensionality reduction. Second, they solve the problem of how to deal with the correlation between features. PCA can transform related features into unrelated principal components, thereby eliminating redundant information between features. Finally, they also solve the problem of how to evaluate the importance of original features, providing a new perspective for feature selection.

[0114] The technical effects achieved by these two steps are significant. First, they provide an effective method for data dimensionality reduction, which can greatly reduce the computational complexity of subsequent modeling. Second, by analyzing the variance ratio of the principal components, we can understand the inherent structure of the data and find the most important patterns in the data. Third, by calculating the principal component contribution of the original features, we can evaluate the importance of features from a new perspective, which is of great significance for feature selection and understanding of data.

[0115] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 5 , and can also include the following:

[0116] Step S501: After normalizing the index correlation, the feature importance and the principal component contribution, corresponding weights are assigned through a cross-validation method, and a comprehensive score of each of the original data features is calculated;

[0117] Step S502: sorting the original data features in descending order according to the comprehensive scores, and determining key factor features in the original data features.

[0118] Optionally, in this embodiment, steps S501 and S502 constitute a process of comprehensively evaluating the importance of the original data features, which combines the results of multiple feature selection methods to obtain a more comprehensive and robust feature importance evaluation. The main goal of these two steps is to integrate the feature importance indicators of different dimensions obtained in the previous steps to form a unified scoring standard and ultimately screen out the most critical features.

[0119] First, step S501 involves standardizing the three feature importance indicators (indicator correlation, feature importance, and principal component contribution) obtained previously. The purpose of standardization is to convert these three indicators that may have different scales to the same scale so that they can be directly compared and integrated. The standardization method can use minimum-maximum scaling or Z-score standardization. For example, when using minimum-maximum scaling, we will convert the value of each indicator to a value between 0 and 1, where 0 represents the least important feature under the indicator and 1 represents the most important feature.

[0120] After standardization, we need to assign weights to these three indicators. Cross-validation is used here to determine the optimal weight combination. Cross-validation is a model evaluation method that can evaluate the performance of a model on an independent data set. In this scenario, we can try different weight combinations, each time using the current weight combination to select features and train the model, and then evaluate the model performance on the validation set. We choose the weight combination that enables the best model performance as the final weight.

[0121] For example, we can use the grid search method to try different weight combinations. Assuming that we set the weight range to 0 to 1 and the step size to 0.1, there are 66 combinations we need to try (because the sum of the three weights must be 1). For each combination, we perform k-fold cross validation (for example, 5 folds) and finally select the weight combination with the best average performance.

[0122] After obtaining the weights, we can calculate the comprehensive score of each raw data feature. The comprehensive score is the weighted sum of the three standardized indicators multiplied by their respective weights. This comprehensive score reflects the comprehensive importance of the feature in multiple dimensions.

[0123] Step S502 is to sort and filter the original data features based on the comprehensive score. We sort all the features from high to low according to the comprehensive score, and the features at the top are considered to be more important. As for how many features to select as key factor features, this needs to be determined according to the actual situation. You can choose a fixed number (such as the first 10 features), or you can choose features whose cumulative importance reaches a certain threshold (such as 80%), or determine a suitable cutoff point by observing the changing trend of the comprehensive score.

[0124] These two steps solve several key technical problems. First, they solve the problem of how to integrate the results of multiple feature selection methods. Different feature selection methods may be based on different principles and therefore may produce different results. By combining these results, we can get a more comprehensive and robust evaluation. Second, they solve the problem of how to balance the importance of different feature selection methods. By determining the weights through cross-validation, we can find a balance point that maximizes model performance. Finally, they also solve the problem of how to screen out the most critical features from a large number of features, providing a basis for subsequent modeling work.

[0125] The technical effects achieved by these two steps are significant. First, they provide a more comprehensive and robust method for evaluating feature importance. By combining the results of multiple methods, we can reduce the bias that may be caused by a single method. Second, this method is highly adaptable and flexible. By adjusting the weights, we can adjust the importance of different feature selection methods according to the characteristics of the specific problem. Third, this method can greatly improve the efficiency and effectiveness of subsequent modeling. By screening out the most critical features, we can reduce the complexity of the model and improve the generalization ability of the model.

[0126] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 6 , and can also include the following:

[0127] Step S601: input each indicator in the target data into the indicator classification model to obtain a classification result output by the indicator classification model, wherein the classification result includes a type indicator and a measurement indicator;

[0128] Step S602: Perform semantic analysis on the type indicators and measurement indicators to extract the core business meaning and calculation logic of the indicators. For atomic indicators classified as type indicators, determine their possible value ranges and business rules; for atomic indicators classified as measurement indicators, determine their measurement units, aggregation methods and calculation formulas.

[0129] Optionally, in this embodiment, steps S601 and S602 constitute a process of data indicator classification and semantic analysis, which is a key link in data governance and intelligent analysis. The main goal of these two steps is to automatically classify the indicators in the target data and further extract the semantic information of these indicators, laying the foundation for subsequent data analysis and decision support.

[0130] First, step S601 involves using an indicator classification model to classify each indicator in the target data. The indicator classification model is usually a pre-trained machine learning model, which may be based on algorithms such as decision trees, random forests, support vector machines, or deep learning. The input of this model is the relevant features of the indicator, which may include the indicator name, data type, value range, statistical characteristics, etc. The output of the model is the classification result of the indicator, which is mainly divided into two categories: type indicators and measurement indicators.

[0131] Type indicators are usually discrete, categorized data used to describe the characteristics or attributes of data. For example, product category, customer gender, order status, etc. are all typical type indicators. Metric indicators are usually continuous, quantifiable data used to measure a phenomenon or performance. For example, sales, number of customers, order amount, etc. are all typical metrics.

[0132] The technical principle of this classification process is to use machine learning algorithms to learn classification rules from a large amount of labeled indicator data. The model learns how to distinguish between type indicators and measurement indicators by analyzing various characteristics of the indicators. In practical applications, this model may need to be regularly updated and fine-tuned with new data to adapt to the ever-changing business environment.

[0133] Step S602 is to conduct a more in-depth semantic analysis of the classified indicators. This process involves natural language processing (NLP) technology, and the goal is to extract the core business meaning and calculation logic from the name and description of the indicator. For type indicators, we need to determine their possible value ranges and related business rules. For example, for the type indicator of "customer level", we need to determine the possible levels (such as gold card, silver card, ordinary card) and the definition rules for each level. For measurement indicators, we need to determine their measurement units (such as yuan, pieces, hours, etc.), aggregation methods (such as sum, average, maximum value, etc.) and specific calculation formulas.

[0134] This semantic parsing process may involve a variety of NLP technologies, such as named entity recognition, relationship extraction, semantic role labeling, etc. For example, we can use named entity recognition technology to identify business entities and measurement units in indicator names, use relationship extraction technology to identify the calculation relationship between indicators, and use semantic role labeling technology to understand the calculation logic in indicator descriptions.

[0135] These two steps solve several key technical problems. First, they solve the problem of how to automate indicator classification. In a big data environment, manually classifying a large number of indicators is very time-consuming and error-prone, while automated classification can greatly improve efficiency and consistency. Second, they solve the problem of how to extract structured information from unstructured indicator descriptions. Through semantic parsing, we can convert human-readable indicator descriptions into machine-understandable structured information. Finally, they also solve the problem of how to unify the indicator caliber, providing standardized indicator definitions for subsequent data analysis and report generation.

[0136] The technical effects achieved by these two steps are significant. First, they realize the automatic classification and semantic understanding of indicators, greatly improving the efficiency and accuracy of data governance. Second, by extracting the core business meaning and calculation logic of indicators, they provide clear indicator definitions for subsequent data analysis and decision support, reducing errors caused by deviations in indicator understanding. Third, this automated method can handle large-scale indicator data, providing strong support for enterprise data asset management.

[0137] In one embodiment of the automatic data standard construction method based on machine learning of the present application, see Figure 7 , and can also include the following:

[0138] Step S701: receiving and parsing the data standard construction request sent by the user, extracting the target business indicator and target business dimension information therein, matching the target business indicator with the measurement indicator, and matching the target business dimension with the type indicator according to the pre-established indicator library;

[0139] Step S702: Determine the hierarchy and layout of the report based on the logical relationship between the type indicators, add calculation logic to the report based on the calculation formula, aggregation method and display format of the measurement indicators, and generate the corresponding business report.

[0140] Optionally, in this embodiment, steps S701 and S702 constitute an intelligent data standard construction and report generation process. The main goal of these two steps is to automatically match corresponding indicators according to user needs and generate structured business reports based on these indicators. This process involves multiple technical fields such as natural language processing, data matching, data structure design, and report logic construction, aiming to improve the efficiency and accuracy of report generation.

[0141] First, step S701 involves receiving and parsing the user's data standard construction request. This process uses natural language processing (NLP) technology to understand the user's needs. The system needs to extract key information from the user's request, mainly including target business indicators and target business dimensions. For example, if the user requests "I need a report showing sales and profit margin by month and product category", the system needs to recognize that "month" and "product category" are target business dimensions, and "sales" and "profit margin" are target business indicators.

[0142] After extracting this information, the system will match it with the pre-established indicator library. The indicator library is a database that contains all standardized indicator definitions. The indicators in it have been classified according to type indicators and measurement indicators, and contain detailed definitions, calculation logic and business rules for each indicator. The system will match the target business indicators in the user request with the measurement indicators in the indicator library, and match the target business dimensions with the type indicators in the indicator library. This matching process may involve fuzzy matching and semantic similarity calculation to handle different expressions that users may use.

[0143] Step S702 is to design the structure and logic of the report based on the matching results. First, the system analyzes the logical relationship between the type indicators to determine the hierarchy and layout of the report. For example, if "product category" and "month" are both type indicators, the system needs to decide which one is the primary dimension and which one is the secondary dimension to determine the row and column structure of the report. This decision may be based on predefined business rules or on the learning of past report structures by the machine learning model.

[0144] Next, the system will add calculation logic based on the attributes of the metric. Each metric has a defined calculation formula, aggregation method, and display format in the metric library. The system needs to convert this information into actual data processing logic. For example, if the calculation formula for "profit margin" is "(sales - cost) / sales", the aggregation method is "average", and the display format is "percentage", the system needs to build a corresponding SQL query or data processing script to implement this calculation logic.

[0145] Finally, the system will integrate the designed structure and logic to generate a complete business report. This report may be an interactive data visualization interface or an exportable static report file, depending on the system design and user needs.

[0146] These two steps solve several key technical problems. First, they solve the problem of how to convert users' natural language needs into clear data query needs. Through NLP technology and indicator matching, the system can accurately understand the user's intentions and map them to standardized data indicators. Second, they solve the problem of how to automate report structure design. By analyzing the relationship between indicators, the system can generate a reasonable report structure without human intervention. Finally, they also solve the problem of how to convert abstract indicator definitions into specific data processing logic, realizing the automated conversion from indicators to reports.

[0147] The technical effects achieved by these two steps are significant. First, they greatly improve the efficiency of report generation. The traditional report generation process usually requires data analysts to manually design report structures, write SQL queries, and configure visualization parameters, while this automated process can complete the same work in seconds. Second, this process improves the accuracy and consistency of reports. Due to the use of standardized indicator definitions, reports generated by different users will use the same indicator caliber and calculation logic, reducing errors caused by misunderstandings. Third, this approach improves the scalability and flexibility of the system. When new business indicators appear, you only need to add new definitions to the indicator library, and the system will automatically incorporate them into the report generation process.

[0148] Let's use a specific example to illustrate the application of these two steps. Suppose we are the data analysis department of a retail company and we receive a request from the sales manager: "I need a report by month and product category, including sales, gross profit margin, and average order value."

[0149] In step S701, the system first parses the request and extracts the following information:

[0150] Target business dimension: Monthly, product category

[0151] Target business indicators: sales, gross profit margin, average order value

[0152] Then, the system will match it in the indicator library. Assume that the matching results are as follows:

[0153] Type indicators: month (corresponding to "monthly"), product category

[0154] Metrics: Sales, gross profit margin, average customer unit price (corresponding to "customer unit price")

[0155] In step S702, the system first analyzes the relationship between the two types of indicators, "month" and "product category". Assume that the system decides to use "month" as the primary dimension (row) and "product category" as the secondary dimension (column) based on historical report usage.

[0156] Next, the system will add calculation logic for each metric. Assume that in the metric library, these metrics are defined as follows:

[0157] Sales: The calculation formula is "sum(order amount)", the aggregation method is "sum", and the display format is "thousand yuan"

[0158] Gross profit margin: The calculation formula is "(sales - cost) / sales", the aggregation method is "weighted average", and the display format is "percentage"

[0159] Average customer unit price: the calculation formula is "sales amount / order quantity", the aggregation method is "average", and the display format is "yuan"

[0160] The system will generate corresponding SQL queries or data processing scripts based on these definitions.

[0161] Finally, the system will organize these query results into a structured report, which may be an interactive data visualization interface that allows users to drill down, filter, and perform other operations.

[0162] Through this process, the system successfully converts the user's natural language request into a complete business report. The entire process does not require human intervention, greatly improving efficiency and accuracy. This automated report generation method not only reduces the workload of data analysts, but also provides business users with faster and more flexible data insights, thereby supporting more data-driven decision-making.

[0163] In order to comprehensively consider multi-dimensional feature evaluation, understand business semantics, and improve the accuracy of data standard construction, the present application provides an embodiment of an automatic data standard construction device based on machine learning for implementing all or part of the content of the automatic data standard construction method based on machine learning, see Figure 8 , the automatic data standard construction device based on machine learning specifically includes the following contents:

[0164] The data feature analysis module 10 is used to obtain the original data features in the historical business data of the enterprise, determine the indicator correlation of each of the original data features according to the linear correlation degree between the original data features and the target business indicators, determine the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at the preset decision tree nodes, and determine the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0165] An indicator model building module 20 is used to determine the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, input the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and perform weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0166] The data standard construction module 30 is used to perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report according to the combination of the type indicators and the measurement indicators.

[0167] From the above description, it can be seen that the automatic data standard construction device based on machine learning provided in the embodiment of the present application can obtain the original data features in the historical business data of the enterprise, determine the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determine the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determine the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; perform business abstraction on the target data through the indicator classification model to obtain atomic indicators; receive data standard construction requests sent by users, and automatically combine and generate corresponding business reports, thereby being able to comprehensively consider multi-dimensional feature evaluations, understand business semantics, and improve the accuracy of data standard construction.

[0168] From the hardware level, in order to comprehensively consider multi-dimensional feature evaluation, understand business semantics, and improve the accuracy of data standard construction, the present application provides an embodiment of an electronic device for implementing all or part of the content of the automatic data standard construction method based on machine learning, and the electronic device specifically includes the following content:

[0169] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the automatic data standard construction device based on machine learning and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the automatic data standard construction method based on machine learning and the embodiment of the automatic data standard construction device based on machine learning in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0170] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0171] In practical applications, part of the automatic data standard construction method based on machine learning can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0172] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0173] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0174] In one embodiment, the automatic data standard construction method function based on machine learning can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0175] Step S101: obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0176] Step S102: determining the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, inputting the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and performing weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0177] Step S103: Perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, where the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report based on the combination of the type indicators and the measurement indicators.

[0178] From the above description, it can be seen that the electronic device provided in the embodiment of the present application obtains the original data features in the historical business data of the enterprise, determines the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determines the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determines the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; performs business abstraction on the target data through the indicator classification model to obtain atomic indicators; receives data standard construction requests sent by users, and automatically combines and generates corresponding business reports, thereby being able to comprehensively consider multi-dimensional feature evaluations, understand business semantics, and improve the accuracy of data standard construction.

[0179] In another embodiment, the automatic data standard construction device based on machine learning can be configured separately from the central processing unit 9100. For example, the automatic data standard construction device based on machine learning can be configured as a chip connected to the central processing unit 9100, and the function of the automatic data standard construction method based on machine learning can be implemented through the control of the central processing unit.

[0180] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0181] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0182] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0183] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0184] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0185] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0186] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0187] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0188] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the method for automatically constructing data standards based on machine learning in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps in the method for automatically constructing data standards based on machine learning in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0189] Step S101: obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0190] Step S102: determining the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, inputting the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and performing weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0191] Step S103: Perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, where the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report based on the combination of the type indicators and the measurement indicators.

[0192] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application obtains the original data features in the historical business data of the enterprise, determines the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determines the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determines the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; performs business abstraction on the target data through the indicator classification model to obtain atomic indicators; receives the data standard construction request sent by the user, and automatically combines and generates the corresponding business report, thereby comprehensively considering multi-dimensional feature evaluation, understanding business semantics, and improving the accuracy of data standard construction.

[0193] The embodiments of the present application also provide a computer program product capable of implementing all the steps in the automatic data standard construction method based on machine learning in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the automatic data standard construction method based on machine learning are implemented. For example, the computer program / instruction implements the following steps:

[0194] Step S101: obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features;

[0195] Step S102: determining the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, inputting the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and performing weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model;

[0196] Step S103: Perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, where the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report based on the combination of the type indicators and the measurement indicators.

[0197] From the above description, it can be seen that the computer program product provided by the embodiment of the present application obtains the original data features in the historical business data of the enterprise, determines the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicators, determines the feature importance of each original data feature according to the Gini index gain generated when the original data features are split at the preset decision tree node, and determines the principal component contribution of each original data feature according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; performs business abstraction on the target data through the indicator classification model to obtain atomic indicators; receives data standard construction requests sent by users, and automatically combines and generates corresponding business reports, thereby being able to comprehensively consider multi-dimensional feature evaluation, understand business semantics, and improve the accuracy of data standard construction.

[0198] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0199] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0202] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for constructing automatic data standards based on machine learning, characterized in that: The method comprises: Obtaining original data features in the enterprise's historical business data, determining the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at a preset decision tree node, and determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; According to the indicator correlation, the feature importance and the principal component contribution, the key factor features in the original data features are determined, the original data features and the key factor features are input into the input layer of the multi-layer perceptron model for model training, and the weights are tuned by calculating the gradient mean and the gradient uncentered variance in the model training process until the model convergence condition is reached, thereby obtaining an indicator classification model; The target data is business abstracted through the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators; a data standard construction request sent by a user is received, and the corresponding type indicators and measurement indicators are determined according to the target business indicators and target business dimensions in the data standard construction request, and a corresponding business report is generated according to the combination of the type indicators and the measurement indicators.

2. The method for constructing automatic data standards based on machine learning according to claim 1, characterized in that: The acquiring of original data features in the historical business data of the enterprise and determining the indicator correlation of each original data feature according to the linear correlation degree between the original data features and the target business indicator include: Obtain the company's historical business data and perform feature engineering to obtain the original data features; The Pearson correlation coefficient between the original data features and the target business indicators is calculated to determine the indicator correlation of each of the original data features.

3. The method for constructing automatic data standards based on machine learning according to claim 1, characterized in that: Determining the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at the preset decision tree nodes includes: Building a decision tree based on the original data features, evaluating all possible feature splits at each node of the decision tree, and calculating the Gini impurity before and after the split to obtain the Gini index gain generated during the split; The feature importance of each of the original data features is determined according to the accumulated value of the Gini index gain generated during the splitting.

4. The method for constructing automatic data standards based on machine learning according to claim 1, characterized in that: Determining the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features includes: Standardizing the original data features to obtain corresponding standardized features, calculating the covariance matrix between the standardized features and performing eigenvalue decomposition on the covariance matrix to obtain corresponding eigenvalues ​​and eigenvectors; Calculate the proportion of each eigenvalue to the total eigenvalues ​​to obtain the variance proportion explained by each principal component, calculate the load corresponding to the eigenvector, square the load and multiply it by the variance proportion of each principal component to obtain the weighted square load, and for each original feature, calculate the sum of its weighted square loads on all principal components to determine the principal component contribution of each original data feature.

5. The method for automatically constructing data standards based on machine learning according to claim 1, characterized in that: Determining the key factor features in the original data features according to the indicator correlation, the feature importance and the principal component contribution includes: After normalizing the index correlation, the feature importance and the principal component contribution, corresponding weights are assigned through a cross-validation method, and a comprehensive score of each of the original data features is calculated; The original data features are sorted in descending order according to the comprehensive scores to determine key factor features in the original data features.

6. The method for constructing automatic data standards based on machine learning according to claim 1, characterized in that: The target data is business abstracted by the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators, including: Input each indicator in the target data into the indicator classification model to obtain a classification result output by the indicator classification model, wherein the classification result includes a type indicator and a measurement indicator; Perform semantic analysis on the type indicators and measurement indicators to extract the core business meaning and calculation logic of the indicators. For atomic indicators classified as type indicators, determine their possible value ranges and business rules; for atomic indicators classified as measurement indicators, determine their measurement units, aggregation methods and calculation formulas.

7. The method for constructing automatic data standards based on machine learning according to claim 1, characterized in that: The receiving a data standard construction request sent by a user, determining a corresponding type indicator and a metric indicator according to a target business indicator and a target business dimension in the data standard construction request, and generating a corresponding business report according to a combination of the type indicator and the metric indicator, includes: Receive and parse the data standard construction request sent by the user, extract the target business indicators and target business dimension information, match the target business indicators with the measurement indicators, and match the target business dimensions with the type indicators according to the pre-established indicator library; Determine the report's hierarchy and layout based on the logical relationship between indicators of different types. Add calculation logic to the report based on the calculation formula, aggregation method, and display format of the metrics to generate the corresponding business report.

8. An automatic data standard construction device based on machine learning, characterized in that: The device comprises: A data feature analysis module is used to obtain the original data features in the enterprise's historical business data, determine the indicator correlation of each of the original data features according to the linear correlation between the original data features and the target business indicators, determine the feature importance of each of the original data features according to the Gini index gain generated when the original data features are split at the preset decision tree nodes, and determine the principal component contribution of each of the original data features according to the eigenvalues ​​and eigenvectors of the covariance matrix corresponding to the original data features; An indicator model building module is used to determine the key factor features in the original data features according to the indicator relevance, the feature importance and the principal component contribution, input the original data features and the key factor features into the input layer of the multi-layer perceptron model for model training, and perform weight tuning by calculating the gradient mean and the gradient uncentered variance during the model training process until the model convergence condition is met, thereby obtaining an indicator classification model; The data standard construction module is used to perform business abstraction on the target data through the indicator classification model to obtain atomic indicators, wherein the atomic indicators include type indicators and measurement indicators; receive a data standard construction request sent by a user, determine the corresponding type indicators and measurement indicators according to the target business indicators and target business dimensions in the data standard construction request, and generate a corresponding business report according to the combination of the type indicators and the measurement indicators.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the automatic data standard construction method based on machine learning described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the automatic data standard construction method based on machine learning described in any one of claims 1 to 7 are implemented.

Images