Automatic data analysis system and method based on artificial intelligence

Abstract

The invention discloses an automatic data analysis system and method based on artificial intelligence, and relates to the technical field of data management, and the method comprises the following steps: S1, data collection and preparation; s2, data annotation and classification; s3, feature selection and extraction; s4, data standardization and normalization are carried out; s5, model selection; s6, training and optimizing the model; s7, performing model evaluation; s8, model interpretation and visualization are carried out; s9, generating an automatic report; and the system is constructed based on the steps. According to the automatic data analysis system and method based on artificial intelligence, by setting a classification and labeling model, a training optimization mechanism and a dynamic model evaluation method, the efficiency and precision of automatic data analysis are remarkably improved, and the quality of data preprocessing is improved; meanwhile, through self-adaptive hyper-parameter adjustment and optimization, the efficiency of the training process and the model performance are optimized; the dynamic evaluation mechanism ensures that the model is more accurate in performance under different data characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to an automatic data analysis system and method based on artificial intelligence. Background Art

[0002] With the rapid development of big data and artificial intelligence, automated data analysis systems and methods have been widely used in many industries. Traditional data analysis systems and methods rely on manual labeling, model selection, and model optimization, which are inefficient and have poor adaptability to complex and diverse data. With the continuous increase in data volume and data type, the efficiency bottleneck of manual intervention has become more and more obvious. In order to improve the efficiency and accuracy of analysis, automated data analysis systems and methods based on artificial intelligence have gradually become a research focus. Through automated data labeling, classification, training optimization, and evaluation processes, manual intervention can be significantly reduced, and data processing speed and model performance can be improved.

[0003] The disadvantages of the prior art are as follows:

[0004] Unintelligent labeling and classification models: Existing automated data analysis systems and methods often rely on predefined rules or manual labeling and classification processes. Although some systems have applied machine learning models for preliminary labeling and classification, the accuracy and generalization capabilities of these models are often insufficient, making it difficult to handle diverse labels and categories in complex, heterogeneous data sets.

[0005] Inefficient training optimization methods: Traditional model training processes usually use static methods such as grid search or random search for hyperparameter tuning, which are time-consuming and difficult to dynamically adjust model parameters based on data characteristics. As the size and complexity of data sets increase, traditional optimization methods show obvious limitations in model performance and efficiency.

[0006] Evaluation methods lack flexibility: Current model evaluation methods, such as standard cross-validation or fixed evaluation indicators (such as precision, F1 score), are difficult to dynamically adapt to the needs of different scenarios and data characteristics. In particular, for class imbalance or time series data, traditional evaluation methods cannot fully reflect the true performance of the model, resulting in insufficient evaluation of generalization ability. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides an automated data analysis system and method based on artificial intelligence to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: an automated data analysis method based on artificial intelligence, comprising the following steps:

[0009] S1. Data collection and preparation

[0010] In the process of automated data analysis, data is collected based on multiple channels and the collected data is cleaned;

[0011] S2. Data labeling and classification

[0012] Classify or label data based on artificial intelligence models. Data classification is used to learn and predict subsequent models, and data labeling is used to evaluate the performance and iteratively optimize subsequent models.

[0013] S3. Feature selection and extraction

[0014] Through feature selection and feature extraction, statistical methods are used to reduce the dimension of data, new features are generated through feature interaction and aggregation, data is converted into features suitable for model use, and features are extracted;

[0015] S4. Data standardization and normalization

[0016] Standardize and normalize the data to convert data of different scales to the same scale to avoid weight imbalance between features;

[0017] S5. Model selection

[0018] Select the type of artificial intelligence model based on the data type and analysis objectives. The types of artificial intelligence models include decision trees, random forests, support vector machines, and neural networks. When selecting a model, evaluate the performance of different models through cross-validation methods to select the most appropriate model.

[0019] S6. Model training and optimization

[0020] Divide the data into training sets and test sets, fit the model through the training set, optimize the model performance by adjusting the hyperparameters, and maximize the prediction accuracy of the model through the tuning process;

[0021] S7. Model evaluation

[0022] Use the test set to evaluate the model. The evaluation indicators include accuracy, precision, recall, F1 score, and mean square error. The cross-validation method is used to evaluate the model.

[0023] S8. Model interpretation and visualization

[0024] Use LIME and SHAP tools to explain the prediction results of the model, and display them through visualization components;

[0025] S9. Automated report generation

[0026] Automatically generate analysis reports, which include data description, reasons for model selection, feature selection and extraction, evaluation results of model performance, and explanations and suggestions for analysis results.

[0027] To further optimize the technical solution, in step S1, the data comes from multiple channels, such as a database, an external API, a file (CSV, JSON), or a web crawl;

[0028] When cleaning the collected data, it includes removing duplicate data, processing missing values, and converting data formats.

[0029] To further optimize the technical solution, in step S2, the artificial intelligence model is divided into a classification model and a labeling model;

[0030] The classification model classifies data based on an adaptive hierarchical neural network, including:

[0031] The input data is ( ),in( ) is the dimension of the data;

[0032] The output of the hidden layer is ( ),in( ) is the first ( ) layer weight matrix, ( ) is the bias, ( ) is the activation function, such as ReLU, Sigmoid;

[0033] The final output of the model is the classification probability ( ),in( ) is the weight matrix of the output layer, ( ) is the first ( ) layer’s output, ( ) is the output bias;

[0034] Set an adaptive parameter ( ), used to dynamically adjust the number of layers and nodes in the network:

[0035]

[0036] in,( ) is the input data ( ), according to ( )'s size dynamically adjusts the number of layers ( ) and the number of nodes ( ):

[0037] if( ) is large, indicating that the data complexity is high, and the number of hidden layers is automatically increased ( ) and the number of nodes ( );

[0038] if( ) is small, reducing the number of layers and nodes to avoid overfitting;

[0039] The annotation model classifies data based on weighted hierarchical annotation, including:

[0040] Assume that the input data is text or image data, and the label set is ( ), each label corresponds to a level in a hierarchy;

[0041] The feature representation of the input data is ( );

[0042] The probability distribution of the label is ( ),in( ) is the weight matrix corresponding to the label, ( ) is the bias term;

[0043] Based on the weighted mechanism, the priority of different tags is strengthened, and the weight ( ) is used to measure the ) tags’ priorities:

[0044]

[0045] in,( ) represents the label ( ), higher-level labels have larger weights, which is used to ensure that high-level labels have higher priority in complex annotation scenarios.

[0046] To further optimize the technical solution, in step S3, different feature types are used for extraction for different types of data, and the data types and feature types include:

[0047] Time series data: extract trends, seasonality, stationarity, autocorrelation features, frequency domain features, and time difference features;

[0048] Categorical data: Use one-hot encoding, target encoding, frequency encoding, and embedding features to convert discrete data into numerical form;

[0049] Numerical data: extract basic statistical features, perform normalization or standardization, generate polynomial features, binning, and differential features;

[0050] Text data: using term frequency-inverse document frequency (TF-IDF), bag-of-words model, word embedding, sentiment features, and topic model features;

[0051] Image data: extract pixel values, edge detection, color histogram, texture features, convolutional neural network (CNN) features;

[0052] Network data: extraction degree, centrality, clustering coefficient, path characteristics.

[0053] Further optimizing the technical solution, in step S6, during the training process, a gradient descent algorithm is used to optimize the model parameters, and a gradient descent algorithm based on an adaptive learning rate automatically adjusts the learning rate according to the gradient change;

[0054] The dynamic regularization model is used to optimize the model performance by adjusting hyperparameters such as learning rate and regularization coefficient, by sparse model parameters and by introducing time-dependent regularization coefficient ( ), so that the regularization can be dynamically adjusted according to the data characteristics during the training process.

[0055] To further optimize the technical solution, in the gradient descent algorithm:

[0056] Assume the loss function is ( ),in( ) is the weight vector of the model. The goal of the model is to minimize the loss function. The standard gradient descent formula is as follows:

[0057]

[0058] In the algorithm, an adaptive learning rate is introduced ( ), and its update rule is:

[0059]

[0060] in,( ) is a hyperparameter for adjusting the learning rate, ( ) is the norm of the current gradient; when the gradient is large, the learning rate will automatically decrease to prevent the model from falling into oscillation; when the gradient is small, the learning rate will automatically increase to speed up the convergence.

[0061] To further optimize the technical solution, in the dynamic regularization model,

[0062] Assume that the loss function of the model is ( ), and introduce the Laplace regularization term ( ), whose optimization goal is to minimize the following objective function:

[0063]

[0064] in,( ) is the dynamically adjusted regularization coefficient, and its update rule is:

[0065]

[0066] in,( ) is a hyperparameter for adjusting the regularization coefficient, ( ) represents the rate of weight change;

[0067] Through dynamic regularization, a larger regularization is applied in the early stage of training to suppress overfitting. As training progresses, the impact of regularization is reduced, so that the model can maximize its attention to complex features.

[0068] To further optimize the technical solution, in step S7, when the model evaluation is performed using a cross-validation method, the method includes:

[0069] The data is dynamically divided into several subsets, each of which is based not only on the independence of the samples but also on the distribution trend of the samples. Assume that there are ( ) samples, and dynamically adjust the validation set weight according to the following formula:

[0070]

[0071] in,( ) is the first ( ) verification weight, ( ) is the adjustment factor, ( ) represents the data in the ( )Distribution trend of folds;

[0072] If the trend of adjacent validation sets changes significantly, the weight is reduced to ensure that the model can flexibly respond to changes when validating data with different trends.

[0073] To further optimize the technical solution, in step S8, the visualization component has a built-in visualization display model, which displays the decision path of the model through the visualization of the decision tree, or displays the contribution of the features to the model results through a heat map.

[0074] An artificial intelligence-based automated data analysis system is constructed based on the above-mentioned artificial intelligence-based automated data analysis method, including a data collection module, a labeling and classification module, a feature engineering module, a data processing module, a model selection module, a training and optimization module, a model evaluation module, a visualization module, and a report generation module;

[0075] The system first obtains data through the data collection module, and the data is preliminarily labeled and classified through the labeling and classification module;

[0076] The feature engineering module extracts key features, and then the data processing module standardizes and normalizes the data. The processed data is transmitted to the model selection module to select the most appropriate model according to the task type;

[0077] The model enters the training optimization module for training, and the model performance is improved through parameter adjustment and optimization technology. The trained model is evaluated and tested in the model evaluation module;

[0078] The visualization module generates charts and data visualization results to help users understand the analysis process, and the report generation module organizes the entire analysis process into detailed automated reports to support business decisions and strategic planning.

[0079] Compared with the prior art, the present invention provides an automated data analysis system and method based on artificial intelligence, which has the following beneficial effects:

[0080] This AI-based automated data analysis system and method significantly improves the efficiency and accuracy of automated data analysis by setting classification and labeling models, adaptive training optimization mechanisms, and dynamic model evaluation methods. It can adaptively label and classify complex and heterogeneous data sets, improving the quality of data preprocessing. At the same time, it optimizes the efficiency of the training process and model performance through adaptive hyperparameter tuning. The dynamic evaluation mechanism ensures that the model performs more accurately under different data characteristics. Ultimately, the system not only reduces manual intervention, but also improves the accuracy and speed of overall analysis, providing strong support for large-scale data processing and intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A schematic diagram of a process flow of an automated data analysis method based on artificial intelligence proposed by the present invention;

[0082] Figure 2 A classification diagram of an artificial intelligence model in an artificial intelligence-based automated data analysis method proposed by the present invention;

[0083] Figure 3 A classification diagram of data types in an automated data analysis method based on artificial intelligence proposed by the present invention;

[0084] Figure 4 A schematic diagram of the process of model training and optimization in an automated data analysis method based on artificial intelligence proposed by the present invention;

[0085] Figure 5 A schematic diagram of a cross-validation process in an automated data analysis method based on artificial intelligence proposed by the present invention;

[0086] Figure 6 This is a schematic diagram of the structure of an automated data analysis system based on artificial intelligence proposed by the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Embodiment 1:

[0089] See also Figure 1 , an automated data analysis method based on artificial intelligence, comprising the following steps:

[0090] S1. Data collection and preparation

[0091] In the process of automated data analysis, data is collected based on multiple channels and the collected data is cleaned.

[0092] In this example, the data comes from multiple channels, such as databases, external APIs, files (CSV, JSON), or web crawling. When collecting data, the reliability of the data source and the diversity of the data should be ensured to ensure the accuracy and representativeness of the analysis results.

[0093] When cleaning the collected data, it includes removing duplicate data, processing missing values, and converting data formats.

[0094] S2. Data labeling and classification

[0095] Data is classified or labeled based on artificial intelligence models. Data classification is used to learn and predict subsequent models, and data labeling is used to evaluate the performance and iteratively optimize subsequent models.

[0096] like Figure 2 As shown, in this embodiment, the artificial intelligence model is divided into a classification model and a labeling model;

[0097] The classification model classifies data based on an adaptive hierarchical neural network, including:

[0098] The input data is ( ),in( ) is the dimension of the data;

[0099] The output of the hidden layer is ( ),in( ) is the first ( ) layer weight matrix, ( ) is the bias, ( ) is the activation function, such as ReLU, Sigmoid;

[0100] The final output of the model is the classification probability ( ),in( ) is the weight matrix of the output layer, ( ) is the first ( ) layer’s output, ( ) is the output bias;

[0101] Set an adaptive parameter ( ), used to dynamically adjust the number of layers and nodes in the network:

[0102]

[0103] in,( ) is the input data ( ), according to ( )'s size dynamically adjusts the number of layers ( ) and the number of nodes ( ):

[0104] if( ) is large, indicating that the data complexity is high, and the number of hidden layers is automatically increased ( ) and the number of nodes ( );

[0105] if( ) is smaller, and the number of layers and nodes is reduced to avoid overfitting.

[0106] The classification model is used when:

[0107] Data input: The model requires input data to be preprocessed and standardized to ensure that the data characteristics are within a reasonable range.

[0108] Model training: Through forward propagation and back propagation, adjust the weight matrix ( )and( ), and according to the adaptive parameters ( ) Dynamically adjust the number of layers. The loss function of the model can use the cross entropy loss function and use the gradient descent method for optimization.

[0109] Model evaluation: The model is verified through the test set, and the classification effect is evaluated using indicators such as accuracy and recall.

[0110] The annotation model classifies data based on weighted hierarchical annotation, including:

[0111] Assume that the input data is text or image data, and the label set is ( ), each label corresponds to a level in a hierarchy;

[0112] The feature representation of the input data is ( );

[0113] The probability distribution of the label is ( ),in( ) is the weight matrix corresponding to the label, ( ) is the bias term;

[0114] Based on the weighted mechanism, the priority of different tags is strengthened, and the weight ( ) is used to measure the ) tags’ priorities:

[0115]

[0116] in,( ) represents the label ( ), higher-level labels have larger weights, which is used to ensure that high-level labels have higher priority in complex annotation scenarios.

[0117] When using this annotation model:

[0118] Building a hierarchical labeling system: The labeling system in business scenarios is usually multi-level. Before labeling, you need to set the hierarchy and weights for these labels.

[0119] Data annotation: Input data ( ) After feature extraction, it is input into the model for annotation. The model uses a weighted mechanism to prioritize high-level labels according to the hierarchical structure, and then performs refined annotation.

[0120] Weight optimization: During the training process, the model will dynamically adjust the weights ( ), increase or decrease the weight of specific tags according to the actual annotation effect, so as to optimize the accuracy and hierarchy of annotation.

[0121] S3. Feature selection and extraction

[0122] Through feature selection and feature extraction, statistical methods are used to reduce the dimension of the data, new features are generated through feature interaction and aggregation, the data is converted into features suitable for model use, and the features are extracted.

[0123] like Figure 3 As shown, in this embodiment, different feature types are used for extraction for different types of data, and the data types and feature types include:

[0124] Time series data: extract trends, seasonality, stationarity, autocorrelation features, frequency domain features, and time difference features;

[0125] Categorical data: Use one-hot encoding, target encoding, frequency encoding, and embedding features to convert discrete data into numerical form;

[0126] Numerical data: extract basic statistical features, perform normalization or standardization, generate polynomial features, binning, and differential features;

[0127] Text data: using term frequency-inverse document frequency (TF-IDF), bag-of-words model, word embedding, sentiment features, and topic model features;

[0128] Image data: extract pixel values, edge detection, color histogram, texture features, convolutional neural network (CNN) features;

[0129] Network data: extraction degree, centrality, clustering coefficient, path characteristics.

[0130] S4. Data standardization and normalization

[0131] Standardize and normalize the data to convert data of different scales to the same scale to avoid weight imbalance between features.

[0132] In this embodiment, different types of data may have different scales, which may affect the training effect of the model. For example, some data values ​​have a large range, while other data values ​​may be small, and the model will tend to learn data features with a larger value range. Ensure that the model can treat each feature fairly and improve the learning efficiency and prediction accuracy of the model.

[0133] S5. Model selection

[0134] According to the type of data and analysis objectives, select the type of artificial intelligence model, which includes decision tree, random forest, support vector machine, and neural network. When selecting a model, use the cross-validation method to evaluate the performance of different models and select the most appropriate model.

[0135] In this embodiment, each model is suitable for different scenarios: if the target is a classification task, logistic regression or support vector machine can be selected; if the target is a regression task, linear regression or gradient boosting tree can be selected. When selecting a model, the interpretability, computational complexity and generalization ability of the model should also be considered.

[0136] S6. Model training and optimization

[0137] The data is divided into training set and test set, the model is fitted through the training set, the model performance is optimized by adjusting the hyperparameters, and the prediction accuracy of the model is maximized through the tuning process.

[0138] like Figure 4 As shown, in this embodiment, during the training process, a gradient descent algorithm is used to optimize the model parameters, and a gradient descent algorithm based on an adaptive learning rate automatically adjusts the learning rate according to the gradient change;

[0139] The dynamic regularization model is used to optimize the model performance by adjusting hyperparameters such as learning rate and regularization coefficient, by sparse model parameters and by introducing time-dependent regularization coefficient ( ), so that the regularization can be dynamically adjusted according to the data characteristics during the training process.

[0140] In the gradient descent algorithm:

[0141] Assume the loss function is ( ),in( ) is the weight vector of the model. The goal of the model is to minimize the loss function. The standard gradient descent formula is as follows:

[0142]

[0143] In the algorithm, an adaptive learning rate is introduced ( ), and its update rule is:

[0144]

[0145] in,( ) is a hyperparameter for adjusting the learning rate, ( ) is the norm of the current gradient; when the gradient is large, the learning rate will automatically decrease to prevent the model from falling into oscillation; when the gradient is small, the learning rate will automatically increase to speed up the convergence.

[0146] Instructions

[0147] Training set allocation: The data is divided into training set and test set, the model is fitted on the training set, and the weights are updated by gradient descent.

[0148] Optimization process: The model is optimized by adaptive learning rate ( ), ensuring that the model maintains good convergence in different training stages.

[0149] Model fitting: Through repeated iterations, the model will converge near the global or local optimal point, ensuring that the model can effectively capture the patterns in the data.

[0150] In the dynamic regularization model,

[0151] Assume that the loss function of the model is ( ), and introduce the Laplace regularization term ( ), whose optimization goal is to minimize the following objective function:

[0152]

[0153] in,( ) is the dynamically adjusted regularization coefficient, and its update rule is:

[0154]

[0155] in,( ) is a hyperparameter for adjusting the regularization coefficient, ( ) represents the rate of weight change;

[0156] Through dynamic regularization, a larger regularization is applied in the early stage of training to suppress overfitting. As training progresses, the impact of regularization is reduced, so that the model can maximize its attention to complex features.

[0157] Instructions

[0158] Dynamic adjustment during training: The model is adjusted in the early stages of training by using a larger regularization coefficient ( ) to suppress complex features. As training progresses, ( ) decreases dynamically, allowing the model to gradually focus on subtle features in the data.

[0159] Prevent overfitting: By dynamically adjusting the regularization term, the model can be prevented from overfitting the training data, while striking a balance between the generalization ability and complexity of the model.

[0160] Weight sparsification: Regularization helps to sparse the weights of the model so that the model only depends on a small number of important features. This is particularly useful in high-dimensional data (such as text classification, image recognition, etc.).

[0161] S7. Model evaluation

[0162] The test set is used to evaluate the model. The evaluation indicators include accuracy, precision, recall, F1 score, and mean square error. The cross-validation method is used to evaluate the model.

[0163] like Figure 5 As shown, in this embodiment, when the cross-validation method is used to evaluate the model, it includes:

[0164] The data is dynamically divided into several subsets, each of which is based not only on the independence of the samples but also on the distribution trend of the samples. Assume that there are ( ) samples, and dynamically adjust the validation set weight according to the following formula:

[0165]

[0166] in,( ) is the first ( ) verification weight, ( ) is the adjustment factor, ( ) represents the data in the ( )Distribution trend of folds;

[0167] If the trend of adjacent validation sets changes significantly, the weight is reduced to ensure that the model can flexibly respond to changes when validating data with different trends.

[0168] Instructions

[0169] Dynamic data partitioning: Divide the data set into multiple subsets based on the data trend or category distribution. Each fold must not only ensure data independence, but also consider the change pattern in the data.

[0170] Weight adjustment: Through the above formula, the weights of the samples in the validation set are dynamically adjusted to ensure that the validation set represents the diversity of the data.

[0171] Verification process: The model is trained on each subset and the corresponding evaluation indicators (such as accuracy, precision, etc.) are calculated. Finally, the model is trained on each subset and the corresponding evaluation indicators (such as accuracy, precision, etc.) are calculated. ) Comprehensively calculate the overall evaluation results.

[0172] S8. Model interpretation and visualization

[0173] The prediction results of the model are explained through LIME and SHAP tools, and the prediction results are presented through visualization components.

[0174] In this embodiment, the visualization component has a built-in visualization display model, which displays the decision path of the model through the visualization of the decision tree, or displays the contribution of the features to the model results through the heat map.

[0175] S9. Automated report generation

[0176] Automatically generate analysis reports, which include data description, reasons for model selection, feature selection and extraction, evaluation results of model performance, and explanations and suggestions for analysis results.

[0177] In this embodiment, customized reports can also be generated according to user needs. By integrating natural language generation (NLG) technology, report writing can become more flexible and personalized. The automated system can provide both concise and detailed options when generating reports to meet different levels of user needs.

[0178] Embodiment 2:

[0179] See also Figure 6 An artificial intelligence-based automated data analysis system is constructed based on the artificial intelligence-based automated data analysis method described in Example 1, and includes a data collection module, a labeling and classification module, a feature engineering module, a data processing module, a model selection module, a training and optimization module, a model evaluation module, a visualization module, and a report generation module.

[0180] The system first obtains data through the data collection module, and the data is preliminarily labeled and classified through the labeling and classification module;

[0181] The feature engineering module extracts key features, and then the data processing module standardizes and normalizes the data. The processed data is transmitted to the model selection module to select the most appropriate model according to the task type;

[0182] The model enters the training optimization module for training, and the model performance is improved through parameter adjustment and optimization technology. The trained model is evaluated and tested in the model evaluation module;

[0183] The visualization module generates charts and data visualization results to help users understand the analysis process, and the report generation module organizes the entire analysis process into detailed automated reports to support business decisions and strategic planning.

[0184] In this embodiment, after the system is deployed, it is still necessary to continuously monitor the performance of the model and update the model in a timely manner to cope with new data and changes in business needs. In order to ensure the stability and accuracy of the model, the system can also regularly perform incremental training on new data to maintain the timeliness of the model. In addition, maintenance work also includes anomaly detection and alarm mechanisms to ensure the robustness of the system during operation.

[0185] The beneficial effects of the present invention are:

[0186] This AI-based automated data analysis system and method significantly improves the efficiency and accuracy of automated data analysis by setting classification and labeling models, adaptive training optimization mechanisms, and dynamic model evaluation methods. It can adaptively label and classify complex and heterogeneous data sets, improving the quality of data preprocessing. At the same time, it optimizes the efficiency of the training process and model performance through adaptive hyperparameter tuning. The dynamic evaluation mechanism ensures that the model performs more accurately under different data characteristics. Ultimately, the system not only reduces manual intervention, but also improves the accuracy and speed of overall analysis, providing strong support for large-scale data processing and intelligent decision-making.

[0187] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0188] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automated data analysis method based on artificial intelligence, characterized in that: The following steps are involved: S1. Data collection and preparation In the process of automated data analysis, data is collected based on multiple channels and the collected data is cleaned; S2. Data labeling and classification Classify or label data based on artificial intelligence models. Data classification is used to learn and predict subsequent models, and data labeling is used to evaluate the performance and iteratively optimize subsequent models. S3. Feature selection and extraction Through feature selection and feature extraction, statistical methods are used to reduce the dimension of data, new features are generated through feature interaction and aggregation, data is converted into features suitable for model use, and features are extracted; S4. Data standardization and normalization Standardize and normalize the data to convert data of different scales to the same scale to avoid weight imbalance between features; S5. Model selection According to the type of data and analysis objectives, select the type of artificial intelligence model, which includes decision tree, random forest, support vector machine, and neural network. When selecting a model, use the cross-validation method to evaluate the performance of different models and select the most appropriate model. S6. Model training and optimization Divide the data into training sets and test sets, fit the model through the training set, optimize the model performance by adjusting the hyperparameters, and maximize the prediction accuracy of the model through the tuning process; S7. Model evaluation Use the test set to evaluate the model. The evaluation indicators include accuracy, precision, recall, F1 score, and mean square error. The cross-validation method is used to evaluate the model. S8. Model interpretation and visualization Use LIME and SHAP tools to explain the prediction results of the model, and display them through visualization components; S9. Automated report generation Automatically generate analysis reports, which include data description, reasons for model selection, feature selection and extraction, evaluation results of model performance, and explanations and suggestions for analysis results.

2. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S1, the data comes from multiple channels, such as databases, external APIs, files (CSV, JSON), or web crawling; When cleaning the collected data, it includes removing duplicate data, processing missing values, and converting data formats.

3. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S2, the artificial intelligence model is divided into a classification model and a labeling model; The classification model classifies data based on an adaptive hierarchical neural network, including: The input data is ( ),in( ) is the dimension of the data; The output of the hidden layer is ( ),in( ) is the first ( ) layer weight matrix, ( ) is the bias, ( ) is the activation function, such as ReLU, Sigmoid; The final output of the model is the classification probability ( ),in( ) is the weight matrix of the output layer, ( ) is the first ( ) layer’s output, ( ) is the output bias; Set an adaptive parameter ( ), used to dynamically adjust the number of layers and nodes in the network: ； in,( ) is the input data ( ), according to ( )'s size dynamically adjusts the number of layers ( ) and the number of nodes ( ): if( ) is large, indicating that the data complexity is high, and the number of hidden layers is automatically increased ( ) and the number of nodes ( ); if( ) is small, reducing the number of layers and nodes to avoid overfitting; The annotation model classifies data based on weighted hierarchical annotation, including: Assume that the input data is text or image data, and the label set is ( ), each label corresponds to a level in a hierarchy; The feature representation of the input data is ( ); The probability distribution of the label is ( ),in( ) is the weight matrix corresponding to the label, ( ) is the bias term; Based on the weighted mechanism, the priority of different tags is strengthened, and the weight ( ) is used to measure the ) tags’ priorities: ； in,( ) represents the label ( ), higher-level labels have larger weights, which is used to ensure that high-level labels have higher priority in complex annotation scenarios.

4. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S3, different feature types are used for extraction for different types of data. The data types and feature types include: Time series data: extract trends, seasonality, stationarity, autocorrelation features, frequency domain features, and time difference features; Categorical data: Use one-hot encoding, target encoding, frequency encoding, and embedding features to convert discrete data into numerical form; Numerical data: extract basic statistical features, perform normalization or standardization, generate polynomial features, binning, and differential features; Text data: using term frequency-inverse document frequency (TF-IDF), bag-of-words model, word embedding, sentiment features, and topic model features; Image data: extract pixel values, edge detection, color histogram, texture features, convolutional neural network (CNN) features; Network data: extraction degree, centrality, clustering coefficient, path characteristics.

5. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S6, during the training process, the model parameters are optimized and trained using a gradient descent algorithm, and the learning rate is automatically adjusted according to the gradient change based on the gradient descent algorithm with an adaptive learning rate; The dynamic regularization model is used to optimize the model performance by adjusting hyperparameters such as learning rate and regularization coefficient. By sparse model parameters, a time-dependent regularization coefficient is introduced ( ), so that the regularization can be dynamically adjusted according to the data characteristics during the training process.

6. The method of automatic data analysis based on artificial intelligence according to claim 5, characterized in that: In the gradient descent algorithm: Assume the loss function is ( ),in( ) is the weight vector of the model. The goal of the model is to minimize the loss function. The standard gradient descent formula is as follows: ； In the algorithm, an adaptive learning rate is introduced ( ), and its update rule is: ； in,( ) is a hyperparameter for adjusting the learning rate, ( ) is the norm of the current gradient; when the gradient is large, the learning rate will automatically decrease to prevent the model from falling into oscillation; when the gradient is small, the learning rate will automatically increase to speed up the convergence.

7. The method of automatic data analysis based on artificial intelligence according to claim 5, characterized in that: In the dynamic regularization model, Assume that the loss function of the model is ( ), and introduce the Laplace regularization term ( ), whose optimization goal is to minimize the following objective function: ； in,( ) is the dynamically adjusted regularization coefficient, and its update rule is: ； in,( ) is a hyperparameter for adjusting the regularization coefficient, ( ) represents the rate of weight change; Through dynamic regularization, a larger regularization is applied in the early stage of training to suppress overfitting. As training progresses, the impact of regularization is reduced, so that the model can maximize its attention to complex features.

8. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S7, when the model evaluation is performed using a cross-validation method, the following steps are performed: The data is dynamically divided into several subsets, each of which is based not only on the independence of the samples but also on the distribution trend of the samples. Assume that there are ( ) samples, and dynamically adjust the validation set weight according to the following formula: ； in,( ) is the first ( ) verification weight, ( ) is the adjustment factor, ( ) represents the data in the ( )Distribution trend of folds; If the trend of adjacent validation sets changes significantly, the weight is reduced to ensure that the model can flexibly respond to changes when validating data with different trends.

9. The method of automatic data analysis based on artificial intelligence according to claim 1, characterized in that: In step S8, the visualization component has a built-in visualization display model, which displays the decision path of the model through the visualization of the decision tree, or displays the contribution of the features to the model results through the heat map.

10. An artificial intelligence-based automated data analysis system, constructed based on the artificial intelligence-based automated data analysis method according to any one of claims 1 to 9, characterized in that: It includes data collection module, labeling and classification module, feature engineering module, data processing module, model selection module, training optimization module, model evaluation module, visualization module, and report generation module; The system first obtains data through the data collection module, and the data is preliminarily labeled and classified through the labeling and classification module; The feature engineering module extracts key features, and then the data processing module standardizes and normalizes the data. The processed data is transmitted to the model selection module to select the most appropriate model according to the task type; The model enters the training optimization module for training, and the model performance is improved through parameter adjustment and optimization technology. The trained model is evaluated and tested in the model evaluation module; The visualization module generates charts and data visualization results to help users understand the analysis process, and the report generation module organizes the entire analysis process into detailed automated reports to support business decisions and strategic planning.

Images