Automatic data analysis and attribution analysis method and device and intelligent terminal

Abstract

The invention discloses an automatic data analysis and attribution analysis method and device and an intelligent terminal, and the method comprises the steps: collecting and obtaining data needing to be analyzed, carrying out the preprocessing of the obtained data, and obtaining the preprocessed data; selecting and loading a preset training model according to the task type and the performance index of the data needing to be analyzed; inputting the preprocessed data into the preset training model, performing data feature extraction, and extracting feature data representing data features; performing automatic analysis on the extracted feature data to generate a prediction or classification result; and automatically analyzing the generated prediction or classification result by applying an attribution method, and identifying key factors influencing the result. According to the method, a pre-trained large model is used for automatically executing a data analysis task, and an efficient data attribution analysis means is provided, so that key factors influencing a result are accurately identified.

Description

Technical Field

[0001] The present invention relates to the field of intelligent terminal technology, and in particular to an automated data analysis and attribution analysis method, device, intelligent terminal and storage medium. Background Art

[0002] In the field of data analysis, traditional analysis methods usually rely on manual feature extraction and model training, which are time-consuming and inefficient. With the development of large-scale machine learning models, automated analysis has become possible, but existing technologies lack effective data attribution analysis methods and it is difficult to explain the decision-making process of the model. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an automated data analysis and attribution analysis method, device, intelligent terminal and storage medium in response to the problems and defects of the above-mentioned prior art. The present invention uses a pre-trained large-scale model to automatically perform data analysis tasks and provides an efficient data attribution analysis method to accurately identify the key factors that affect the results.

[0004] The technical solution adopted by the present invention to solve the problem is as follows:

[0005] An automated data analysis and attribution analysis method, comprising:

[0006] Collect and obtain data that needs to be analyzed, and preprocess the obtained data to obtain preprocessed data;

[0007] Select and load a preset training model based on the task type and performance indicators of the data to be analyzed;

[0008] Inputting the preprocessed data into the preset training model to perform data feature extraction to extract feature data representing data characteristics;

[0009] Automatically analyze the extracted feature data to generate prediction or classification results;

[0010] Apply attribution methods to automatically analyze the generated prediction or classification results to identify the key factors that affect the results.

[0011] The automated data analysis and attribution analysis method, wherein the step of identifying key factors affecting the results comprises:

[0012] Generate and output visual reports based on the identified key factors affecting the results.

[0013] The automated data analysis and attribution analysis method, wherein the step of collecting and obtaining the data to be analyzed includes:

[0014] Define in advance the type of data that needs to be analyzed and the goals of the data type analyzed.

[0015] In the automated data analysis and attribution analysis method, the steps of collecting and acquiring the data to be analyzed, and preprocessing the acquired data to obtain the preprocessed data include:

[0016] Collect and obtain the data that needs to be analyzed, and use statistical methods of data cleaning technology to identify and deal with outliers;

[0017] For data with outliers, apply Z-score standardization or Min-Max normalization to adjust the data range;

[0018] For data with adjusted data range, feature selection algorithm is used to select informative data features.

[0019] The automated data analysis and attribution analysis method, wherein the step of selecting and loading a preset training model according to the task type and performance index of the data to be analyzed comprises:

[0020] A preset training model library is provided in advance, including metadata of multiple pre-trained models;

[0021] Determine the matching preset training model based on the task type and performance indicators of the data to be analyzed;

[0022] The corresponding model parameters are loaded for the preset matching training model to ensure that the model can reproduce the performance during training in the analysis.

[0023] The automated data analysis and attribution analysis method, wherein the step of inputting the pre-processed data into the preset training model to extract data features and extracting feature data representing data characteristics comprises:

[0024] Inputting the preprocessed data into the preset training model to extract data features;

[0025] And modify the model structure according to the output of the middle layer of the model as required;

[0026] Based on the modified model structure, the layer representing the data characteristics is selected as the basis for feature extraction, and the feature data representing the data characteristics are extracted.

[0027] The automated data analysis and attribution analysis method, wherein the step of automatically analyzing the generated prediction or classification results by applying an attribution method to identify key factors affecting the results comprises:

[0028] For the generated prediction or classification results, apply the attribution method to quantify the contribution rate of the input features to the model output; perform attribution result analysis to identify the data features that have a key impact on the model decision, so as to identify the key factors affecting the results; and output the attribution result explanation to help understand the model prediction.

[0029] An automated data analysis and attribution analysis device, wherein the device comprises:

[0030] A pre-setting module is used to pre-define the data types to be analyzed and the targets of the analyzed data types;

[0031] The data acquisition and preprocessing module is used to collect and obtain the data to be analyzed, and to preprocess the obtained data to obtain the preprocessed data;

[0032] A loading module, used to select and load a preset training model according to the task type and performance index of the data to be analyzed;

[0033] A feature extraction module is used to input the preprocessed data into the preset training model to extract data features and extract feature data representing data characteristics;

[0034] The analysis module is used to automatically analyze the extracted feature data and generate prediction or classification results;

[0035] The attribution analysis module is used to automatically analyze the generated prediction or classification results using attribution methods to identify the key factors that affect the results;

[0036] The visualization generation module is used to generate and output visualization reports based on the identified key factors affecting the results.

[0037] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0038] A computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described above.

[0039] Beneficial effects of the invention: The invention provides an automated data analysis and attribution analysis method, device, intelligent terminal and storage medium. In order to realize automated data analysis and attribution analysis, the invention needs to use a pre-trained model to automatically analyze data. This process includes steps such as data preprocessing, model loading, feature extraction, automated analysis, attribution analysis, and result visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flowchart of the automated data analysis and attribution analysis method provided in Example 1 of the present invention.

[0042] Figure 2 It is a flowchart of the automated data analysis and attribution analysis method provided in Example 2 of the present invention.

[0043] Figure 3 It is a principle block diagram of an embodiment of the automated data analysis and attribution analysis device provided by the present invention.

[0044] Figure 4 It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0047] The present invention proposes an automated data analysis and attribution analysis method. In order to realize automated data analysis and attribution analysis, the present invention needs to use a pre-trained model to automatically analyze data. This process includes steps such as data preprocessing, model loading, feature extraction, automated analysis, attribution analysis, and result visualization.

[0048] Specifically, Figure 1 As shown, an automated data analysis and attribution analysis method according to Embodiment 1 of the present invention comprises the following steps:

[0049] Step S100: predefine the data type to be analyzed and the target of the analyzed data type.

[0050] In the embodiment of the present invention, before conducting data analysis, clarifying the data type and analysis target in advance helps to set the direction and framework of the analysis. In the embodiment of the present invention, the data type may include quantitative data (such as sales, number of users), qualitative data (such as customer feedback, market trends), etc. The analysis target involves the problem that is expected to be solved through data analysis, such as increasing sales, improving customer satisfaction, or optimizing operational efficiency.

[0051] In an embodiment of the present invention, the method of predefining these elements may include: wherein, the selection of data type:

[0052] Quantitative: Data expressed in numerical values, such as transaction records, can be used to analyze trends.

[0053] Qualitative: Non-numerical data, such as user reviews, can be used to understand customer needs.

[0054] The analysis objectives include:

[0055] Specific business goals, such as “increase website traffic by 20% in the next six months.”

[0056] Identify specific pain points, such as “Find out the top reasons for customer churn.”

[0057] The benefits of this step are: clear goals and data types make the entire analysis process more targeted and avoid aimless data processing; defining data types and goals in advance can help allocate resources reasonably and ensure that time and energy are invested in key data; and it helps to quickly filter out data related to the goal and reduce unnecessary data redundancy and complexity. Setting goals in advance helps to stay focused during the analysis process and improve the relevance and accuracy of the final analysis results.

[0058] For example, when an e-commerce company plans to analyze sales data, first of all, the company team needs to clarify several key points: 1) Data type:

[0059] Collect quantitative data, such as monthly sales data, customer purchase frequency, etc.

[0060] Collect qualitative data, such as product reviews and feedback from customers.

[0061] 2) Analysis objectives:

[0062] The goal is to increase customer conversion rate by 15% in the next quarter.

[0063] Identify key areas for analysis, such as understanding which products are most popular so you can optimize inventory and promotional strategies.

[0064] In this way, we can focus on analyzing relevant data, identify key factors that influence customer purchasing decisions, and ultimately develop more effective marketing strategies.

[0065] In short, pre-defining data types and analysis goals can not only improve work efficiency, but also ensure that the analysis results obtained directly serve business goals.

[0066] Step S200, collecting and acquiring data to be analyzed, and preprocessing the acquired data to obtain preprocessed data;

[0067] In this step embodiment, regarding the collection and acquisition of data to be analyzed, data can be obtained from different sources, such as internal systems (such as CRM, ERP), external databases (such as social media data, market research data), etc. When collecting data, the present invention will consider the relevance, accuracy and completeness of the data to ensure that the information obtained has practical value for the analysis target.

[0068] In the present invention, preprocessing the acquired data refers to cleaning and converting the collected data to eliminate noise and redundancy. Specifically, it includes the following steps:

[0069] Data cleaning: remove duplicate values, process missing values, correct erroneous data, etc. from the data that needs to be analyzed.

[0070] Data conversion: standardize the data to be analyzed, convert its format (such as unifying the date format), convert its data type, etc.

[0071] Data integration: Combining data from different sources to form a consistent and usable data set.

[0072] The resulting “preprocessed data” refers to a collection of cleaned and transformed data that is presented in a structured manner, ready for subsequent analysis.

[0073] The benefit of this step is that preprocessing can significantly improve the accuracy and reliability of data through cleaning and integration, thus providing a solid foundation for analysis. Clean and consistent data can reduce analytical bias caused by erroneous data and enhance the reliability of analytical results. Preprocessed data is usually more concise and easy to use, which can help analysts quickly find key data points and shorten analysis time. For data from different sources, preprocessing can ensure that the data has a unified standard and format, making subsequent analysis smoother.

[0074] Furthermore, the step S200 specifically includes:

[0075] S201. Collect and obtain the data to be analyzed, and use statistical methods of data cleaning technology to identify and process outliers;

[0076] In this step embodiment, data related to the analysis target is collected from various data sources (such as databases, sensors, online forms, etc.). After the data is collected, statistical methods (such as box plots, Z-scores, etc.) are used to identify outliers (i.e., outliers). Among them, the outliers refer to data entry errors, measurement errors, or real extreme cases. The methods for handling outliers may include deletion, truncation, or replacement to ensure the integrity and accuracy of the data set.

[0077] S202, applying Z-score standardization or Min-Max normalization to the data with processed outliers to adjust the data range;

[0078] In this step, the data after outliers are processed needs to be normalized to eliminate the impact of data scale differences on the analysis.

[0079] Specifically, Z-score standardization is applied: it means converting the data in units of standard deviation by calculating the deviation of the data points from the mean, so that the data is standardized to a distribution with a mean of 0 and a standard deviation of 1.

[0080] Min-Max Normalization: linearly transform the data to [0,1] or other specific range so that the data can be compared on the same scale.

[0081] This step helps improve the performance of the model, especially when using distance-based methods (such as KNN, cluster analysis).

[0082] S203: For the data whose data range is adjusted, select informative data features through a feature selection algorithm.

[0083] In the embodiment of the present invention, after the data is standardized or normalized, feature selection is required, the purpose of which is to filter out the most useful information for the target variable from a large number of features. The feature selection algorithm used in the present invention may include:

[0084] Variance selection: remove features whose variance is less than a certain threshold.

[0085] LASSO regression: Select important features through penalty terms.

[0086] Tree model feature importance: Use feature importance scores from models such as decision trees to select features.

[0087] The benefit of this step is that by cleaning and processing outliers, the authenticity and reliability of the data set are ensured, and potential analysis errors are reduced. Standardization or normalization helps to balance the scale of features, making the model easier to converge and improving prediction accuracy. Feature selection can also reduce data dimensions, retain important information, improve computational efficiency, and reduce the risk of overfitting. In addition, by selecting relevant features, the present invention makes the analysis results more interpretable, making it easier for decision makers to understand the information behind the data.

[0088] Step S300: Select and load a preset training model according to the task type and performance index of the data to be analyzed;

[0089] The goal of this step is to ensure that the selected model can effectively complete the specific task and achieve the expected analysis results.

[0090] Specifically, in this step, the type of task is determined: At this stage, the type of problem to be solved must be determined. According to the data analysis task, it can be divided into the following types:

[0091] Classification problems: For example, classifying an email as “good” or “spam”.

[0092] Regression problems: such as predicting the price of an item.

[0093] Clustering problems: for example, grouping customers into groups for targeted marketing.

[0094] Time series forecasting: such as predicting future sales.

[0095] In the embodiment of the present invention, different task types will affect the selection of the required model.

[0096] The performance indicators for selection include:

[0097] Determine the indicators for evaluating the model effect, such as: Classification task: accuracy, precision, recall, F1 score, etc. Regression task: mean square error (MSE), root mean square error (RMSE), R 2 Ratings, etc.

[0098] In the embodiment of the present invention, the performance indicators will help analyze and understand the effect of the model to ensure that it meets business needs.

[0099] Select and load the preset training model: The present invention can automatically select the most suitable model to load from multiple preset models (such as logistic regression, random forest, support vector machine, neural network, etc.) according to the task type and performance index. In the present invention, this can be done through machine learning libraries (such as Scikit-learn, TensorFlow, PyTorch, etc.), making the model loading process efficient and convenient.

[0100] The benefit of this step is that by selecting a suitable model according to the task type, the adaptability of the model to the data can be improved, thereby improving the prediction performance. And the present invention loads the preset training model, which can avoid too much model development and training time, and is more conducive to the tuning and result analysis of the analysis data model. And in practical applications, different preset models can be quickly replaced and tested according to different data performances to improve work efficiency. And using a verified training model can obtain a more stable performance and reduce the risks caused by improper model selection.

[0101] For example, when an e-commerce company wants to predict the customer churn rate based on the customer's historical shopping behavior in order to develop a more effective customer retention strategy. About determining the task type: In this analysis embodiment, the goal is to determine whether the customer is at risk of churn, so this is a classification task. Select performance indicators: You can decide to use accuracy and F1 score as performance indicators because they can comprehensively evaluate the model's ability to correctly classify customers. Select and load the preset training model: According to the requirements of the classification task, you can view the preset training model library and select the random forest model because it performs well when processing large-scale data sets and has strong stability. Subsequently, in the embodiment of the present invention, according to the above conditions, it will automatically select to use the Scikit-learn library (an open source machine learning library) to load this preset model, and fine-tune it to adapt to specific data characteristics and business needs.

[0102] It can be seen that through the description of this specific embodiment, the e-commerce company in this example can quickly obtain a preliminary churn prediction model, and further optimize and adjust it on this basis, thereby improving the effectiveness of the customer retention strategy.

[0103] In the embodiment of the present invention, this step combines task objectives, performance evaluation and model selection, ensures the effectiveness and practicality of data analysis, and provides reliable support for subsequent decision-making.

[0104] Furthermore, the step S300 specifically includes:

[0105] S301, pre-providing a preset training model library, including metadata of multiple pre-training models;

[0106] In this step, a model library is pre-built, which contains relevant information (metadata) of various pre-trained models. These metadata may include the type of model, applicable tasks (such as classification, regression), loss function, characteristics and labels of training data, performance indicators (such as accuracy, F1 score), etc.

[0107] S302, determining a matching preset training model according to the task type and performance index of the data to be analyzed;

[0108] In this step, according to the problem to be solved, for example, the problem to be solved is image classification, text analysis or predictive modeling, a preset model matching the task type is selected. At the same time, according to performance indicators (such as accuracy, recall, etc.), the most suitable pre-trained model can be further filtered and determined.

[0109] S303: Load corresponding model parameters for the preset matching training model to ensure that the model can reproduce the performance during training in the analysis.

[0110] In this step, the relevant model parameters are loaded according to the matching model determined in step S302. These parameters are learned during model training, ensuring that the prediction performance of the model in a specific task can be reproduced. This step specifically involves loading model weights and biases, as well as other hyperparameter settings.

[0111] In the embodiment of the present invention, through the preset model library, there is no need to train a new model from scratch, thereby saving a lot of time and computing resources. And loading the pre-trained model and its parameters can ensure that the performance of the model in a specific task reaches a predictable level, reducing the risk of poor results caused by improper model training. And especially when processing large-scale data, training deep learning models often requires high computing resources and time, and using pre-trained models can significantly reduce these expenses.

[0112] For example, a medical technology company wants to develop a model to predict the risk of a certain disease. By following these steps, they can efficiently advance the project:

[0113] Step S301: First, a model library is established, which contains metadata of various disease prediction models, such as logistic regression, random forest and deep learning models. Each model is equipped with information such as accuracy, feature selection method, training data source, etc.

[0114] Step S302: Based on the requirements (for example, they want a higher recall rate to ensure that as many disease cases as possible are identified), the random forest model is automatically selected because it has performed well in past experiments and can handle complex relationships between features.

[0115] Step S303: Load the pre-trained parameters of the random forest model, so that you can immediately start evaluating and predicting new data without having to go through the tedious training process. You can quickly verify the model performance, analyze the results, and even deploy it to a production environment.

[0116] Through the above steps, the development and application of disease prediction models can be quickly realized with lower time and resource investment, and more focus can be placed on achieving business goals.

[0117] Step S400: input the preprocessed data into the preset training model to extract data features and extract feature data representing data characteristics;

[0118] In this step, the preprocessed data is input into a pre-selected preset training model to extract data features. Feature extraction is an important part of the machine learning process, and its main purpose is to extract useful information for analysis and prediction from the raw data and convert the data into a format that can be effectively processed by the model.

[0119] In the embodiment of the present invention, pre-processed data (such as normalized, standardized or noise-removed data) is input into the model. This process ensures that the input data meets the requirements of the model.

[0120] The model then analyzes the input data based on its internal structure and algorithm, extracting features that represent the characteristics of the data. These features can be statistical characteristics of the data, important variables, pattern recognition results, or other more complex representations.

[0121] After feature extraction is completed, the generated feature data is output. This feature data set will become the basis for subsequent model training or analysis.

[0122] For example, a company wants to predict users’ purchasing intentions by analyzing social media data. Here’s how this process works:

[0123] Regarding input data, a large amount of user comment data can be collected from social media platforms. These data are pre-processed, including cleaning (removing irrelevant information), word segmentation (converting comment text into a word list), and sentiment analysis (determining the positive or negative sentiment of the comment).

[0124] Regarding feature extraction, the review text can be analyzed through a preset feature extraction model (for example, using TF-IDF or Word2Vec algorithm). Through these methods, the model can extract many key features, such as the frequency of certain keywords, the relationship between words, and the review sentiment score.

[0125] After feature extraction, a feature vector can be obtained, which represents the main features of each comment, and these feature vectors can well characterize the user's attitude and emotion. Then, these feature data will be input into the subsequent classification model to predict which users are most likely to make a purchase.

[0126] Through this implementation approach, the company can not only quickly adapt to market changes, but also conduct targeted marketing and advertising more accurately, thereby improving business efficiency and user satisfaction.

[0127] The benefit of this step is that effective features can greatly enhance the predictive ability of the model. By extracting key features, the model can better learn the potential patterns in the data, thereby accurately classifying or regressing. And feature extraction can help reduce the dimension of the data and remove redundant and irrelevant information. This makes model training more efficient while reducing computational complexity. Moreover, the extracted features are usually more explanatory, helping data scientists understand the reasons and patterns behind the data, thereby providing a basis for business decisions. And through feature extraction, the preset model can be flexibly applied to a variety of different tasks, such as image recognition, text document classification, etc.

[0128] In a further embodiment, the step S400 specifically includes:

[0129] S401: Input the preprocessed data into the preset training model to extract data features;

[0130] In this step, the preprocessed data is fed into a preset machine learning model (such as a deep learning model). The model processes the input data and analyzes the data through an internal neural network or algorithm to extract feature vectors representing the characteristics of the data. These feature vectors will provide support for downstream tasks (such as classification, clustering, etc.).

[0131] S402: modifying the model structure according to the output of the middle layer of the model as required;

[0132] In this step, the embodiment is to determine whether the model structure needs to be adjusted based on the output of the middle layer of the model (i.e., the representation obtained after the data passes through certain layers of the model). For example, if it is observed that the output of a specific layer cannot effectively represent the data characteristics, it may be necessary to add layers, reduce layers, or change the connection method (such as activation function, convolution stride, etc.) to improve the model's ability to learn data characteristics.

[0133] S403: Based on the modified model structure, a layer representing data characteristics is selected as a basis for feature extraction, and feature data representing data characteristics is extracted;

[0134] In this step, according to the adjusted model architecture, the layer that can best represent the data characteristics is selected as the feature extraction source. Through these optimized feature extraction layers, the model can more accurately identify important features, thereby generating feature vectors, which will be used for subsequent decision-making and analysis.

[0135] The benefit of this embodiment is that by evaluating the output of the middle layer and adjusting the model structure, it can be ensured that the extracted features can better reflect the true characteristics of the data, thereby improving the performance of the overall model. And in the present invention, according to the specific data and task requirements, the model structure is dynamically adjusted so that the model can adapt to different types of data more flexibly, thereby improving the scope of application. And after adjusting the model structure, by selecting a suitable feature extraction layer, it can help analysts better understand the decision-making process of the model, as well as the relationship between features and prediction results. Moreover, the deep learning model of the present invention usually has multiple layers, each layer extracts information at different levels, and optimizing and selecting suitable layers can make full use of its deep characteristics to obtain richer data representations.

[0136] Step S500: Automated analysis of the extracted feature data to generate prediction or classification results;

[0137] In this step, the extracted feature data is used for automated analysis to generate corresponding prediction or classification results.

[0138] Specifically, in this step, the extracted and processed feature data is input into the prediction or classification model. These features contain important descriptions of the input data and can reflect the key attributes of the sample.

[0139] The model then calculates and analyzes the input feature data, using specific algorithms (such as regression, decision trees, support vector machines, etc.) to identify relationships between the data in order to make predictions or classification decisions.

[0140] After the analysis is completed, the model outputs prediction results (such as numerical predictions, probability distributions) or classification results (such as category labels). These results will provide a basis for decision-making or support subsequent business operations.

[0141] For example, when an e-commerce company wants to analyze user purchasing behavior to predict future sales trends. Here is an example of how this process is implemented:

[0142] About the feature data extraction step: The user's purchase characteristics can be extracted through the previous steps, including each user's average shopping cart value, shopping frequency, favorite product categories, and past purchase trends.

[0143] Regarding automated analysis: These feature data can be input into a trained predictive model (such as a regression model or a random forest model). The model will analyze the relationship between these features, for example: "If a user's shopping frequency has increased in the past three months, the likelihood of future purchases will also increase."

[0144] As for result generation: the model can output a series of prediction results, including the expected growth rate of sales, the potential popularity of various products, and the possible user churn rate, etc. These results provide a basis for the company to formulate promotion strategies, inventory management and marketing plans.

[0145] In this way, the company can make decisions based on accurate forecast results, optimize resource allocation, and thus increase market share in the competition. At the same time, this automated analysis process also enables the company to flexibly respond to market changes, quickly adjust strategies, and ensure its continued development and profitability.

[0146] It can be seen that the benefit of the step embodiment is that the need for manual intervention is greatly reduced through automated analysis, which significantly speeds up data processing. In a big data environment, manual analysis is almost impossible to complete, and automated analysis ensures high efficiency. And because all predictions and classifications are automatically performed by machine learning models, human errors and biases are reduced. Whether the amount of data is large or small, the model can maintain consistent analysis results. And the automated analysis of the present invention can quickly process new data, provide instant feedback and decision support, and is particularly suitable for business scenarios that require rapid response, such as financial transactions, network security detection, etc. And as the amount of data increases, automated analysis can be expanded accordingly, maintaining performance without affecting the quality of analysis. This capability is particularly important for companies that must cope with a constantly changing data environment.

[0147] Step S600: Apply an attribution method to automatically analyze the generated prediction or classification results to identify key factors that affect the results.

[0148] In this step embodiment, the generated prediction or classification results are automatically analyzed using an attribution method to identify key factors that affect the results. The specific steps of this process include:

[0149] Result input is to import the previously generated prediction or classification results into the attribution analysis tool. The purpose of attribution analysis is to understand which factors or characteristics play an important role in the final result.

[0150] Applying the attribution method, various attribution techniques can be used for analysis in the present invention, such as linear regression, SHAP (SHapley Additive exPlanations) value, LIME (Local Interpretable Model-agnosticExplanations), etc. These techniques can help identify the contribution of features to the model output value and reveal how the model makes the current prediction or classification.

[0151] Then identify the key factors. The ultimate goal of the analysis in this invention is to determine the key factors that affect the results. By sorting the importance of the features, it is clear which features are the main driving force for the change of results, thereby providing an accurate basis for subsequent decision-making.

[0152] For example, let’s take a health technology company that uses analytical models to predict users’ health risks, such as the risk of chronic diseases such as diabetes. Here’s how the process works:

[0153] Regarding the generated results, in an embodiment of the present invention, a health risk prediction result can be generated based on the user's living habits, dietary data and physical examination information. For example, a user's diabetes risk score is 80 (out of 100).

[0154] Regarding the application attribution method, in this embodiment, the SHAP value can be used to analyze the prediction results to understand which factors have the greatest impact on the user's diabetes risk score. Through analysis, it is found that the user's eating habits and exercise frequency are the two most important factors. Among them, the SHAP value is an important tool for interpreting the prediction results of machine learning models. The SHAP value of each feature represents the size and direction of the feature's contribution to the model's prediction results.

[0155] In this embodiment, the key factors to be identified are as follows: according to the analysis results, the high-sugar diet and low activity level of consumers are the main reasons for their high diabetes risk scores. At the same time, saturated fat intake also shows a certain influence.

[0156] By applying attribution methods, we can not only clearly understand the key factors that affect the user's health risks, but also develop personalized health management plans based on these insights. For example, we can recommend that the user adjust his diet and increase the frequency of exercise to effectively reduce the risk of diabetes.

[0157] In this way, attribution analysis enables companies to gain a deeper understanding of model prediction results, helping to optimize user health management, enhance customer satisfaction, and improve the company's market competitiveness.

[0158] Furthermore, the step S600 specifically includes:

[0159] For the generated prediction or classification results, apply the attribution method to quantify the contribution rate of the input features to the model output; perform attribution result analysis to identify the data features that have a key impact on the model decision, so as to identify the key factors affecting the results; and output the attribution result explanation to help understand the model prediction.

[0160] That is, in the embodiments of the present invention, attribution methods are used to quantify the specific contribution of input features to the model output. Through algorithms (such as SHAP values, LIME, etc.), the role of each feature in a specific prediction can be calculated. For example, if the model predicts that a customer may churn, the attribution method will reveal which features lead to this prediction and quantify their influence.

[0161] In a specific embodiment of the present invention, after obtaining the feature contribution evaluation, analyzing the attribution results is a key step. This process can help understand which features play a decisive role in the prediction. In this way, the key factors affecting the results can be identified, thereby providing a basis for subsequent decision-making.

[0162] Regarding the explanation of the output results, the embodiments of the present invention can generate easy-to-understand explanations based on the results of the attribution analysis to help stakeholders understand why the model makes a specific prediction. These explanations can be displayed through visualization tools, such as bar charts, scatter plots, etc., to better convey information.

[0163] For example, let’s take a financial institution that uses a machine learning model to predict the default risk of loan applicants. After applying the attribution method, the features and their contributions are as follows:

[0164] Applicant income: SHAP value is +0.4;

[0165] Credit score: SHAP value is +0.6;

[0166] Debt-to-income ratio: SHAP value is -0.3;

[0167] Past overdue records: SHAP value is -0.8;

[0168] In this example, credit score and applicant income have a positive impact on the perceived risk of default, while debt-to-income ratio and past delinquencies indicate an increased risk.

[0169] Through this attribution analysis, financial institutions can develop targeted strategies, such as:

[0170] Increase transparency in loan approval: By emphasizing the importance of credit scores, applicants can be guided to improve their credit status to reduce the risk of default.

[0171] Invest resources in education: For applicants with high debt-to-income ratios, provide education on financial management and credit repair to improve their application success rate.

[0172] Finally, by visually presenting these results to decision makers, financial institutions can more easily understand the model’s predictions, allowing them to make more informed decisions and improve business efficiency and customer satisfaction.

[0173] It can be seen that by understanding the specific contribution of input features to model predictions, the embodiments of the present invention can enable users to gain a deep understanding of model behavior, thereby increasing trust in the model. After identifying key features, decision makers can focus on optimizing these features, thereby improving overall business results. For example, by strengthening customer satisfaction management, churn rates can be reduced. And in some industries, transparency and explainability of model decisions are compliance requirements. Attribution analysis helps meet these requirements and ensure the legitimacy and transparency of the decision-making process.

[0174] Step S700: Generate and output a visualization report based on the identified key factors affecting the results.

[0175] In this step embodiment, generating a visualization report is a key step in the process of machine learning model analysis. Specifically, it includes the following aspects:

[0176] Visualization based on key factors: that is, after identifying the key factors that have a significant impact on the model output in the input features and attribution analysis, the present invention generates visualization content based on these factors. Visualization can be in various forms, such as bar charts, pie charts, scatter plots, etc., in order to clearly show the importance and impact of the features.

[0177] The embodiments of the present invention highlight the main findings and conclusions through the generated visual report. The SHAP value or other contribution value of each key feature can be displayed through a chart, and historical data and trend analysis corresponding to the model prediction can also be included. This process can also be combined with text descriptions to make the report easier to understand.

[0178] The final report can be exported into multiple formats (such as PDF, PPT, etc.) to facilitate sharing with project teams, management and other stakeholders. This format allows the report to be used in different environments, such as meetings, training or decision-making evaluation.

[0179] Because visualization can transform complex data and abstract outputs into intuitive information, so that people without a deep data analysis background can also understand the model's predictions and the role of key factors. This helps a wider audience obtain information and participate in decision-making. And visual reports can serve as a basis for discussion, allowing team members, management and other stakeholders to reach a consensus on key data contributions, thereby promoting cross-departmental cooperation and coordination. Such sharing can accelerate the decision-making process and improve efficiency. And when decision makers face complex choices, clear visual reports can help quickly identify which factors have a decisive impact on the results, so as to make more informed judgments based on data and improve the effectiveness of decisions.

[0180] In short, generating visual reports not only enhances the readability of data, but also greatly improves the scientificity and effectiveness of organizational decision-making.

[0181] Based on the above embodiments, the present invention is further described in detail through another specific application embodiment.

[0182] like Figure 2 As shown, the second embodiment provides an automated data analysis and attribution analysis method, comprising the following steps:

[0183] S10, start and go to step S11;

[0184] S11, define the problem and goal, and proceed to step S12;

[0185] In this step, defining the problem and goal specifically means defining what data to analyze and what the purpose of analyzing this data is. For example, an online e-commerce platform wants to improve the purchase conversion rate of customers. The data to be analyzed is defined as follows: The platform decides to analyze the following data:

[0186] Customer behavior data: such as page views, dwell time, items added to the shopping cart, and access sources (such as search engines, social media, etc.).

[0187] Transaction data: such as user’s purchase history, order amount, and payment method.

[0188] Customer information: Demographic information such as age, gender, and geographic location.

[0189] Define the specific problem: The problem is defined as: "Why do customers fail to complete the purchase after browsing the product? Are there certain specific factors (such as the website's user experience, product pricing, promotional activities) that affect the customer's conversion rate?"

[0190] Set goals and expected results: The goals are: Identify the key factors that influence purchase decisions. Increase conversion rate by 10%. Provide data support and suggestions for adjusting marketing strategies and improving user experience.

[0191] Through this series of definitions, e-commerce platforms can clearly track relevant data, analyze customer behavior, and implement targeted strategies based on this to improve sales performance.

[0192] S12, data collection and preprocessing, and then proceed to step S13;

[0193] Regarding data collection and preprocessing in this step, for example, data including faces is used as an example to illustrate: data collection and preprocessing can be performed through the following steps:

[0194] 1) Use data cleaning techniques, such as using statistical methods to identify and handle outliers.

[0195] 2). Apply normalization algorithms, such as Z-score normalization or Min-Max normalization, to adjust the data range.

[0196] 3). Select the most informative features through feature selection algorithms, such as mutual information or correlation coefficient.

[0197] S13, select and load the model, and proceed to step S14;

[0198] In this step, you can first maintain a model library that contains metadata of multiple pre-trained models. Then select a model based on the task type (such as text, image, speech, etc.) and performance indicators (such as accuracy, speed, etc.); and load the model parameters to ensure that the model can reproduce the performance during training in the analysis.

[0199] S14, extract data features and proceed to step S15;

[0200] In the embodiment of the present invention, in the step of extracting data features, the preprocessed data can be input into the model to obtain the intermediate layer representation of the model; and the layer that can represent the data characteristics is selected as the basis for feature extraction. The processing in this step may include feature dimensionality reduction technology, such as PCA or t-SNE, to reduce the feature dimension.

[0201] S15, automatic analysis, and proceed to step S16;

[0202] S16, data attribution analysis, and proceed to step S17;

[0203] In the embodiment of the present invention, regarding the automated analysis and attribution analysis, specifically:

[0204] 1). Attribution methods such as Integrated Gradients or LIME can be applied to quantify the contribution of input features to the model output.

[0205] 2) Analyze the attribution results and identify the features that have the greatest impact on the model’s decision.

[0206] 3) Provide explanations of the attribution results to help understand the model’s predictions.

[0207] S17, visualization and reporting of results, and proceeding to step S18;

[0208] In the embodiment of the present invention, the visualization of the analysis results is specifically as follows:

[0209] 1). Select appropriate visualization charts according to the result type, such as bar charts, scatter plots, heat maps, etc.

[0210] 2). Use a visualization library, such as Matplotlib or Seaborn, to generate charts.

[0211] 3) Provide interactive visualizations that allow users to explore the data in depth.

[0212] S18, end;

[0213] In the specific application embodiment of the present invention, regarding the following steps:

[0214] 1) Define the data types that need to be analyzed, such as user transaction data, user repayment timeliness rate, etc. The goal is to predict the user's default risk;

[0215] 2) Collect user data and pre-process the data;

[0216] That is, before analyzing the data, the raw data needs to be cleaned and transformed. This may include processing missing values, removing outliers, standardizing data formats, etc., so that subsequent operations are more efficient and accurate.

[0217] 3) Select a suitable model to analyze the data;

[0218] That is, the pre-trained model in the present invention refers to a machine learning or deep learning model that has been trained on a large-scale data set and has achieved certain performance. In this step, these models are first loaded into the environment for subsequent analysis.

[0219] 4) Data feature extraction;

[0220] Feature extraction is the process of identifying and extracting relevant features from data. These features will be used for model analysis and prediction. Pre-trained models can usually automatically identify important features in the data.

[0221] 5) Automated analysis and attribution analysis;

[0222] Once feature extraction is complete, the system can automatically run the model, analyze the data, and generate predictions or classification results. This process does not require human intervention, greatly improving efficiency. Attribution analysis is used to identify and understand the causes of specific results. In this step, you can analyze the model output to understand, for example, which features have the greatest impact on the final prediction results, helping to make data-driven decisions.

[0223] 6) Visualization of results.

[0224] Finally, the results of the analysis need to be presented in an intuitive way so that the results can be understood by human users. This can be achieved through charts, dashboards, etc. to facilitate decision support.

[0225] For example, let’s say an e-commerce company wants to analyze customer purchasing behavior to improve sales strategies. The implementation process is as follows:

[0226] In the data preprocessing stage: After collecting customer data, it is found that some customers’ ages and purchase records are missing. Data preprocessing is completed by filling the average value and deleting incomplete records.

[0227] In the model loading stage: you can use a recommendation system model that has been pre-trained on other e-commerce datasets and load the model for current data analysis.

[0228] In the feature extraction stage: the model extracts features related to customer purchasing behavior, such as customer browsing history, purchase frequency, shopping cart abandonment rate, etc.

[0229] In the automated analysis phase: the model automatically analyzes the extracted features, predicts which products are most likely to be purchased by specific customers, and generates corresponding recommendations.

[0230] In the attribution analysis stage: the present invention can show in the analysis results that the feature of "browsing history" has the greatest impact on the increase in purchase probability. The team can further study why this feature is so important and consider adjusting the strategy.

[0231] In the result visualization stage: Finally, visualization tools can be used to generate charts to show the purchasing trends of different customer groups so that decision makers can understand and develop new marketing strategies.

[0232] Through this series of steps, using pre-trained models to analyze data not only improves efficiency, but also provides valuable insights and provides data support for the company to improve sales performance.

[0233] Exemplary Devices

[0234] like Figure 3 As shown, an embodiment of the present invention provides an automated data analysis and attribution analysis device, the device comprising:

[0235] A presetting module 310 is used to predefine the data type to be analyzed and the target of the analyzed data type;

[0236] The data acquisition and preprocessing module 320 is used to collect and obtain data to be analyzed, and to preprocess the obtained data to obtain preprocessed data;

[0237] A loading module 330, for selecting and loading a preset training model according to the task type and performance index of the data to be analyzed;

[0238] The feature extraction module 340 is used to input the preprocessed data into the preset training model to extract data features and extract feature data representing data characteristics;

[0239] The analysis module 350 is used to automatically analyze the extracted feature data and generate prediction or classification results;

[0240] Attribution analysis module 360, used to automatically analyze the generated prediction or classification results by applying attribution methods to identify key factors affecting the results;

[0241] The visualization generation module 370 is used to generate and output a visualization report based on the identified key factors affecting the results, as described above.

[0242] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 4 As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a database connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an automated data analysis and attribution analysis method is implemented. The database of the intelligent terminal is used to store automated data analysis and attribution analysis programs.

[0243] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0244] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0245] Collect and obtain data that needs to be analyzed, and preprocess the obtained data to obtain preprocessed data;

[0246] Select and load a preset training model based on the task type and performance indicators of the data to be analyzed;

[0247] Inputting the preprocessed data into the preset training model to perform data feature extraction to extract feature data representing data characteristics;

[0248] Automatically analyze the extracted feature data to generate prediction or classification results;

[0249] Apply attribution methods to automatically analyze the generated prediction or classification results to identify the key factors that affect the results.

[0250] The step of identifying the key factors affecting the results includes:

[0251] Generate and output visual reports based on the identified key factors affecting the results.

[0252] The step of collecting and acquiring the data to be analyzed includes:

[0253] Define in advance the type of data that needs to be analyzed and the goals of the data type analyzed.

[0254] The steps of collecting and acquiring data to be analyzed, and preprocessing the acquired data to obtain the preprocessed data include:

[0255] Collect and obtain the data that needs to be analyzed, and use statistical methods of data cleaning technology to identify and deal with outliers;

[0256] For data with outliers, apply Z-score standardization or Min-Max normalization to adjust the data range;

[0257] For data with adjusted data range, feature selection algorithm is used to select informative data features.

[0258] The step of selecting and loading a preset training model according to the task type and performance index of the data to be analyzed includes:

[0259] A preset training model library is provided in advance, including metadata of multiple pre-trained models;

[0260] Determine the matching preset training model based on the task type and performance indicators of the data to be analyzed;

[0261] The corresponding model parameters are loaded for the preset matching training model to ensure that the model can reproduce the performance during training in the analysis.

[0262] The step of inputting the preprocessed data into the preset training model to extract data features and extracting feature data representing data characteristics comprises:

[0263] Inputting the preprocessed data into the preset training model to extract data features;

[0264] And modify the model structure according to the output of the middle layer of the model as required;

[0265] Based on the modified model structure, the layer representing the data characteristics is selected as the basis for feature extraction, and the feature data representing the data characteristics are extracted.

[0266] The step of automatically analyzing the generated prediction or classification results by applying an attribution method to identify key factors affecting the results includes:

[0267] For the generated prediction or classification results, apply the attribution method to quantify the contribution rate of the input features to the model output; perform attribution result analysis to identify the data features that are critical to the model decision, so as to identify the key factors affecting the results; and output the attribution result explanation to help understand the model prediction, as described above.

[0268] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0269] In summary, the present invention provides an automated data analysis and attribution analysis method, device, intelligent terminal and storage medium. In order to realize automated data analysis and attribution analysis, the present invention needs to use a pre-trained model to automatically analyze data. This process includes steps such as data preprocessing, model loading, feature extraction, automated analysis, attribution analysis, and result visualization.

Claims

1. An automated data analysis and attribution analysis method, characterized in that: include: Collect and obtain data that needs to be analyzed, and preprocess the obtained data to obtain preprocessed data; Select and load a preset training model based on the task type and performance indicators of the data to be analyzed; Inputting the preprocessed data into the preset training model to perform data feature extraction to extract feature data representing data characteristics; Automatically analyze the extracted feature data to generate prediction or classification results; Apply attribution methods to automatically analyze the generated prediction or classification results to identify the key factors that affect the results.

2. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The step of identifying the key factors affecting the results then includes: Generate and output visual reports based on the identified key factors affecting the results.

3. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The step of collecting and acquiring the data to be analyzed includes: Define in advance the type of data that needs to be analyzed and the goals of the data type analyzed.

4. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The steps of collecting and acquiring data to be analyzed, and preprocessing the acquired data to obtain the preprocessed data include: Collect and obtain the data that needs to be analyzed, and use statistical methods of data cleaning technology to identify and deal with outliers; For data with outliers, apply Z-score standardization or Min-Max normalization to adjust the data range; For data with adjusted data range, feature selection algorithm is used to select informative data features.

5. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The step of selecting and loading a preset training model according to the task type and performance index of the data to be analyzed includes: A preset training model library is provided in advance, including metadata of multiple pre-trained models; Determine the matching preset training model based on the task type and performance indicators of the data to be analyzed; The corresponding model parameters are loaded for the preset matching training model to ensure that the model can reproduce the performance during training in the analysis.

6. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The step of inputting the preprocessed data into the preset training model to extract data features and extracting feature data representing data characteristics comprises: Inputting the preprocessed data into the preset training model to extract data features; And modify the model structure according to the output of the middle layer of the model as required; Based on the modified model structure, the layer representing the data characteristics is selected as the basis for feature extraction, and the feature data representing the data characteristics are extracted.

7. The automated data analysis and attribution analysis method according to claim 1, characterized in that: The step of automatically analyzing the generated prediction or classification results by applying the attribution method to identify the key factors affecting the results includes: For the generated prediction or classification results, apply the attribution method to quantify the contribution rate of the input features to the model output; perform attribution result analysis to identify the data features that have a key impact on the model decision, so as to identify the key factors affecting the results; and output the attribution result explanation to help understand the model prediction.

8. An automated data analysis and attribution analysis device, characterized in that: The device comprises: A pre-setting module is used to pre-define the data types to be analyzed and the targets of the analyzed data types; The data acquisition and preprocessing module is used to collect and obtain the data to be analyzed, and to preprocess the obtained data to obtain the preprocessed data; A loading module, used to select and load a preset training model according to the task type and performance index of the data to be analyzed; A feature extraction module is used to input the preprocessed data into the preset training model to extract data features and extract feature data representing data characteristics; The analysis module is used to automatically analyze the extracted feature data and generate prediction or classification results; The attribution analysis module is used to automatically analyze the generated prediction or classification results using attribution methods to identify the key factors that affect the results; The visualization generation module is used to generate and output visualization reports based on the identified key factors affecting the results.

9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of claims 1 to 7.

Images