A method, medium and electronic device for automated data analysis
By automatically selecting data analysis methods through a dynamic knowledge base and reinforcement learning decision framework, the operational difficulties faced by non-professional users are solved, achieving efficient and accurate data analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI NAT GRP HEALTH TECH CO LTD
- Filing Date
- 2026-01-28
- Publication Date
- 2026-06-05
AI Technical Summary
Existing data analysis tools have high barriers to entry for non-professional users, low efficiency in data cleaning and preprocessing, and difficulty in selecting appropriate analysis methods, leading to inaccurate analysis conclusions.
We construct an analytical framework driven by both a dynamic knowledge base and multi-objective reinforcement learning decision-making. We automatically select analytical methods based on data features and introduce a closed-loop feedback mechanism for optimization.
It enables the automation, adaptability, and continuous evolution of analytical methods, lowers the professional threshold, and ensures the accuracy and robustness of analytical conclusions.
Smart Images
Figure CN122153280A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and more specifically, embodiments of this application relate to a method, medium, and electronic device for automated data analysis. Background Technology
[0002] With the advent of the big data era, the demand for data analysis in various industries has exploded.
[0003] However, current data analysis practices still face challenges because the analysis process heavily relies on the user's professional statistical knowledge. For a large number of non-professional users (e.g., clinicians, social science researchers, or business analysts), facing complex datasets (i.e., raw data files), it is difficult to select the appropriate analysis method from among numerous methods (e.g., parametric tests, survival analysis, or regression models), leading to inaccurate or even misleading analytical conclusions.
[0004] Existing technical solutions suffer from the following shortcomings: First, mainstream tools (such as SPSS, R, and Python) require users to independently determine and select analysis methods or write code, resulting in a high operational threshold. Second, tedious preprocessing steps such as data cleaning and handling missing values usually need to be performed manually, which is inefficient and prone to errors. Therefore, how to provide an analysis tool that can automatically select analysis methods to lower the professional threshold has become an urgent technical problem to be solved in this field. Summary of the Invention
[0005] The purpose of this application is to provide a method, medium, and electronic device for automated data analysis. This application constructs a dual-driven analysis method recommendation framework—combining dynamic knowledge base recommendation and multi-objective reinforcement learning decision-making—to select matching analysis methods for the analysis objectives and original data files. It also introduces a closed-loop feedback mechanism based on the analysis results, achieving for the first time automation, adaptability, and continuous evolution in analysis method selection. This transforms the statistical decision-making process, which relies on expert experience, into an intelligent system capable of continuous learning and optimization from the data itself and historical practice. This lowers the professional threshold while ensuring the accuracy, robustness, and practicality of the analysis conclusions.
[0006] In a first aspect, embodiments of this application provide an automated data analysis method, the method comprising: acquiring an original data file and the target of the current analysis; obtaining a target data table based on initial data features of the original data file, wherein the initial data features include: field type, distribution pattern, outliers and / or missing values, and the target data table is obtained by correcting the data of corresponding fields based on the initial data features; acquiring updated data features corresponding to the original data file based on the target data table; inputting the updated data features and the target of the current analysis into a pre-built analysis method recommendation knowledge base to obtain a set of candidate analysis methods; and, based on the target... The target data table is used to perform a performance simulation pre-evaluation on at least some of the analysis methods in the candidate analysis method set to obtain the real-time performance evaluation index of each method; the updated data features, the state constituted by the current analysis target, the action constituted by the candidate analysis method set, and the real-time performance evaluation index are input into a pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target; the target analysis method is used to analyze the target data table to generate analysis results; based on the evaluation feedback of the analysis results, the confidence weight of the mapping relationship in the recommendation knowledge base of the analysis method is updated, and the strategy of the reinforcement learning decision model is updated.
[0007] The embodiments of this application construct an intelligent analysis framework driven by both a dynamic analysis method recommendation knowledge base and multi-objective reinforcement learning decision-making. A closed-loop feedback mechanism based on analysis results is introduced to select a matching analysis method for the current analysis objective and the original data file, achieving for the first time automation, adaptability, and continuous evolution in analysis method selection. This transforms the statistical decision-making process, which relies on expert experience, into an intelligent system capable of continuous learning and optimization from the data itself and historical practice. This lowers the professional threshold while ensuring the accuracy, robustness, and practicality of the analysis conclusions.
[0008] In some embodiments, obtaining the target data table based on the initial data characteristics of the original data file includes: parsing the original data file to obtain an initial data table; identifying the field type of each field in the initial data table, wherein the field type includes: continuous, categorical, time-based, or text-based; determining attribute information of at least some fields based on the field type, wherein the attribute information includes: the distribution pattern corresponding to the continuous field, the missing value status of each field, and the outlier status; performing data preprocessing operations on the initial data table at least according to the attribute information to obtain the target data table, wherein the preprocessing operations include: filling in fields with missing values, adjusting fields with outliers, encoding conversion for categorical fields, and standardization for continuous fields.
[0009] The embodiments of this application drive adaptive data preprocessing through feature recognition based on statistical tests, automatically transforming raw data files into a standardized form suitable for statistical analysis, providing a reliable data foundation for subsequent steps, and avoiding subsequent analysis errors caused by data quality issues.
[0010] In some embodiments, the original data file is a case file, and the objective of this analysis is to compare the differences in clinical outcomes between the two drug groups. The initial data table includes the following fields: patient ID, age, gender, medication type, baseline blood pressure, multiple blood pressure records during follow-up, study cutoff time, whether lost to follow-up or chief complaint text. The continuous fields include the age and the blood pressure records, the categorical fields include the gender and the medication type, the time fields include the study cutoff time, and the text fields include the chief complaint text.
[0011] In some embodiments, the step of performing data preprocessing operations on the initial data table at least according to the attribute information to obtain the target data table includes: for fields indicated by the attribute information to have missing values, automatically selecting the corresponding filling strategy for filling according to the field type and distribution pattern of the corresponding field, wherein if the field type is continuous and the distribution pattern is skewed, the median is used for filling; if the field type is time series data, the time series method is used for filling; for fields with categorical data type, automatic encoding conversion is performed; for fields with continuous data type, automatic normalization processing is performed.
[0012] The embodiments of this application realize intelligent and adaptive data preparation process, ensure the uniformity and reliability of data quality input into subsequent analysis models, and ultimately improve the accuracy of analysis conclusions.
[0013] In some embodiments, before inputting the updated data features and the current analysis objective into a pre-built knowledge base for recommending analysis methods, the method further includes: constructing an initial mapping rule base, wherein each mapping rule in the initial mapping rule base is used to provide an analysis method and initial confidence weight corresponding to each type of field; after the i-th analysis task is completed, adjusting the confidence weight of the corresponding mapping rule according to the accuracy of the analysis results and user feedback, wherein the accuracy of the analysis results is determined by model fitting indicators and prediction accuracy; and adjusting the confidence weight of the mapping rule corresponding to the corresponding analysis method according to the reward of the reinforcement learning model for the selected analysis method.
[0014] The embodiments of this application transform the analysis method recommendation knowledge base from a static rule set into an intelligent decision-making center that can continuously learn and dynamically adjust from the effects of actual applications, thereby improving the accuracy and reliability of the analysis methods recommended by the analysis method recommendation knowledge base.
[0015] In some embodiments, the analysis method recommendation knowledge base includes multiple mapping rules, each mapping rule including: a rule identifier for uniquely identifying the corresponding mapping rule; a trigger condition, wherein the trigger condition is a logical judgment condition based on the updated data features and the current analysis objective; a set of recommended analysis methods, including one or more statistical analysis methods recommended when the trigger condition is met; a confidence weight, which is a dynamically adjusted value used to characterize the priority of using the corresponding mapping rule in the current state; and supporting evidence and update history, used to record the source of the corresponding mapping rule, the time, magnitude, and reason for each confidence weight adjustment, wherein the reason for adjustment includes at least one of feedback based on the accuracy of the analysis results, user feedback, reinforcement learning reward signals, or external literature updates.
[0016] The analytical method recommendation knowledge base of this application transforms abstract statistical knowledge into a quantifiable, traceable, and sustainably optimized intelligent decision-making system through structured rule representation and a dynamic weighting mechanism driven by multi-source feedback, providing an adaptive and interpretable reasoning basis for analytical method recommendations.
[0017] In some embodiments, the reinforcement learning decision model includes: an input layer, wherein the input layer includes a state encoder, a real-time intelligence input interface, and a historical decision interface, the state encoder being configured to fuse the updated data features with the current analysis objective to encode a state vector, the real-time intelligence input interface being used to receive the real-time performance evaluation index, and the historical policy interface being used to connect to the analysis method recommendation knowledge base to obtain an initial policy prior based on historical experience; and a decision layer, wherein the decision layer includes: an action space generator, a value function network, and a multi-objective reward synthesizer, the action space generator being used to receive a set of candidate methods output by the analysis method recommendation knowledge base and output a set of legal actions for the current round of decision, the value function network being used to receive the set of legal actions for the current round of decision and the state vector and output the estimated value of each action in the set of legal actions, and the multi-objective reward synthesizer being used to receive multiple original reward signals and synthesize a target reward according to weights.
[0018] The embodiments of this application achieve adaptive and refined analysis method selection in complex statistical analysis scenarios by integrating a comprehensive decision-making architecture that combines real-time performance pre-evaluation, historical strategy priors, and multi-objective delayed rewards. Its decision-making is not only based on the exploration of the optimal solution for the current data, but also incorporates long-term experience learning and multi-dimensional effect trade-offs.
[0019] Secondly, some embodiments of this application provide an automated data analysis apparatus, the apparatus comprising: an input module configured to acquire an original data file and the current analysis target; a target data table acquisition module configured to obtain a target data table based on initial data features of the original data file, wherein the initial data features include: field type, distribution pattern, outliers and / or missing values, and the target data table is obtained by correcting the data of corresponding fields based on the initial data features; an updated data feature acquisition module configured to acquire updated data features corresponding to the original data file based on the target data table; and a candidate analysis method set acquisition module configured to input the updated data features and the current analysis target into a pre-built analysis method recommendation knowledge base to obtain a candidate analysis method set; real-time performance. The evaluation index acquisition module is configured to perform effect simulation pre-evaluation on at least some of the analysis methods in the candidate analysis method set based on the target data table, and obtain the real-time performance evaluation index of each method; the target analysis method acquisition module is configured to input the updated data features and the state formed by the current analysis target, the action formed by the candidate analysis method set, and the real-time performance evaluation index into a pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target; the analysis result acquisition module is configured to use the target analysis method to analyze the target data table and generate analysis results; the update module is configured to update the confidence weight of the mapping relationship in the recommendation knowledge base of the analysis method based on the evaluation feedback of the analysis results, and update the strategy of the reinforcement learning decision model.
[0020] Secondly, some embodiments of this application provide a computer program readable storage medium having a computer program stored thereon, which, when executed, can implement the methods described in any of the embodiments included in the first aspect.
[0021] Thirdly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any of the embodiments included in the first aspect. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 A layered architecture diagram of an automated data analysis system provided in this application embodiment;
[0024] Figure 2 This is a flowchart of an automated data analysis method according to an embodiment of this application;
[0025] Figure 3 A block diagram of the automated data analysis apparatus provided in the embodiments of this application;
[0026] Figure 4 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0027] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0029] The inventors of this application discovered in their research that existing data analysis systems lack intelligence, are unable to dynamically recommend appropriate analysis methods based on the specific characteristics of the data (such as distribution patterns or variable types), and do not have the ability to learn from historical analysis results and optimize future decisions.
[0030] To address these technical problems, some embodiments of this application provide an automated data analysis method. This method can automatically extract data features, automatically select the best analysis method, and generate an analysis report. For example, some embodiments of this application's automated data analysis method include: automatically identifying data features (such as data type, distribution pattern, missing values, etc.), recommending analysis methods based on an analysis method recommendation knowledge base, continuously optimizing the analysis method selection decision through a reinforcement learning framework, and generating a structured report based on the selected target analysis method, thereby reducing the difficulty of data analysis.
[0031] Please refer to Figure 1 , Figure 1 An automated data analysis system provided in some embodiments of this application includes: a data input and parsing layer 110, an intelligent decision-making layer 120, an analysis execution layer 130, and a feedback adjustment layer 140.
[0032] The data input and parsing layer 110 includes a raw data input module 111, a data parsing and feature recognition module 112, and a data preprocessing module 113.
[0033] The raw data input module 111 is configured to receive raw data files uploaded by users. For example, the raw data file can be in the format of CSV or Excel.
[0034] The data parsing and feature recognition module 112 is configured to parse the original data file, identify the type of each field in the original data file (for example, the field type includes continuous, categorical, time or text, etc.), and calculate the distribution pattern, missing values and outliers of each field.
[0035] The data preprocessing module 113 is configured to automatically perform preprocessing operations such as missing value filling, outlier handling, encoding conversion or standardization based on the feature recognition results, output a standardized data table (i.e., the target data table), and generate data feature descriptions.
[0036] The intelligent decision-making layer 120 includes: a knowledge base module 121, a candidate method generation module 122, an effect pre-evaluation module 123, and a reinforcement learning decision-making module 124.
[0037] The knowledge base module 121 is configured to store mapping rules, each with a corresponding confidence weight. The confidence weights in the analysis method recommendation knowledge base stored in this knowledge base module can be dynamically updated.
[0038] The candidate method generation module 122 is configured to retrieve a set of candidate analysis methods from the knowledge base based on data characteristics and corresponding analysis objectives.
[0039] The effect pre-evaluation module 123 is configured to perform rapid simulation (such as cross-validation) on each analysis method in the candidate method set or on each of the top-ranked analysis methods to obtain real-time performance evaluation indicators corresponding to the corresponding analysis methods in the candidate method set.
[0040] The reinforcement learning decision module 124 is configured to receive a set of states (i.e., data features and analysis objectives), an action space (i.e., a set of candidate methods), and real-time performance evaluation metrics, and output the target analysis method in combination with historical strategies.
[0041] The analysis execution layer 130 includes an analysis method execution engine 131 and a report generation module 132. The analysis method execution engine 131 is configured to invoke the corresponding analysis method (e.g., survival analysis or regression analysis) to analyze the data in the preprocessed raw data file (i.e., the target data table). The report generation module 132 is configured to convert the analysis results into structured reports and visualizations, and to describe the conclusions in natural language.
[0042] The feedback adjustment layer 140 includes: a result evaluation module 141, a knowledge base update module 142, and a reinforcement learning model update module 143. The result evaluation module 141 is configured to automatically evaluate the accuracy and stability of the analysis results and collect user feedback. The knowledge base update module 142 is configured to adjust the confidence weights of the corresponding mapping rules in the analysis method recommendation knowledge base based on the evaluation results and user feedback. The reinforcement learning model update module 143 is configured to store the state, actions, and rewards (calculated by a multi-objective reward function) of this analysis task into an experience pool for updating the reinforcement learning model's strategy.
[0043] The following is combined with Figure 2 This application provides an example of an automated data analysis method according to some embodiments, the method comprising:
[0044] S210, Obtain the original data file and the target of this analysis.
[0045] S220, a target data table is obtained based on the initial data characteristics of the original data file, wherein the initial data characteristics include: field type, distribution pattern, outliers and / or missing values, and the target data table is obtained by correcting the data of the corresponding fields based on the initial data characteristics.
[0046] S230, obtain the updated data features corresponding to the original data file according to the target data table.
[0047] It should be noted that the updated data features include field type, distribution pattern, outliers and / or missing values. The difference between the updated data features and the initial data features is that the updated data features are based on the preprocessed data, while the initial data features are based on the original data in the original data file before preprocessing.
[0048] S240, the updated data features and the current analysis target are input into a pre-built analysis method recommendation knowledge base to obtain a set of candidate analysis methods.
[0049] For example, in some embodiments of this application, the analysis method recommendation knowledge base stores multiple mapping relationships, each mapping relationship being used to provide at least an analysis target, data features, and analysis methods matching the analysis target and data features. Accordingly, for this task, the analysis method recommendation knowledge base obtains a set of candidate analysis methods by searching for analysis methods matching the current analysis target and updated data features. S250, based on the target data table, at least some of the analysis methods in the candidate analysis method set are subjected to effect simulation pre-evaluation to obtain real-time performance evaluation indicators for each method.
[0050] For example, in some embodiments of this application, S250 includes: performing a preliminary ranking of each analysis method in the candidate analysis method set according to priority rules (e.g., priority rules include survival analysis > regression analysis > descriptive statistics) to obtain a preliminary ranking result; selecting the top N analysis methods (N is an integer greater than 1 and less than the total number of all analysis methods included in the candidate method set) for effect simulation pre-evaluation based on the preliminary ranking result to obtain the real-time performance evaluation index of each analysis method among the N analysis methods.
[0051] S260, the updated data features, the state constituted by the current analysis target, the actions constituted by the candidate analysis method set, and the real-time performance evaluation index are input into the pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target.
[0052] S270, The target data table is analyzed using the target analysis method to generate analysis results.
[0053] S280, based on the evaluation feedback of the analysis results, update the confidence weights of the mapping relationships in the knowledge base recommended by the analysis method, and update the strategy of the reinforcement learning decision model.
[0054] The embodiments of this application automate the selection of target analysis methods by constructing a dual-driven intelligent analysis framework of a dynamic analysis method recommendation knowledge base and multi-objective reinforcement learning decision-making. Furthermore, it introduces a closed-loop feedback mechanism based on analysis results, achieving for the first time the automation, adaptability, and continuous evolution of statistical analysis method selection. This transforms the statistical decision-making process, which relies on expert experience, into an intelligent system capable of continuous learning and optimization from the data itself and historical practice. This lowers the professional threshold while ensuring the accuracy, robustness, and practicality of the analysis conclusions.
[0055] For example, in some embodiments of this application, S220, obtaining the target data table based on the initial data characteristics of the original data file, includes: parsing the original data file to obtain an initial data table; identifying the field type of each field in the initial data table, wherein the field type includes: continuous, categorical, time-based, or text-based; determining attribute information of at least some fields based on the field type, wherein the attribute information includes: the distribution pattern corresponding to the continuous field, the missing value status of each field, and the outlier status; performing data preprocessing operations on the initial data table at least according to the attribute information to obtain the target data table, wherein the preprocessing operations include: filling in fields with missing values, adjusting fields with outliers, encoding conversion for categorical fields, and standardization for continuous fields.
[0056] For example, in some embodiments of this application, both the initial data features and the updated data features include field types and attribute information.
[0057] The embodiments of this application drive adaptive data preprocessing through feature recognition based on statistical tests, automatically transforming raw data files into a standardized form suitable for statistical analysis, providing a reliable data foundation for subsequent steps, and avoiding subsequent analysis errors caused by data quality issues.
[0058] For example, in some embodiments of this application, the original data file is a case file (which is a file that has undergone sensitive information processing and does not include personal information), and the objective of this analysis is to compare the differences in clinical outcomes between the two groups of drugs. The initial data table includes the following fields: patient ID, age, gender, medication type, baseline blood pressure, multiple blood pressure records during follow-up, major adverse cardiovascular events (MACE) (yes / no), MACE occurrence time, study cutoff time, whether lost to follow-up or chief complaint text field. The continuous field includes the age and the blood pressure record, the categorical field includes the gender and the medication type, the time field includes the study cutoff time, and the text field includes the chief complaint text.
[0059] In some embodiments of this application, the step of performing data preprocessing operations on the initial data table at least according to the attribute information to obtain the target data table includes: for fields indicated by the attribute information to have missing values, automatically selecting the corresponding filling strategy for filling according to the field type and distribution pattern of the corresponding field, wherein if the field type is continuous and the distribution pattern is skewed, the median is used for filling; if the field type is time series data, the time series method is used for filling; for fields of categorical type, automatic encoding conversion is performed; for fields of continuous type, automatic normalization processing is performed.
[0060] The embodiments of this application realize intelligent and adaptive data preparation process, ensure the uniformity and reliability of data quality input into subsequent analysis models, and ultimately improve the accuracy of analysis conclusions.
[0061] In some embodiments of this application, before inputting the updated data features and the current analysis target into a pre-built analysis method recommendation knowledge base, the method further includes: constructing an initial mapping rule base, wherein each mapping rule in the initial mapping rule base is used to provide an analysis method and initial confidence weight corresponding to each type of field; after the i-th analysis task is completed, adjusting the confidence weight of the corresponding rule according to the accuracy of the analysis results and user feedback, wherein the accuracy of the analysis results is determined by model fitting indicators and prediction accuracy, and i is an integer greater than or equal to 1; adjusting the confidence weight of the rule corresponding to the corresponding analysis method according to the reward of the reinforcement learning model for the selected analysis method.
[0062] The embodiments of this application transform the analysis method recommendation knowledge base from a static rule set into an intelligent decision-making center that can continuously learn and dynamically adjust from the effects of actual applications, thereby improving the accuracy and reliability of the analysis methods recommended by the analysis method recommendation knowledge base.
[0063] In some embodiments of this application, the analysis method recommendation knowledge base includes multiple mapping rules, each mapping rule including: a rule identifier, used to uniquely identify the corresponding mapping rule; a trigger condition, wherein the trigger condition is a logical judgment condition based on the updated data features and the current analysis objective; a set of recommended analysis methods, including one or more statistical analysis methods recommended when the trigger condition is met; a confidence weight, which is a dynamically adjusted value used to characterize the priority of using the corresponding mapping rule in the current state; supporting evidence and update history, used to record the source of the corresponding mapping rule, the time, magnitude and reason for each confidence weight adjustment, wherein the reason for adjustment includes at least one of feedback based on the accuracy of the analysis results, user feedback, reinforcement learning reward signals or external literature updates.
[0064] The knowledge base of this application transforms abstract statistical knowledge into a quantifiable, traceable, and sustainably optimized intelligent decision-making system through structured rule representation and a dynamic weighting mechanism driven by multi-source feedback, providing an adaptive and interpretable reasoning basis for the recommendation of analytical methods.
[0065] In some embodiments of this application, the reinforcement learning decision model includes: an input layer, wherein the input layer includes a state encoder, a real-time intelligence input interface, and a historical decision interface, the state encoder being configured to fuse the updated data features with the current analysis objective to encode a state vector, the real-time intelligence input interface being used to receive the real-time performance evaluation index, and the historical policy interface being used to connect to the analysis method recommendation knowledge base to obtain an initial policy prior based on historical experience; and a decision layer, wherein the decision layer includes: an action space generator, a value function network, and a multi-objective reward synthesizer, the action space generator being used to receive a set of candidate methods output by the analysis method recommendation knowledge base and output a set of legal actions for the current round of decision, the value function network being used to receive the set of legal actions for the current round of decision and the state vector and output the estimated value of each action in the set of legal actions, and the multi-objective reward synthesizer being used to receive multiple original reward signals and synthesize a target reward according to weights.
[0066] The embodiments of this application achieve adaptive and refined method selection in complex statistical analysis scenarios by integrating a comprehensive decision-making architecture that combines real-time performance pre-evaluation, historical policy priors, and multi-objective delayed rewards. Its decision-making is not only based on the exploration of the optimal solution for the current data, but also incorporates long-term experience learning and multi-dimensional effect trade-offs.
[0067] The following example illustrates some embodiments of the automated data analysis methods provided in this application.
[0068] For example, in some embodiments of this application, the method for automated data analysis includes:
[0069] Step S1: Automatic Data Identification and Preprocessing
[0070] Users upload the dataset corresponding to the original data file, and the system automatically parses the data format (such as CSV, Excel).
[0071] Automatically identify and parse the data type (continuous, categorical, time, text, etc.), distribution pattern (normal, skewed, etc.), missing value ratio, outlier information, and other attribute information of each field in the input data file.
[0072] Automatically perform the following preprocessing steps: missing value imputation (e.g., mean imputation, median imputation, or imputation using model prediction), outlier handling (e.g., outlier handling using the IQR method), encoding transformation (e.g., one-hot encoding), or data standardization.
[0073] It should be noted that in some embodiments of this application, a hierarchical processing strategy is adopted based on the identification results of the missing proportion. When the missing proportion is less than 5%, deletion or simple imputation is used; when the missing proportion is between 5% and 20%, the mean, median, or mode is used for imputation; when the missing proportion is greater than 20%, model prediction is used for imputation. In other embodiments of this application, the model prediction uses a random forest model as the basic architecture, which predicts missing features by integrating multiple decision trees. For the technical scenario of this application, the model training process includes: extracting samples from the historical complete dataset, manually generating missing data according to the preset missing proportion to construct a training set; standardizing the input features; training with a random forest model, using mean squared error (numerical) or cross-entropy loss (categorical) as the loss function; optimizing hyperparameters through grid search; and integrating the trained model into the preprocessing module.
[0074] The embodiments of this application automatically perform hierarchical processing based on the outlier detection results:
[0075] (1) Handling a small number of outliers (proportion ≤ 5%)
[0076] Direct processing method adopted:
[0077] Deletion: When the data volume is sufficient, deleting a small number of outliers has a negligible impact on subsequent analysis; simply delete the row containing the outlier.
[0078] Truncation: Use percentiles to determine boundaries. For example, set values less than the 1st percentile (P1) to P1, and values greater than the 99th percentile (P99) to P99. Alternatively, use the more robust Turkey's Fences (IQR method) boundary: [Q1 - k * IQR, Q3 + k * IQR], where IQR is the interquartile range, and k is typically 1.5 (for general outliers) or 3.0 (for extreme outliers). Values exceeding the lower bound are set as the lower bound, and values exceeding the upper bound are set as the upper bound.
[0079] (2) Handling of a large number of outliers (proportion > 15%)
[0080] Simply deleting or truncating data may distort its original distribution and result in the loss of valuable information. In this case, the system employs robust statistical methods that are insensitive to outliers:
[0081] Robust standardization: Use (x - median) / MAD instead of the traditional (x - mean) / std
[0082] Statistical substitution: Use the median and IQR instead of the mean and standard deviation for description.
[0083] Data transformation: Automatically apply logarithmic transformations, etc., to skewed data.
[0084] Step S2: Construction of the analysis method recommendation knowledge base, which consists of analysis methods and data feature mapping.
[0085] The recommended knowledge base structure for analytical methods includes mapping rules between statistical methods (or analytical methods) and data features, for example:
[0086] Mapping rule one: continuous data that conforms to a normal distribution (used to describe data characteristics) and parametric tests (as an analysis method corresponding to the aforementioned data characteristics, for example, the analysis method of parametric tests includes: t test or ANOVA, etc.).
[0087] Mapping rule two: Categorical data (an example of data features) and chi-square test or logistic regression (an example of an analytical method that corresponds to categorical data features).
[0088] Mapping rule three: Time series data with survival analysis and ARIMA model.
[0089] In some embodiments of this application, the analysis method recommends updating the knowledge base in the following ways:
[0090] Initial mapping rules were constructed based on statistics textbooks and clinical guidelines.
[0091] A user feedback mechanism is introduced to adjust the confidence weights corresponding to each mapping rule based on the accuracy of the analysis results.
[0092] We regularly integrate the latest research findings and update the analysis methods in the knowledge base through API interfaces.
[0093] Step S3: Decision-making mechanism for the best analysis method.
[0094] Multi-method screening: When data features simultaneously meet the applicable conditions of multiple analysis methods, the system performs screening by prioritizing and pre-evaluating the effects.
[0095] Priority rules: Prioritize methods based on their complexity, computational efficiency, and applicable scenarios (e.g., survival analysis > regression analysis > descriptive statistics).
[0096] Preliminary evaluation of results: The accuracy and stability of each method are preliminarily evaluated through simulation analysis (such as cross-validation), and the method with the highest score is selected.
[0097] Reinforcement learning framework: A reinforcement learning model is constructed using the accuracy of analysis results and the clinical applicability of reports as reward signals.
[0098] The state space includes: data characteristics and analysis objectives.
[0099] The action space includes: optional statistical methods.
[0100] The reward function includes a multi-objective reward design that combines accuracy, clinical interpretability, and computational efficiency.
[0101] Learning algorithm: Use Q-learning or policy gradient method to continuously optimize the method selection strategy.
[0102] Step S4: Report Generation and Output
[0103] Structured Reports: Automatically generates reports that include descriptive statistics, inferential statistics, and visualizations (such as scatter plots and heatmaps).
[0104] Natural Language Description: Use a template engine to transform statistical results into easy-to-understand textual descriptions.
[0105] Output formats: Supports PDF, Word and other formats, and can be integrated into third-party systems.
[0106] It is easy to understand that the embodiments of this application achieve the following beneficial effects through automated processes and intelligent decision-making mechanisms: Non-professional users only need to upload raw data files and select analysis targets to obtain professional-grade data reports, lowering the barrier to entry. Intelligent method selection and reinforcement learning optimization reduce errors caused by inappropriate analysis methods. Automated preprocessing and analysis method optimization shorten analysis time from hours to minutes. Continuous learning adapts to new data characteristics, enabling long-term evolution. In medical statistics, integrating clinical guideline analysis reports provides greater practical guidance value.
[0107] like Figure 3 As shown, some embodiments of this application provide an apparatus for automated data analysis. It should be understood that this apparatus is similar to the one described above. Figure 2 Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. The automated data analysis device includes: an input module 310, a target data table acquisition module 320, an update data feature acquisition module 330, a candidate analysis method set acquisition module 340, a real-time performance evaluation index acquisition module 350, a target analysis method acquisition module 360, an analysis result acquisition module 370, and an update module 380.
[0108] Input module 310 is configured to acquire the raw data file and the target of this analysis.
[0109] The target data table acquisition module 320 is configured to obtain a target data table based on the initial data characteristics of the original data file. The initial data characteristics include: field type, distribution pattern, outliers and / or missing values. The target data table is obtained by correcting the data of the corresponding fields based on the initial data characteristics.
[0110] The update data feature acquisition module 330 is configured to acquire the update data features corresponding to the original data file based on the target data table.
[0111] The candidate analysis method set acquisition module 340 is configured to input the updated data features and the current analysis target into a pre-built analysis method recommendation knowledge base to obtain a candidate analysis method set.
[0112] The real-time performance evaluation index acquisition module 350 is configured to perform effect simulation pre-evaluation on at least some of the analysis methods in the candidate analysis method set based on the target data table, and obtain the real-time performance evaluation index of each method.
[0113] The target analysis method acquisition module 360 is configured to input the updated data features, the state constituted by the current analysis target, the action constituted by the candidate analysis method set, and the real-time performance evaluation index into a pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target.
[0114] The analysis result acquisition module 370 is configured to analyze the target data table using the target analysis method and generate analysis results.
[0115] The update module 380 is configured to update the confidence weights of the mapping relationships in the recommended knowledge base of the analysis method based on the evaluation feedback of the analysis results, and to update the strategy of the reinforcement learning decision model.
[0116] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.
[0117] Some embodiments of this application provide a computer program readable storage medium having a computer program stored thereon, which, when executed, can perform the methods described in any of the above embodiments.
[0118] like Figure 4As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads and executes the program via a bus 430, it can implement the method described in any of the above embodiments.
[0119] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.
[0120] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiments of this disclosure can be used to execute the instructions in memory 410 to implement… Figure 2 The method shown. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.
[0121] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0122] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0123] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0124] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0125] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A method for automated data analysis, characterized in that, The method includes: Obtain the original data file and the objectives of this analysis; The target data table is obtained based on the initial data characteristics of the original data file, wherein the initial data characteristics include: field type, distribution pattern, outliers and / or missing values, and the target data table is obtained by correcting the data of the corresponding fields based on the initial data characteristics; Obtain the updated data features corresponding to the original data file based on the target data table; The updated data features and the current analysis objective are input into a pre-built analysis method recommendation knowledge base to obtain a set of candidate analysis methods; Based on the target data table, at least some of the analysis methods in the candidate analysis method set are subjected to effect simulation pre-evaluation to obtain the real-time performance evaluation index of each method; The updated data features, the state constituted by the current analysis target, the action constituted by the candidate analysis method set, and the real-time performance evaluation index are input into a pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target. The target data table is analyzed using the target analysis method described above, and analysis results are generated. Based on the evaluation feedback of the analysis results, the confidence weights of the mapping relationships in the knowledge base recommended by the analysis method are updated, and the strategy of the reinforcement learning decision model is updated.
2. The method as described in claim 1, characterized in that, The step of obtaining the target data table based on the initial data characteristics of the original data file includes: The original data file is parsed to obtain the initial data table; Identify the field type of each field in the initial data table, wherein the field type includes: continuous, categorical, time, or text. Based on the field type, attribute information of at least some fields is determined, wherein the attribute information includes: the distribution pattern corresponding to the continuous field, the missing value situation of each field, and the outlier situation; At least based on the attribute information, data preprocessing operations are performed on the initial data table to obtain the target data table. The preprocessing operations include: filling in fields with missing values, adjusting fields with outliers, encoding conversion for categorical fields, and standardization for continuous fields.
3. The method as described in claim 2, characterized in that, The original data file is a case file. The goal of this analysis is to compare the differences in clinical outcomes between the two drug groups. The initial data table includes the following fields: patient ID, age, gender, medication type, baseline blood pressure, multiple blood pressure records during follow-up, study cutoff time, whether lost to follow-up or chief complaint text. The continuous fields include the age and blood pressure records, the categorical fields include the gender and medication type, the time fields include the study cutoff time, and the text fields include the chief complaint text.
4. The method as described in claim 3, characterized in that, The step of performing data preprocessing operations on the initial data table based at least on the attribute information to obtain the target data table includes: For fields whose attribute information indicates the presence of missing values, the corresponding filling strategy is automatically selected based on the field type and distribution pattern of the field. Specifically, if the field type is continuous and the distribution pattern is skewed, the median is used for filling; if the field type is time series data, the time series method is used for filling. For fields with a data type of "fractional", automatic encoding conversion is performed; For fields of continuous type, normalization is performed automatically.
5. The method as described in claim 1, characterized in that, Before inputting the updated data features and the current analysis objective into the pre-built analysis method recommendation knowledge base, the method further includes: Construct an initial mapping rule base, wherein each mapping rule in the initial mapping rule base is used to provide the analysis method and initial confidence weight corresponding to each type of field; After the i-th analysis task is completed, the confidence weight of the corresponding mapping rule is adjusted based on the accuracy of the analysis results and user feedback. The accuracy of the analysis results is determined by the model fitting index and the prediction accuracy. Based on the reward of the reinforcement learning model for the selected analysis method, adjust the confidence weight of the rule corresponding to the analysis method.
6. The method as described in claim 4, characterized in that, The analysis method recommends a knowledge base that includes multiple mapping rules, each of which includes: Rule identifier, used to uniquely identify the corresponding mapping rule; Triggering conditions, wherein the triggering conditions are logical judgment conditions based on the updated data features and the current analysis target; A set of recommended analysis methods, including one or more statistical analysis methods recommended when the triggering conditions are met; The confidence weight is a dynamically adjusted value used to characterize the priority of applying the corresponding mapping rule in the current state. Supporting evidence and update history are used to record the source of the corresponding mapping rules, the time, magnitude and reason for each confidence weight adjustment, and the reasons for the adjustment include at least one of the following: feedback based on the accuracy of the analysis results, user feedback, reinforcement learning reward signals or external literature updates.
7. The method according to any one of claims 1-6, characterized in that, The reinforcement learning decision model includes: The input layer includes a state encoder, a real-time intelligence input interface, and a historical decision interface. The state encoder is configured to fuse the updated data features with the current analysis target and encode them into a state vector. The real-time intelligence input interface is used to receive the real-time performance evaluation index. The historical strategy interface is used to connect to the analysis method recommendation knowledge base to obtain initial strategy priors based on historical experience. The decision layer includes an action space generator, a value function network, and a multi-objective reward synthesizer. The action space generator receives the candidate method set output by the knowledge base recommended by the analysis method and outputs the set of legal actions for the current round of decision. The value function network receives the set of legal actions for the current round of decision and the state vector and outputs the estimated value of each action in the set of legal actions. The multi-objective reward synthesizer receives multiple original reward signals and synthesizes a target reward according to weights.
8. An automated data analysis device, characterized in that, The device includes: The input module is configured to acquire the raw data file and the target of this analysis. The target data table acquisition module is configured to obtain a target data table based on the initial data characteristics of the original data file. The initial data characteristics include: field type, distribution pattern, outliers and / or missing values. The target data table is obtained by correcting the data of the corresponding fields based on the initial data characteristics. The update data feature acquisition module is configured to acquire the update data features corresponding to the original data file based on the target data table. The candidate analysis method set acquisition module is configured to input the updated data features and the current analysis target into a pre-built analysis method recommendation knowledge base to obtain a candidate analysis method set; The real-time performance evaluation index acquisition module is configured to perform effect simulation pre-evaluation on at least some of the analysis methods in the candidate analysis method set based on the target data table, and obtain the real-time performance evaluation index of each method. The target analysis method acquisition module is configured to input the updated data features, the state constituted by the current analysis target, the action constituted by the candidate analysis method set, and the real-time performance evaluation index into a pre-trained reinforcement learning decision model to obtain the target analysis method selected for the current analysis target. The analysis result acquisition module is configured to analyze the target data table using the target analysis method and generate analysis results; The update module is configured to update the confidence weights of the mapping relationships in the knowledge base recommended by the analysis method based on the evaluation feedback of the analysis results, and to update the policy of the reinforcement learning decision model.
9. A computer program readable storage medium having a computer program stored thereon, said computer program being executed to perform the method as described in any one of claims 1-7.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it can implement the method as described in any one of claims 1-7.