Multi-level data intelligent statistical analysis system based on machine learning

Abstract

The invention belongs to the technical field of data management and analysis, and particularly relates to a multi-level data intelligent statistical analysis system based on machine learning, which comprises an intelligent statistical analysis end, an operation monitoring output end and a remote optimization early warning end, the intelligent statistical analysis end comprises a multi-source heterogeneous data adaptation preprocessing module, a multi-level feature dynamic extraction module, a multi-level data association statistical analysis module and a visual output module; the intelligent statistical analysis end is used as a core analysis carrier, and the operation monitoring output end is used as a guarantee for stable operation, so that the data processing precision, the correlation analysis depth and the result utilization efficiency are remarkably improved, and the operation stability and reliability are guaranteed through full-process monitoring; and users in various industries are effectively supported to make accurate decisions based on data analysis results, and an integrated intelligent solution is provided for data management and analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management analysis, in particular to a multi-level data intelligent statistical analysis system based on machine learning. BACKGROUND

[0002] Under the development background of deep integration of big data and artificial intelligence technology, the demand for data statistical analysis of various industries has shifted from single-dimensional information aggregation to multi-level and deep-level correlation insight. However, the current data statistical analysis technology still has many limitations in practical application, and it is difficult to meet the precise analysis and efficient operation requirements in complex scenarios.

[0003] For example, a multi-level real-time analysis processing method for multi-source data is disclosed in Chinese patent CN117786542A. Although the technical solution of this invention focuses on multi-level processing of multi-source data, it proposes to use historical multi-source data features to construct a neural network classification model, and to realize data optimization scheduling through consistency comparison. It has made some breakthroughs in multi-source data integration and real-time output efficiency, but its limitations are significant:

[0004] Firstly, the "multi-level" of the invention technical solution only focuses on the level of data classification model, and does not build a "basic layer (original detailed data) - business layer (business dimension aggregated data) - decision layer (management dimension aggregated data)" division mechanism that fits business logic. It cannot correlate different granularity data such as device sensor readings (basic layer), store operation indicators (business layer), and regional strategic targets (decision layer), and it is difficult to output decision-making basis with business guidance;

[0005] Secondly, the invention technical solution focuses on data processing and scheduling, and cannot real-time perceive the processing delay caused by the surge of multi-source data, nor can it identify the impact of faults on the analysis process. It is prone to "analysis interruption" or "result lag" problems, which is not conducive to meeting the demand for stable operation of the whole process. Therefore, a solution is proposed. SUMMARY

[0006] The purpose of the present application is to provide a multi-level data intelligent statistical analysis system based on machine learning to solve the technical defects proposed in the background art.

[0007] To achieve the above purpose, the present application provides the following technical solution: a multi-level data intelligent statistical analysis system based on machine learning, comprising an intelligent statistical analysis end, a running monitoring output end and a remote optimization warning end. The running monitoring output end monitors and analyzes the intelligent statistical analysis end, and sends the monitoring and analysis results to the remote optimization warning end.

[0008] The intelligent statistical analysis end comprises a multi-source heterogeneous data adaptive preprocessing module, a multi-level feature dynamic extraction module, a multi-level data correlation statistical analysis module and a visualization output module.

[0009] The multi-source heterogeneous data adaptive preprocessing module is used for adapting multi-source data, and outputs regular multi-level basic data sets classified according to a basic layer, a business layer and a decision layer through intelligent cleaning, format standardization and missing value repair; the multi-level feature dynamic extraction module extracts basic statistical features, business derived features and high-order fusion features in layers based on the multi-level basic data sets, and generates multi-level feature sets adapted to different analysis scenes through feature redundancy elimination and importance sorting;

[0010] The multi-level data correlation statistical analysis module identifies potential correlations of data in different layers through intra-layer statistical analysis combined with cross-layer correlation mining based on the multi-level feature sets, and outputs multi-dimensional statistical analysis reports; the visualization output module converts the multi-dimensional statistical analysis reports of the multi-level data correlation statistical analysis module into an interactive visualization interface, supports a user to view, filter and export analysis results, and receives user-defined parameter adjustment instructions.

[0011] Further, the multi-source heterogeneous data adaptive preprocessing module receives external data through a built-in multi-source data interface, and simultaneously starts a data type intelligent identification engine to determine the type of the input data, automatically parses field attributes and checks field integrity for structured data, extracts key fields and converts them into a structured format for semi-structured data through Schema automatic matching technology, and extracts business-related entities and converts them into structured key-value pairs for unstructured text data using BERT-based entity recognition and word segmentation technology;

[0012] The intelligent cleaning engine is started, and for numerical data, outliers are identified and marked through the interquartile range method, and the outliers are automatically selected to be deleted or repaired based on the neighborhood mean value in combination with the business scene to which the data belongs; for missing values, the mean or median filling, mode filling or LSTM-based time series interpolation is used according to the importance of the field; after cleaning, the data is classified according to the data level rule library, and finally the regular multi-level basic data set is output and transmitted to the multi-level feature dynamic extraction module.

[0013] Further, the multi-level feature dynamic extraction module receives the multi-level basic data set, starts a basic feature extraction unit for the basic layer data, calculates statistical features including mean, variance, extreme value and fluctuation frequency for numerical basic data, extracts time features including hour segment, day number in the week and holiday mark for time-based basic data, and converts categorical basic data into numerical features through one-hot encoding;

[0014] For business layer data, a business-derived feature extraction unit is initiated, which calls the built-in business rule library to generate business-related features. For decision layer data, a high-order fusion feature extraction unit is initiated, which concatenates the features extracted from the basic layer and the business layer, uses mutual information-based feature redundancy removal, calculates feature importance using the LightGBM algorithm, retains the top 80% of highly important features, and compresses high-dimensional features using PCA dimensionality reduction technology. Finally, the basic layer feature subset, business layer feature subset, and decision layer feature subset are integrated into a multi-level feature set and transmitted to the multi-level data association statistical analysis module.

[0015] Furthermore, the multi-level data association statistical analysis module receives multi-level feature sets, initiates statistical analysis units within each level, performs descriptive statistics and trend analysis on basic layer features; performs comparative analysis and anomaly attribution analysis on business layer features; and performs macro trend prediction and target achievement analysis on decision-making layer features.

[0016] The cross-level association mining unit is launched, and the Apriori algorithm based on association rules is used to mine the association relationships between data at different levels, calculate the support and confidence, and retain strong association rules with support >10% and confidence >80%. At the same time, abnormal data within the level is associated with cross-level features to locate the core cause of the anomaly. Finally, the statistical results within the level and the cross-level association rules are integrated to generate a multi-dimensional statistical analysis report, which is transmitted to the visualization interactive output module.

[0017] Furthermore, the operation monitoring output includes a task load monitoring module and a continuous stability monitoring module. The task load monitoring module monitors and analyzes the load status of the intelligent statistical analysis terminal and sends the load monitoring and analysis results to the remote optimization and early warning terminal. The continuous stability monitoring module monitors and analyzes the continuous stability status of the intelligent statistical analysis terminal and sends the continuous stability monitoring and analysis results to the remote optimization and early warning terminal.

[0018] Furthermore, the specific analysis process of the task load monitoring module includes:

[0019] The total number of tasks Ntotal within the statistical period is collected. The task concurrency C is calculated using the formula C=Ntotal / tunit, where tunit is the statistical time unit. A time trend curve of task concurrency is plotted to identify the daily and weekly load peak periods. It is determined whether the task concurrency during the load peak period exceeds 80% of the designed maximum concurrency. If the task concurrency during the load peak period exceeds 80% of the designed maximum concurrency, it is marked as "potential overload risk".

[0020] The backlog of unprocessed tasks is collected from the task queue status table and the task queue backlog rate Q is calculated using the formula Q=Nbacklog / Ntotal. The CPU resource utilization rate R1 is calculated by comparing the actual CPU usage with the corresponding total configuration, and the memory resource utilization rate R2 is calculated by comparing the actual memory usage with the corresponding total configuration.

[0021] By combining the task queue backlog rate Q with the resource utilization rates R1 and R2, the backlog causes are analyzed. If Q > 5% and R1 > 80% or R2 > 70%, it is determined that "insufficient resources lead to backlog"; if Q > 5% and the resource utilization rate is normal, it is determined that "a sudden increase in task complexity leads to processing delay".

[0022] Furthermore, the specific analysis process of the task load monitoring module also includes:

[0023] The time from submission to output of each task is collected and marked as ti, where i is the task number; the average response latency D of the task is calculated, and when the average response latency D of the task exceeds the latency threshold of the corresponding scenario, the source of the latency is located by combining the proportion of task types.

[0024] Furthermore, the specific analysis process for the continuous stability monitoring module includes:

[0025] The data received by the target module (Datain) and the data output by the source module (Dataout) are collected in the intelligent statistical analysis terminal. The data transmission integrity coefficient is calculated using the formula I = (Datain / Dataout) × 100%. The data transmission integrity coefficient is then compared with a preset data transmission integrity coefficient threshold. If the data transmission integrity coefficient does not exceed the preset data transmission integrity coefficient threshold, a transmission anomaly symbol YL-1 is assigned.

[0026] The number of times the abnormal transmission symbol YL-1 is generated within a unit time is obtained and marked as the transmission anomaly statistical coefficient. The integrity feature value is obtained by averaging the ratios of all data transmission integrity coefficients within a unit time to a preset data transmission integrity coefficient threshold. If the transmission anomaly statistical coefficient exceeds the preset transmission anomaly statistical coefficient threshold or the integrity feature value does not exceed the preset integrity feature threshold, a continuous stability alarm signal is generated.

[0027] Furthermore, if the output anomaly statistical coefficient does not exceed the preset output anomaly statistical coefficient threshold and the integrity characteristic value exceeds the preset integrity characteristic threshold, then the continuous stability decision value is obtained through continuous stability hazard analysis. The continuous stability decision value is then compared with the preset continuous stability decision threshold. If the continuous stability decision value exceeds the preset continuous stability decision threshold, then a continuous stability alarm signal is generated.

[0028] Furthermore, the specific analysis process for the ongoing stability hazard analysis is as follows:

[0029] The number of times the intelligent statistical analysis terminal fails to operate due to a fault within a unit of time is collected and marked as the fault occurrence rate. The total duration of the intelligent statistical analysis terminal's failure due to a fault within a unit of time is marked as the fault time measurement value. The number of times the duration of a single failure due to a fault exceeds a preset single duration threshold within a unit of time is marked as the fault high frequency value. The continuous stability decision value is obtained by weighted summation of the fault occurrence rate, fault time measurement value, and fault high frequency value.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] 1. In this invention, an intelligent statistical analysis terminal is used as the core analysis carrier. The multi-source heterogeneous data adaptation and preprocessing module organizes various types of heterogeneous data into multi-level datasets. The multi-level feature dynamic extraction module extracts basic statistical, business-derived, and high-order fusion features in layers. Combined with redundancy elimination and dimensionality reduction techniques, a feature set adapted to the scenario is generated. The multi-level data association statistical analysis module accurately locates the cause of anomalies and outputs multi-dimensional reports, significantly improving the accuracy of data processing, the depth of association analysis, and the efficiency of result utilization. This effectively supports users in various industries to make accurate decisions based on data analysis results.

[0032] 2. In this invention, the operation monitoring output terminal serves as a guarantee for stable operation. The task load monitoring module identifies load peaks and backlog causes to prevent the intelligent statistical analysis terminal from reducing efficiency due to overload or delay. The continuous stability monitoring module can promptly detect transmission anomalies and potential faults, preventing operation interruptions from affecting the analysis process. This facilitates rapid intervention and optimization, ensuring the operational stability and reliability of the intelligent statistical analysis terminal. Attached Figure Description

[0033] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0034] Figure 1 This is a system block diagram of Embodiment 1 of the present invention;

[0035] Figure 2 This is a system block diagram of Embodiments 2 and 3 of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1: As Figure 1 As shown, the multi-level intelligent statistical analysis system based on machine learning proposed in this invention includes an intelligent statistical analysis terminal, an operation monitoring output terminal, and a remote optimization early warning terminal. The operation monitoring output terminal monitors and analyzes the intelligent statistical analysis terminal and sends the monitoring and analysis results to the remote optimization early warning terminal.

[0038] The intelligent statistical analysis module includes a multi-source heterogeneous data adaptation and preprocessing module, a multi-level feature dynamic extraction module, a multi-level data association statistical analysis module, and a visualization output module.

[0039] The multi-source heterogeneous data adaptation and preprocessing module is used to adapt structured (such as database tables and Excel spreadsheets), semi-structured (such as JSON and XML logs), and unstructured (such as text reports and audio-to-text transcription) multi-source data. Through intelligent cleaning, format standardization, and missing value repair, it outputs a well-organized multi-level basic dataset classified into "basic layer - business layer - decision layer". This solves the problem of subsequent analysis errors caused by low quality and inconsistent formats of multi-source data, and avoids the impact of defects in the original data on the accuracy of the analysis results.

[0040] Specifically, the multi-source heterogeneous data adaptation and preprocessing module first receives external data through its built-in multi-source data interface (supporting JDBC, API, FTP, and local file reading). Simultaneously, it activates the "data type intelligent recognition engine" to determine the type of the input data: for structured data, it automatically parses field attributes (numeric, character, and time types) and verifies field integrity; for semi-structured data, it extracts key fields using schema automatic matching technology and converts them into a structured format; for unstructured text data, it employs BERT-based entity recognition and word segmentation technology to extract business-related entities (such as "transaction amount" and "device number") and converts them into structured key-value pairs.

[0041] Subsequently, the intelligent cleaning engine is activated: for numerical data, outliers are identified and marked using the interquartile range (IQR) method, and the system automatically selects "delete outliers" or "repair based on neighborhood mean" based on the business scenario to which the data belongs (such as financial transactions or equipment monitoring); for missing values, the system uses "mean / median filling" (numerical type), "mode filling" (character type), or "LSTM-based time series interpolation" (time series type) according to the importance of the field (preset by the user or determined by the system through business rules).

[0042] After cleaning, the data is classified according to the "data hierarchy rule library" (which has built-in general hierarchy classification rules for various industries and supports user customization): basic layer (raw detailed data, such as a single transaction record), business layer (data aggregated by business dimension, such as the daily transaction amount of a store), and decision layer (data aggregated by management dimension, such as the quarterly revenue of a region). Finally, a well-organized multi-level basic dataset is output and transmitted to the multi-level feature dynamic extraction module.

[0043] The multi-level feature dynamic extraction module is based on a multi-level basic dataset and extracts basic statistical features, business-derived features, and high-order fusion features in a hierarchical manner. Through feature redundancy elimination and importance ranking, it generates multi-level feature sets adapted to different analysis scenarios. This not only solves the problems of "insufficient representativeness and high redundancy" of traditional single feature extraction, but also generates multi-level feature sets adapted to different analysis scenarios, reducing the computational load of subsequent correlation analysis. At the same time, it ensures the adaptability of features to business scenarios, providing accurate and efficient feature input for multi-level data correlation statistical analysis.

[0044] Specifically, after receiving the multi-level basic dataset, the multi-level feature dynamic extraction module first activates the "basic feature extraction unit" for the "basic layer data": for numerical basic data (such as device sensor readings), it calculates statistical features such as mean, variance, extreme values, and fluctuation frequency; for time-based basic data (such as transaction time), it extracts time features such as hourly segments, number of days in a week, and holiday markers; for categorical basic data (such as user gender and device model), it converts them into numerical features through one-hot encoding.

[0045] Next, the "Business Derivative Feature Extraction Unit" is launched for the "Business Layer Data": the built-in "Business Rule Library" (which supports preset rules for multiple industries such as finance, manufacturing, and retail, and can be customized and expanded) is called to generate business-related features; for example, in the retail scenario, "Transaction Amount per Unit Area" is derived based on "Daily Transaction Amount of Stores" and "Store Area", and "Monthly Total Consumption of Users" is derived based on "User Purchase Frequency" and "Average Order Value";

[0046] Subsequently, a "high-order fusion feature extraction unit" is initiated for the "decision-level data": features extracted from the basic layer and business layer are concatenated, feature redundancy is eliminated based on mutual information (redundant features with mutual information values ​​> 0.8 are eliminated), feature importance is calculated using the LightGBM algorithm, and the top 80% of highly important features are retained. At the same time, PCA dimensionality reduction technology is combined to compress high-dimensional features, reducing the computational load of subsequent association analysis. Finally, the "basic layer feature subset - business layer feature subset - decision-level feature subset" are integrated into a multi-level feature set and transmitted to the multi-level data association statistical analysis module.

[0047] The multi-level data association statistical analysis module is based on multi-level feature sets. Through intra-level statistical analysis combined with cross-level association mining, it identifies potential associations between data at different levels and outputs multi-dimensional statistical analysis reports. This effectively breaks through the limitations of traditional analysis, such as "hierarchical fragmentation and insufficient depth of association mining". It can not only identify potential associations between data at different levels, but also accurately locate the core causes of anomalies. The output multi-dimensional statistical analysis reports are more in line with actual business needs, greatly improving the business guidance of statistical analysis results and helping to make more accurate decisions.

[0048] Specifically, the multi-level data association statistical analysis module first receives multi-level feature sets (feature subsets of the basic layer, business layer, and decision layer), and then starts the "intra-level statistical analysis unit": for the basic layer features, it performs descriptive statistics (such as the distribution histogram of device sensor readings, frequency statistics of transaction records) and trend analysis (such as plotting the change curve of basic data according to time series).

[0049] For business-level characteristics, conduct comparative analysis (such as horizontal comparison of business indicators across different stores, and vertical comparison of the same store across different months) and anomaly attribution analysis (such as checking whether a sudden drop in transaction volume at a store is caused by "decreased average order value" or "reduced transaction frequency" in the associated basic transaction records); for decision-making-level characteristics, conduct macro-trend prediction (such as predicting annual revenue based on quarterly revenue data) and target achievement analysis (such as comparing the gap between actual quarterly revenue and preset targets).

[0050] Next, the module activates the "Cross-Level Association Mining Unit": using the Apriori algorithm based on association rules, it mines the association relationships between data at different levels. For example, in a manufacturing scenario, it mines the association rules between "basic layer equipment vibration frequency > threshold" (basic layer) and "business layer equipment failure frequency increases" (business layer) and "decision layer equipment maintenance cost increases" (decision layer). It calculates the support (the proportion of frequency of rule occurrence) and confidence (the probability of rule being true), and retains strong association rules with support > 10% and confidence > 80%.

[0051] Simultaneously, abnormal data within a level is associated with cross-level features. For example, "abnormally low sales in a certain area of ​​the business layer" is associated with "purchase frequency and average order value of users in that area of ​​the basic layer" to pinpoint the core reasons for the anomaly (such as user churn or insufficient promotional efforts). Finally, "statistical results within a level" and "cross-level association rules" are integrated to generate a multi-dimensional statistical analysis report (including data tables, trend charts, and a list of association rules).

[0052] The visualization output module transforms the multi-dimensional statistical analysis reports from the multi-level data association statistical analysis module into an interactive visualization interface, allowing users to view, filter, and export analysis results, and receive user-defined parameter adjustment instructions, thus achieving the goal of "making analysis results intuitive and user operations convenient".

[0053] Example 2: Figure 2 As shown, the difference between this embodiment and Embodiment 1 is that the operation monitoring output terminal includes a task load monitoring module. This module monitors and analyzes the load status of the intelligent statistical analysis terminal and sends the results to the remote optimization and early warning terminal. This allows for real-time monitoring of the load status of the intelligent statistical analysis terminal, preventing a decrease in operating efficiency due to excessive load or processing delays, and ensuring that the intelligent statistical analysis terminal maintains a stable processing rhythm under different task loads. The specific analysis process of the task load monitoring module is as follows:

[0054] The total number of tasks Ntotal within the statistical period is collected. The task concurrency C is calculated using the formula C=Ntotal / tunit, where tunit is the statistical time unit. A time trend curve of task concurrency is plotted to identify the daily / weekly peak load periods (e.g., 8-10 pm for retail systems). It is determined whether the task concurrency during the peak load period exceeds 80% of the designed maximum concurrency. If the task concurrency during the peak load period exceeds 80% of the designed maximum concurrency, it is marked as "potential overload risk".

[0055] The backlog of unprocessed tasks is collected from the task queue status table and the task queue backlog rate Q is calculated using the formula Q=Nbacklog / Ntotal. The CPU resource utilization rate R1 is calculated by comparing the actual CPU usage with the corresponding total configuration, and the memory resource utilization rate R2 is calculated by comparing the actual memory usage with the corresponding total configuration.

[0056] By combining the task queue backlog rate Q with the resource utilization rates R1 and R2, the backlog causes are analyzed. If Q > 5% and R1 > 80% or R2 > 70%, it is determined that "insufficient resources lead to backlog". If Q > 5% and the resource utilization rate is normal, it is determined that "a sudden increase in task complexity (such as unstructured data accounting for > 60%) leads to processing delays".

[0057] Furthermore, the task load monitoring module is also used to collect the time from submission to output of each task and mark it as ti, where i is the task number;

[0058] Through formula The average response delay D of the task is calculated.

[0059] Where n represents the total number of tasks completed within the statistical period;

[0060] When the average response latency D of a task exceeds the corresponding scenario latency threshold (e.g., 30 seconds in a retail scenario), the source of the latency is located by combining the proportion of task types (e.g., the proportion of unstructured data). For example, the multi-level feature dynamic extraction module takes too long to process unstructured text.

[0061] Example 3: Figure 2 As shown, the difference between this embodiment and Embodiments 1 and 2 is that the operation monitoring output terminal also includes a continuous stability monitoring module. This module monitors and analyzes the continuous stability of the intelligent statistical analysis terminal and sends the results to the remote optimization and early warning terminal. This effectively solves the problem of missing stability monitoring during the operation of the intelligent statistical analysis terminal, enabling timely detection of data transmission anomalies and potential faults, preventing operation analysis interruptions due to stability issues, and ensuring the long-term continuous and stable operation of the intelligent statistical analysis terminal. The specific analysis process is as follows:

[0062] The data received by the target module (Datain) and the data output by the source module (Dataout) are collected in the intelligent statistical analysis terminal. The data transmission integrity coefficient is calculated using the formula I = (Datain / Dataout) × 100%. The data transmission integrity coefficient is compared with the preset data transmission integrity coefficient threshold. If the data transmission integrity coefficient does not exceed the preset data transmission integrity coefficient threshold, it indicates that the data transmission loss is serious, and the transmission anomaly symbol YL-1 is assigned.

[0063] The number of times the transmission anomaly symbol YL-1 is generated per unit time is obtained and marked as the transmission anomaly statistical coefficient. The integrity feature value is obtained by averaging the ratios of all data transmission integrity coefficients per unit time to the preset data transmission integrity coefficient threshold.

[0064] The output anomaly statistical coefficient and integrity feature value are compared with the preset output anomaly statistical coefficient threshold and the preset integrity feature threshold respectively. If the output anomaly statistical coefficient exceeds the preset output anomaly statistical coefficient threshold or the integrity feature value does not exceed the preset integrity feature threshold, it indicates that the continuous stability performance of the intelligent statistical analysis terminal is poor, and a continuous stability alarm signal is generated.

[0065] Furthermore, if the output error statistical coefficient does not exceed the preset output error statistical coefficient threshold and the integrity feature value exceeds the preset integrity feature threshold, then the number of times the intelligent statistical analysis terminal cannot operate due to a fault within a unit time is collected and marked as the fault occurrence rate, and the total duration of the intelligent statistical analysis terminal being unable to operate due to a fault within a unit time is marked as the fault time measurement value, and the single duration of the inability to operate due to a fault within a unit time is compared with the preset single duration threshold, and the number of occurrences exceeding the preset single duration threshold is marked as the high frequency fault value;

[0066] The decision value for sustained stability is obtained by weighted summation of failure incidence rate, failure time measurement and failure high frequency value. Specifically, the failure incidence rate, failure time measurement and failure high frequency value are assigned corresponding preset weight coefficients, and the failure incidence rate, failure time measurement and failure high frequency value are multiplied by the corresponding preset weight coefficients. The sum of the three sets of product results is marked as the decision value for sustained stability.

[0067] It should be noted that the larger the value of the continuous stability decision value, the worse the overall continuous stability performance of the intelligent statistical analysis terminal. The continuous stability decision value is compared with the preset continuous stability decision threshold. If the continuous stability decision value exceeds the preset continuous stability decision threshold, it indicates that the overall continuous stability performance of the intelligent statistical analysis terminal is poor, and a continuous stability alarm signal is generated.

[0068] The working principle of this invention is as follows: During use, the intelligent statistical analysis terminal serves as the core analysis carrier. The multi-source heterogeneous data adaptation and preprocessing module, through intelligent cleaning, format standardization, and missing value repair, organizes various types of heterogeneous data into multi-level datasets. The multi-level feature dynamic extraction module extracts basic statistical, business-derived, and high-order fusion features layer by layer, and generates feature sets suitable for the specific scenario by combining redundancy removal and dimensionality reduction techniques. The multi-level data association statistical analysis module, through multi-dimensional statistics within each level and strong cross-level correlation mining, accurately locates the causes of anomalies and outputs multi-dimensional reports with strong business guidance. Furthermore, the operation monitoring output terminal ensures stable operation. The task load monitoring module identifies load peaks and backlog causes to prevent the intelligent statistical analysis terminal from reducing efficiency due to overload or delay. The continuous stability monitoring module can promptly detect transmission anomalies and potential faults, preventing operational interruptions from affecting the analysis process and assisting maintenance personnel in quickly intervening for optimization. This not only significantly improves data processing accuracy, association analysis depth, and result utilization efficiency, but also ensures operational stability and reliability through full-process monitoring, effectively supporting users in various industries to make accurate decisions based on data analysis results, and providing an integrated intelligent solution for data management and analysis.

[0069] In this invention, the threshold, preset value, or preset range settings are for result comparison and analysis to determine whether the result is good or bad. The magnitude of these values ​​is determined by a combination of large-scale model analysis of sample data and human experience, and can also be appropriately adjusted based on seasonal or common-sense influence conditions. Similarly, the preset weight coefficients and influence factors are assigned specific values ​​based on the magnitude of each parameter's influence on the result, ultimately reflecting the impact on the result. These settings are also determined by a combination of large-scale model analysis of sample data and human experience, and can also be appropriately adjusted based on seasonal or common-sense influence conditions.

[0070] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize it. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-level intelligent statistical analysis system for data based on machine learning, characterized in that: It includes an intelligent statistical analysis terminal, an operation monitoring output terminal, and a remote optimization early warning terminal. The operation monitoring output terminal monitors and analyzes the intelligent statistical analysis terminal and sends the monitoring and analysis results to the remote optimization early warning terminal. The intelligent statistical analysis module includes a multi-source heterogeneous data adaptation and preprocessing module, a multi-level feature dynamic extraction module, a multi-level data association statistical analysis module, and a visualization output module. The multi-source heterogeneous data adaptation and preprocessing module outputs a well-organized multi-level basic dataset classified into basic layer, business layer, and decision layer through intelligent cleaning, format standardization, and missing value repair. The multi-level feature dynamic extraction module is based on a multi-level basic dataset, extracting basic statistical features, business-derived features, and high-order fusion features at different levels. Through feature redundancy removal and importance ranking, it generates a multi-level feature set. The multi-level data association statistical analysis module is based on the multi-level feature set. Through intra-level statistical analysis combined with cross-level association mining, it identifies potential associations between data at different levels and outputs a multi-dimensional statistical analysis report. The visualization output module transforms the multi-dimensional statistical analysis report into an interactive visualization interface, allowing users to view, filter, and export the analysis results.

2. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 1, characterized in that, The multi-source heterogeneous data adaptation and preprocessing module receives external data through the built-in multi-source data interface, starts the data type intelligent recognition engine to determine the type of the input data, and starts the intelligent cleaning engine. After cleaning, the data is classified according to the data hierarchy rule library, and finally outputs a well-organized multi-level basic dataset, which is then transmitted to the multi-level feature dynamic extraction module.

3. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 2, characterized in that, The multi-level feature dynamic extraction module receives multi-level basic datasets, initiates basic feature extraction unit for basic layer data, initiates business-derived feature extraction unit for business layer data, and initiates high-order fusion feature extraction unit for decision layer data, and finally integrates the basic layer feature subset, business layer feature subset, and decision layer feature subset into a multi-level feature set.

4. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 1, characterized in that, The multi-level data association statistical analysis module receives multi-level feature sets, initiates intra-level statistical analysis units and cross-level association mining units, and associates intra-level abnormal data with cross-level features to locate the core causes of the anomalies; finally, it integrates intra-level statistical results with cross-level association rules to generate a multi-dimensional statistical analysis report.

5. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 1, characterized in that, The operation monitoring output includes a task load monitoring module and a continuous stability monitoring module. The task load monitoring module monitors and analyzes the load status of the intelligent statistical analysis terminal and sends the load monitoring and analysis results to the remote optimization and early warning terminal. The continuous stability monitoring module monitors and analyzes the continuous stability of the intelligent statistical analysis terminal and sends the results of the continuous stability monitoring and analysis to the remote optimization and early warning terminal.

6. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 5, characterized in that, The specific analysis process of the task load monitoring module includes: Plot the time trend curve of task concurrency to identify daily and weekly load peak periods. If the task concurrency during the load peak period exceeds 80% of the designed maximum concurrency, it is marked as "potential overload risk". Calculate the task queue backlog rate Q, CPU resource utilization R1, and memory resource utilization R2. If Q > 5% and R1 > 80% or R2 > 70%, it is determined as "backlog caused by insufficient resources". If Q > 5% and resource utilization is normal, it is determined as "processing delay caused by a sudden increase in task complexity".

7. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 6, characterized in that, The specific analysis process of the task load monitoring module also includes: The time from submission to output of each task is collected and marked as ti, where i is the task number; the average response latency D of the task is calculated, and when the average response latency D of the task exceeds the latency threshold of the corresponding scenario, the source of the latency is located by combining the proportion of task types.

8. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 5, characterized in that, The specific analysis process of the continuous stability monitoring module is as follows: obtain the number of times the transmission anomaly symbol YL-1 is generated per unit time and mark it as the transmission anomaly statistical coefficient; and calculate the mean of the ratio of all data transmission integrity coefficients per unit time to the preset data transmission integrity coefficient threshold to obtain the integrity feature value. If the output anomaly statistical coefficient exceeds the preset output anomaly statistical coefficient threshold or the integrity characteristic value does not exceed the preset integrity characteristic threshold, a continuous stability alarm signal will be generated.

9. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 8, characterized in that, If the output anomaly statistical coefficient does not exceed the preset output anomaly statistical coefficient threshold and the integrity characteristic value exceeds the preset integrity characteristic threshold, then the continuous stability decision value is obtained through continuous stability hazard analysis. If the continuous stability decision value exceeds the preset continuous stability decision threshold, a continuous stability alarm signal is generated.

10. The multi-level intelligent statistical analysis system for data based on machine learning according to claim 9, characterized in that, The specific analysis process for the analysis of potential ongoing stability risks is as follows: The number of times the intelligent statistical analysis terminal fails to operate due to a fault within a unit of time is collected and marked as the fault occurrence rate. The total duration of the intelligent statistical analysis terminal's failure due to a fault within a unit of time is marked as the fault time measurement value. The number of times the duration of a single failure due to a fault exceeds a preset single duration threshold within a unit of time is marked as the fault high frequency value. The continuous stability decision value is obtained by weighted summation of the fault occurrence rate, fault time measurement value, and fault high frequency value.

Images