An intelligent feature exploration system and method for structured data

Abstract

The present invention relates to the field of data processing technology, and in particular to an intelligent feature exploration system and method for structured data. The system comprises: a preprocessing module that obtains a structured data set; a feature acquisition module that analyzes the structured data set to obtain structural features and type features; an algorithm determination module that calculates an actual evaluation value based on data structure features and data type features, compares it with a preset evaluation value, and determines an execution algorithm unit; a generation module that generates an initial execution workflow according to preset operation rules; an allocation module that divides the initial execution workflow, calculates task complexity and allocates several processing nodes, monitors real-time load conditions and allocates computing power; a monitoring module that executes the computing power allocation results, collects operation data and calculates an actual health score; and an adjustment module that compares the actual health score with a preset health score, adjusts the initial execution workflow, and obtains a target execution workflow. The present invention improves the efficiency of the data analysis and processing process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent feature exploration system and method for structured data. Background Art

[0002] Data preprocessing and feature engineering are crucial in machine learning, especially when processing structured data. However, this process often faces numerous challenges. First, the raw datasets uploaded by users often cannot directly meet specific analysis requirements and require a series of complex processing steps, including data cleaning, transformation, feature engineering, and synthetic data generation. Second, faced with complex structured data, AI data engineers often find it difficult to intuitively determine which features are effective. They need to try a large number of different data feature processing algorithms and evaluate the effectiveness of each method to determine the final feature set for deep learning.

[0003] The patent document with Chinese patent application number CN112579582A discloses a data exploration method and system for a data analysis engine. The method includes collecting external big data and cleaning and structuring the external big data to obtain a structured data set; characterizing the structured data in the structured data set to obtain the characteristic value of each structured data and form a characteristic value set; and performing data exploration on the characteristic value set based on a deep learning data model to obtain data exploration results.

[0004] The existing technology requires writing and managing a large number of processing programs based on feature algorithms. For different data sets or different features in the same data set, program parameters need to be frequently adjusted, and a lot of time is spent on modifying the input parameters of existing algorithms, which reduces the efficiency of the data feature exploration process. Summary of the Invention

[0005] To this end, the present invention provides an intelligent feature exploration system and method for structured data, which can solve the problem of low feature exploration efficiency.

[0006] To achieve the above objectives, the present invention provides an intelligent feature exploration system for structured data, the system comprising:

[0007] The preprocessing module is used to obtain the initial data set uploaded by the user, preprocess the initial data set to obtain a structured data set;

[0008] a feature acquisition module, connected to the preprocessing module, for analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results;

[0009] an algorithm determination module, connected to the feature acquisition module, configured to calculate an actual evaluation value based on the data structure feature and the data type feature, compare the actual evaluation value with preset evaluation values of a plurality of preset algorithm units, and determine an execution algorithm unit according to the comparison result;

[0010] A generation module, connected to the algorithm determination module, for generating an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0011] an allocation module, connected to the generation module, configured to divide the initial execution workflow into a plurality of subtasks, calculate the task complexity corresponding to the plurality of subtasks, allocate a plurality of processing nodes based on the task complexity, monitor the real-time load conditions corresponding to the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result;

[0012] a monitoring module, connected to the allocation module, configured to execute the computing power allocation result through each processing node, monitor the operating status of each processing node in real time, collect operating data, and calculate the actual health score of the initially executed workflow based on the operating data;

[0013] An adjustment module is connected to the monitoring module and is used to compare the actual health score with a preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0014] Furthermore, the preprocessing module includes:

[0015] an integrity detection unit, configured to detect the integrity of the initial data set and obtain an integrity detection result;

[0016] a correctness detection unit, configured to detect the correctness of the initial data set and obtain a correctness detection result;

[0017] a calibration unit, configured to calibrate the initial data set according to the integrity test result and the correctness test result to obtain a calibration data set;

[0018] A processing unit is connected to the calibration unit and is used to perform structured processing on the calibration data set to obtain the structured data set.

[0019] Furthermore, the feature acquisition module includes:

[0020] a type determination unit, configured to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types;

[0021] an identification unit, configured to identify data distribution characteristics and data relationship characteristics of the structured data set;

[0022] A structure determination unit is connected to the identification unit and is used to determine the structure feature based on the data distribution feature and the data relationship feature.

[0023] Furthermore, the identification unit includes:

[0024] A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set;

[0025] a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness, and the kurtosis;

[0026] The correlation determination subunit is used to calculate the actual correlation coefficient corresponding to each field in the structured data set based on a correlation analysis algorithm, and determine the data relationship feature based on a number of actual correlation coefficients.

[0027] Furthermore, the algorithm determination module includes:

[0028] a first calculation unit, configured to calculate an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness, and the kurtosis;

[0029] a second calculation unit, configured to determine an actual correlation coefficient evaluation value according to the actual correlation coefficients;

[0030] A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

[0031] Furthermore, the generation module includes:

[0032] A first determining unit, configured to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit;

[0033] A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

[0034] Furthermore, the allocation module includes:

[0035] A division unit, configured to divide the initial execution workflow according to a plurality of work nodes to obtain a plurality of subtasks;

[0036] a complexity calculation unit, configured to determine an actual data volume and an actual data type of the subtask, compare the actual data volume and the actual data type with a historical execution database, obtain a historical average processing time and a historical resource consumption, and determine the task complexity based on the historical average processing time and the historical resource consumption;

[0037] a node allocation unit, connected to the complexity calculation unit, for allocating the subtasks to corresponding processing nodes according to the task complexity;

[0038] The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load conditions of the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result.

[0039] Furthermore, the monitoring module includes:

[0040] A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor;

[0041] An operation monitoring unit is used to monitor the performance parameters of the task in real time, wherein the performance parameters include average response time and resource utilization;

[0042] A score calculation unit is used to determine a task completion rate based on the running status, and calculate an actual health score based on the task completion rate, average response time and resource utilization.

[0043] Furthermore, the adjustment module includes:

[0044] a comparing unit, configured to compare the actual health score with the preset health score to obtain a comparison result;

[0045] a classification unit, configured to determine that the initial workflow execution has failed when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a plurality of processing nodes, and classify the initial workflow execution failure into a temporary failure and a permanent failure based on the analysis results;

[0046] The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain a target workflow.

[0047] On the other hand, the present invention also provides a method based on an intelligent feature exploration system for structured data, the method comprising:

[0048] Obtain the initial data set uploaded by the user, preprocess the initial data set to obtain a structured data set;

[0049] Analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results;

[0050] Calculating an actual evaluation value based on the data structure characteristics and the data type characteristics, comparing the actual evaluation value with preset evaluation values in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0051] Generate an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0052] Dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result;

[0053] Executing the computing power allocation result through each processing node, monitoring the operating status of each processing node in real time, collecting operating data, and calculating the actual health score of the initial execution workflow based on the operating data;

[0054] The actual health score is compared with a preset health score to obtain a comparison result, and the initial execution workflow is adjusted based on the comparison result to obtain a target execution workflow.

[0055] Compared with the prior art, the beneficial effect of the present invention is that, by setting the preprocessing module to preprocess the initial data set uploaded by the user, the uniformity, integrity and cleanliness of the data format are ensured, a solid foundation is laid for subsequent data analysis, and it helps to improve the efficiency and accuracy of subsequent processing steps; by setting the feature acquisition module to analyze the preprocessed structured data set, key structural features and type features are extracted, which provides an important basis for the selection and customization of subsequent algorithms, and helps to more accurately identify patterns and information in the data, thereby enhancing the pertinence and effectiveness of data exploration; by setting the algorithm determination module to calculate the actual evaluation value and compare it with the preset evaluation value in the preset algorithm unit, the algorithm unit that best suits the current data characteristics and needs is intelligently selected, thereby improving the flexibility and efficiency of data processing, and ensuring the accuracy and efficiency of algorithm application; by setting the generation module to preset the operating rules according to the selected algorithm unit, the initial execution workflow is automatically generated, and the operation flow is simplified. The program also ensures the consistency and repeatability of the workflow, provides clear guidance for subsequent task allocation and execution, and realizes dynamic optimization of computing resources by setting the allocation module to divide subtasks, calculate task complexity and allocate computing power based on real-time load conditions, ensuring that each processing node can still maintain efficient operation under high load, avoiding waste of resources and the emergence of bottlenecks, and significantly improving the overall performance and stability of the system. By setting the monitoring module to monitor the operating status of each processing node and collect operating data in real time, potential problems can be discovered and handled in a timely manner. The actual health score calculated based on the operating data provides a scientific basis for the evaluation and adjustment of the system health status, ensuring the efficient and stable operation of the system. By setting the adjustment module to compare the actual health score with the preset health score, and adjusting the initial execution workflow according to the comparison results, a more optimized target execution workflow is obtained, which can continuously optimize the design and execution of the workflow and improve the quality and efficiency of data processing.

[0056] In particular, by setting the integrity detection unit to detect the integrity of the initial data set, it is ensured that there is no missing or omitted data, providing necessary preparation for subsequent data processing, and avoiding errors or deviations caused by incomplete data in subsequent steps. By setting the correctness detection unit to verify whether the data in the initial data set conforms to the expected format, range and logical rules, it is helpful to identify outliers, wrong formats or data items that do not conform to business logic in the data, ensure the accuracy and reliability of the data, and provide an accurate data basis for subsequent data analysis and feature extraction. By setting the calibration unit to calibrate the initial data set according to the integrity detection results and the correctness detection results, including filling missing values, correcting erroneous data, adjusting outliers and other measures, to ensure the consistency and accuracy of the data set, and provide an accurate data source for subsequent data analysis and model training. By setting the processing unit to perform structured processing on the calibrated data set, it is converted into a structured data set suitable for subsequent analysis, and the data is organized into a unified format and structure to facilitate subsequent feature extraction and algorithm application, thereby improving the availability and accessibility of the data and improving the efficiency of subsequent data exploration and analysis.

[0057] In particular, by setting the type determination unit to identify the field type of each field in the structured data set, and determining the type characteristics based on these field types, a basis is provided for selecting algorithms in subsequent data processing and analysis, which helps to improve the efficiency and accuracy of data processing. By setting the identification unit to identify data distribution characteristics and data relationship characteristics, an important basis is provided for subsequent data analysis and model establishment. By setting the structure determination unit based on data distribution characteristics and data relationship characteristics, the structural characteristics of the structured data set are further determined, which provides a basis for subsequent data processing and feature engineering, and helps to design more efficient and accurate algorithm models. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A structural block diagram of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0059] Figure 2 A structural block diagram of a preprocessing module of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0060] Figure 3 A structural block diagram of a feature acquisition module of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0061] Figure 4 A flowchart of a method based on an intelligent feature exploration system for structured data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0063] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0064] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present invention.

[0065] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0066] See also Figure 1 As shown, an embodiment of the present invention provides an intelligent feature exploration system for structured data, the system comprising:

[0067] The preprocessing module 10 is used to obtain the initial data set uploaded by the user and preprocess the initial data set to obtain a structured data set;

[0068] a feature acquisition module 20 connected to the preprocessing module 10 and configured to analyze the structured data set to obtain structural features and type features of the structured data set based on the analysis results;

[0069] an algorithm determination module 30 connected to the feature acquisition module 20, configured to calculate an actual evaluation value based on the data structure feature and the data type feature, compare the actual evaluation value with preset evaluation values of a plurality of preset algorithm units, and determine an execution algorithm unit according to the comparison result;

[0070] A generation module 40, connected to the algorithm determination module 30, for generating an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0071] an allocation module 50 connected to the generation module 40, configured to divide the initial execution workflow into a plurality of subtasks, calculate the task complexity corresponding to the plurality of subtasks, allocate a plurality of processing nodes based on the task complexity, monitor the real-time load conditions corresponding to the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result;

[0072] A monitoring module 60, connected to the allocation module 50, is used to execute the computing power allocation result through each processing node, monitor the operating status of each processing node in real time, collect operating data, and calculate the actual health score of the initial execution workflow based on the operating data;

[0073] The adjustment module 70 is connected to the monitoring module 60 and is used to compare the actual health score with the preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0074] Specifically, the embodiment of the present invention sets the preprocessing module to preprocess the initial data set uploaded by the user, thereby ensuring the uniformity of the format, integrity and cleanliness of the data, laying a solid foundation for subsequent data analysis, and helping to improve the efficiency and accuracy of subsequent processing steps. By setting the feature acquisition module to analyze the preprocessed structured data set, key structural features and type features are extracted, which provides an important basis for the selection and customization of subsequent algorithms, and helps to more accurately identify patterns and information in the data, thereby enhancing the pertinence and effectiveness of data exploration. By setting the algorithm determination module to calculate the actual evaluation value and compare it with the preset evaluation value in the preset algorithm unit, the algorithm unit that best suits the current data characteristics and needs is intelligently selected, thereby improving the flexibility and efficiency of data processing and ensuring the accuracy and efficiency of algorithm application. By setting the generation module to preset operating rules according to the selected algorithm unit, the initial execution workflow is automatically generated, the operation process is simplified, and the accuracy and efficiency of the algorithm application are ensured. It ensures the consistency and repeatability of the workflow and provides clear guidance for subsequent task allocation and execution. By setting the allocation module to divide subtasks, calculate task complexity and allocate computing power based on real-time load conditions, dynamic optimization of computing resources is achieved, ensuring that each processing node can still maintain efficient operation under high load, avoiding resource waste and the emergence of bottlenecks, and significantly improving the overall performance and stability of the system. By setting the monitoring module to monitor the operating status of each processing node in real time and collect operating data, potential problems can be discovered and handled in a timely manner. The actual health score calculated based on the operating data provides a scientific basis for the evaluation and adjustment of the system health status, ensuring the efficient and stable operation of the system. By setting the adjustment module to compare the actual health score with the preset health score, and adjusting the initial execution workflow according to the comparison results, a more optimized target execution workflow is obtained, which can continuously optimize the design and execution of the workflow and improve the quality and efficiency of data processing.

[0075] See also Figure 2 As shown, the pre-processing module 10 includes:

[0076] An integrity detection unit 11 is used to detect the integrity of the initial data set and obtain an integrity detection result;

[0077] a correctness detection unit 12, configured to detect the correctness of the initial data set and obtain a correctness detection result;

[0078] a calibration unit 13, configured to calibrate the initial data set according to the integrity detection result and the correctness detection result to obtain a calibration data set;

[0079] The processing unit 14 is connected to the calibration unit 13 and is used to perform structured processing on the calibration data set to obtain the structured data set.

[0080] Specifically, in the embodiment of the present invention, detecting the integrity of the initial data set includes detecting missing data and duplicate data in the initial data set;

[0081] Detecting the correctness of the initial data set includes detecting garbled data and abnormal data in the initial data.

[0082] Specifically, the processing unit described in the embodiment of the present invention can perform structured processing on the calibration dataset through the pandas library in Python.

[0083] Specifically, in the embodiments of the present invention, missing data can be detected by counting the number of null values in each field, and duplicate data can be detected by comparing the unique identifiers of each record in the data set. For the detection of garbled data, the system can use specific encoding rule verification methods, such as checking whether the characters comply with the UTF-8 encoding standard. The detection of abnormal data involves judging the rationality of the data value, such as by setting a threshold to identify outliers or data that does not comply with business rules.

[0084] Specifically, in this embodiment of the present invention, calibrating the initial data set according to the integrity detection result and the correctness detection result includes:

[0085] Fill in missing data, where the filling strategy can use the field's mean, median, mode, or estimated value based on the prediction model;

[0086] Delete duplicate data to ensure the uniqueness of each record in the calibration data set;

[0087] Correct garbled data and repair or replace characters that do not meet the standards through coding rule verification methods;

[0088] Correct abnormal data and adjust outliers or data that does not comply with the rules based on business rules and data distribution.

[0089] Specifically, the embodiment of the present invention sets the integrity detection unit to detect the integrity of the initial data set, ensuring that there is no missing or omission of data, providing necessary preparation for subsequent data processing, and avoiding errors or deviations caused by incomplete data in subsequent steps. By setting the correctness detection unit to verify whether the data in the initial data set conforms to the expected format, range and logical rules, it helps to identify outliers, incorrect formats or data items that do not conform to business logic in the data, ensure the accuracy and reliability of the data, and provide an accurate data basis for subsequent data analysis and feature extraction. By setting the calibration unit, the initial data set is calibrated according to the integrity detection results and the correctness detection results, including filling missing values, correcting erroneous data, adjusting outliers and other measures to ensure the consistency and accuracy of the data set, providing an accurate data source for subsequent data analysis and model training. By setting the processing unit, the calibrated data set is structured and converted into a structured data set suitable for subsequent analysis. The data is organized into a unified format and structure to facilitate subsequent feature extraction and algorithm application, thereby improving the availability and accessibility of the data and improving the efficiency of subsequent data exploration and analysis.

[0090] Specifically, the embodiment of the present invention performs structured processing on the calibration data set by

[0091] See also Figure 3 As shown, the feature acquisition module 20 includes:

[0092] a type determination unit 21, configured to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types;

[0093] an identification unit 22 for identifying data distribution characteristics and data relationship characteristics of the structured data set;

[0094] The structure determination unit 23 is connected to the identification unit 22 and is used to determine the structure feature based on the data distribution feature and the data relationship feature.

[0095] Specifically, the field type described in the embodiment of the present invention may include type characteristics including numeric type (such as integer, floating point number), character type (such as string), date type (such as date, timestamp), Boolean type (true / false), etc. For example, in a user information dataset, the age field may be of integer type, the name field may be of string type, and the registration date field may be of date type;

[0096] Identifying the types of several fields in the structured data set can be done through Pandas library, Matlab library, etc.

[0097] Specifically, the embodiment of the present invention sets the type determination unit to identify the field type of each field in the structured data set, and determines the type characteristics based on these field types, providing a basis for selecting algorithms in subsequent data processing and analysis, which helps to improve the efficiency and accuracy of data processing. By setting the identification unit to identify data distribution characteristics and data relationship characteristics, an important basis is provided for subsequent data analysis and model establishment. By setting the structure determination unit based on data distribution characteristics and data relationship characteristics, the structural characteristics of the structured data set are further determined, which provides a basis for subsequent data processing and feature engineering, and helps to design a more efficient and accurate algorithm model.

[0098] Specifically, the identification unit includes:

[0099] A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set;

[0100] a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness, and the kurtosis;

[0101] The correlation determination subunit is used to calculate the actual correlation coefficient corresponding to each field in the structured data set based on a correlation analysis algorithm, and determine the data relationship feature based on a number of actual correlation coefficients.

[0102] Specifically, the correlation analysis algorithm described in the embodiment of the present invention can be the Pearson correlation coefficient, the Spearman rank correlation coefficient, etc., wherein the value of the Pearson correlation coefficient is between -1 and 1, close to 1 indicates positive correlation, close to -1 indicates negative correlation, and close to 0 indicates no obvious linear correlation.

[0103] Specifically, the mean value in the embodiment of the present invention can be obtained by summing all the values in the field and then dividing it by the number of values;

[0104] The median is obtained by sorting the data from small to large and finding the number in the middle. If the amount of data is odd, the median is the number in the middle; if it is even, the median is the average of the two numbers in the middle.

[0105] The standard deviation is calculated by taking the difference between each value and the mean, averaging the squares of these differences, and then taking the square root.

[0106] Skewness was calculated by using the standardized form of the third-order central moment, with positive values indicating right skewness (positive skewness) and negative values indicating left skewness (negative skewness);

[0107] Kurtosis is measured by the standardized form of the fourth-order central moment. Compared with the normal distribution, a kurtosis value greater than 0 indicates a more peaked distribution, while a kurtosis value less than 0 indicates a flatter distribution.

[0108] Specifically, the embodiment of the present invention calculates the statistical indicators of each field to reveal the central tendency and dispersion of the data set, providing basic information for subsequent data analysis. For example, the mean and median are used to reflect the central position of the data set, the standard deviation is used to measure the degree of fluctuation of the data, and the skewness and kurtosis are used to describe the shape of the data distribution.

[0109] Specifically, the algorithm determination module 30 includes:

[0110] a first calculation unit, configured to calculate an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness, and the kurtosis;

[0111] a second calculation unit, configured to determine an actual correlation coefficient evaluation value according to the actual correlation coefficients;

[0112] A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

[0113] Specifically, the actual structure evaluation value of the embodiment of the present invention = k1×[(mean-mean of means) / standard deviation of means]+k2×[(median-mean of medians) / standard deviation of medians]+k3×[(standard deviation-mean of standard deviations) / standard deviation of standard deviations]+k4×[(skewness-mean of skewness) / standard deviation of skewness]+k5×[(kurtosis-mean of kurtosis) / standard deviation of kurtosis], where k1, k2, k3, k4, and k5 are weight values, k1+k2+k3+k4+k5=1. The weight values k1 to k5 can be dynamically adjusted according to the specific requirements and business scenarios of the data set. For example, in some business scenarios, the mean and median of the data set are more important than the standard deviation, skewness, and kurtosis. In this case, the weights of k1 and k2 are appropriately increased, and the weights of k3, k4, and k5 are reduced.

[0114] The actual correlation coefficient evaluation value is the mean of several actual correlation coefficients;

[0115] An algorithm unit having the smallest corresponding sum of the first difference and the second difference is selected as the execution algorithm unit.

[0116] Specifically, the preset structure evaluation value in the embodiment of the present invention is calculated based on the statistical indicators of the historical data set;

[0117] The preset correlation coefficient evaluation value is calculated based on the historical correlation coefficient between fields in the historical data set.

[0118] Specifically, the embodiment of the present invention selects the most suitable algorithm unit based on the difference between the actual structure evaluation value and the actual correlation coefficient evaluation value and the preset value. If the sum of the first difference and the second difference is small, it means that the structure of the actual data set and the relationship between fields are closer to the preset ideal state. At this time, the selected algorithm unit can better adapt to the characteristics of the current data set, thereby improving the accuracy and efficiency of data analysis.

[0119] Specifically, the generating module 40 includes:

[0120] A first determining unit, configured to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit;

[0121] A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

[0122] Specifically, the preset execution order and dependencies described in the embodiments of the present invention can be pre-set based on the data processing flow and the characteristics of the algorithm units. For example, in a data cleaning and preprocessing workflow, the missing value processing unit is executed first, followed by the outlier detection unit, and finally the data normalization unit. The execution of each unit depends on the completion of the previous unit, ensuring that the data has reached the expected quality standard before entering the next processing stage.

[0123] Specifically, the workflow of the embodiment of the present invention may be a directed acyclic graph (DAG), wherein nodes represent execution units and edges represent dependency relationships between execution units.

[0124] Specifically, the allocation module 50 includes:

[0125] A division unit, configured to divide the initial execution workflow according to a plurality of work nodes to obtain a plurality of subtasks;

[0126] a complexity calculation unit, configured to determine an actual data volume and an actual data type of the subtask, compare the actual data volume and the actual data type with a historical execution database, obtain a historical average processing time and a historical resource consumption, and determine the task complexity based on the historical average processing time and the historical resource consumption;

[0127] a node allocation unit, connected to the complexity calculation unit, for allocating the subtasks to corresponding processing nodes according to the task complexity;

[0128] The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load conditions of the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result.

[0129] Specifically, the task complexity described in the embodiment of the present invention = α1×historical average processing time + α2×historical resource consumption.

[0130] Specifically, an embodiment of the node allocation unit of the embodiment of the present invention can be to evaluate the available processing nodes and obtain the current load, processing capacity and assigned tasks of each processing node;

[0131] For subtasks with high task complexity, the node allocation unit will give priority to allocating them to processing nodes with strong processing capabilities and relatively low current loads;

[0132] For subtasks with lower task complexity, they are assigned to processing nodes with relatively weak processing capabilities or slightly higher current loads, while ensuring load balancing.

[0133] Specifically, an embodiment of the computing power allocation unit according to an embodiment of the present invention includes:

[0134] The workflow execution engine maintains a sliding window that records the resource usage of each node's task type in the last N (configurable) executions, from which it extracts average performance parameters (including: average resource utilization, average task progress, average idleness, historical platform load, and average downstream node demand);

[0135] Utilization deviation = current actual utilization - average resource utilization; schedule deviation = real-time idleness - average idleness; load deviation = platform load - average platform load; downstream demand deviation = current downstream node demand - average downstream node demand;

[0136] New quota = current quota × (1 + adjustment factor), where adjustment factor = w1 × utilization deviation + w2 × progress deviation + w3 × queue backlog deviation + w4 × system load deviation + w5 × downstream demand deviation. Weights w1 to w5 can be adjusted dynamically based on specific scenarios. The default scenario is: w1 > w5 > w2 > w3 > w4, with the sum of w1 to w5 being 1.

[0137] Specifically, the embodiment of the present invention divides the initial execution workflow into multiple subtasks by setting the division unit, and decomposes the complex workflow into smaller and easier to manage units, thereby improving the flexibility and scalability of the system. Each subtask can be executed independently or work in conjunction with other subtasks to complete the entire data processing flow, thereby improving the efficiency of data processing. By setting the complexity calculation unit, the processing difficulty and resource requirements of each subtask are accurately evaluated by comparing the data in the historical execution database. The historical average processing time and historical resource consumption provide an important reference basis for the evaluation, making task allocation more accurate and reasonable. By setting the node allocation unit, subtasks are reasonably allocated to different computing nodes according to the calculated task complexity, task allocation is optimized, computing resources are fully utilized, and the situation where some nodes are overloaded while other nodes are idle is avoided, thereby achieving balanced resource utilization and improving overall processing efficiency.

[0138] Specifically, the monitoring module 60 includes:

[0139] A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor;

[0140] An operation monitoring unit is used to monitor the performance parameters of the task in real time, wherein the performance parameters include average response time and resource utilization;

[0141] A score calculation unit is used to determine a task completion rate based on the running status, and calculate an actual health score based on the task completion rate, average response time and resource utilization.

[0142] Specifically, a possible instance of the task status monitoring unit described in an embodiment of the present invention is: making a preliminary judgment based on a workflow status monitor, which is a custom Kubernetes controller built into the workflow execution engine, monitoring the execution status of each task node in the entire workflow to track the status of each task node, including start, run, complete or fail.

[0143] Specifically, a possible example of the operation detection unit in the embodiment of the present invention is: implemented by a performance monitoring agent integrated in the workflow execution engine, which is responsible for collecting and analyzing performance data during task execution, such as CPU usage, memory usage, disk I / O, and network transmission rate;

[0144] The average response time is calculated by measuring and averaging the response times of multiple tasks. For example, if a task is operated 10 times and the response times for each operation are 100ms, 120ms, 90ms, 110ms, 105ms, 95ms, 115ms, 100ms, 125ms, and 100ms, then the average response time = (100+120+90+110+105+95+115+100+125+100) / 10 = 105ms.

[0145] Resource utilization = CPU utilization × memory usage ratio. Assuming that the CPU utilization ratio is 70% and the memory utilization ratio is 60%, then resource utilization = 70% × 60% = 42%.

[0146] Specifically, a possible example of the scoring calculation unit in the embodiment of the present invention is: task completion rate = total number of tasks / number of completed tasks × 100%. Assuming that the total number of tasks is 100 and the number of completed tasks is 80, then the task completion rate is 80 / 100 × 100% = 80%;

[0147] The actual health score S is calculated as follows: S = 0.5 × task completion rate + 0.3 × average response time + 0.2 × resource utilization rate;

[0148] Assuming that the task completion rate is 80% (converted to 0.8), the average response time is 105ms (normalized to 0.7, assuming the normalization range is 0-1, 105ms is at a medium level), and the resource utilization rate is 42% (converted to 0.42), then the actual health score S = 0.5×0.8+0.3×0.7+0.2×0.42=0.4+0.21+0.084=0.694.

[0149] Specifically, the adjustment module 70 includes:

[0150] a comparing unit, configured to compare the actual health score with the preset health score to obtain a comparison result;

[0151] a classification unit, configured to determine that the initial workflow execution has failed when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a plurality of processing nodes, and classify the initial workflow execution failure into a temporary failure and a permanent failure based on the analysis results;

[0152] The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain a target workflow.

[0153] Specifically, the embodiment of the present invention formulates a processing strategy according to the failure category of the initial execution workflow, including:

[0154] For temporary failures, the processing strategies include: 1. Automatic retry mechanism: Implement a retry strategy with gradually increasing intervals (exponential backoff), set the maximum number of retries, and avoid infinite retries leading to resource waste. 2. Dynamic resource adjustment: Before retrying, moderately increase the resources allocated to the failed task (such as CPU, memory), monitor resource usage, and dynamically adjust to alleviate resource pressure. 3. Load balancing: If the failure is caused by a node overload, try to reschedule the task to a node with a lighter load. 4. Network fault handling: For temporary network-related failures, implement a short waiting strategy before trying to reconnect.

[0155] The handling strategies for permanent failures include: 1. Stop retrying immediately: After identifying a permanent failure, stop automatic retrying immediately to avoid resource waste. 2. Error reporting and log aggregation: Generate detailed error reports, including the cause of failure, relevant logs and environmental information, aggregate the error information and send it to the relevant development team or operation and maintenance personnel. 3. Rollback mechanism: If the failure is caused by recent changes, start the automatic rollback process and restore to the last known stable version. 4. Resource release and cleanup: Ensure that all resources occupied by failed tasks are released to prevent resource leakage. 5. Update the failure knowledge base: Add the newly identified permanent failure mode to the failure knowledge base for future rapid identification and processing.

[0156] See also Figure 4 As shown, an embodiment of the present invention further provides a method for an intelligent feature exploration system for structured data, the method comprising:

[0157] Step S100: obtaining an initial data set uploaded by a user, and preprocessing the initial data set to obtain a structured data set;

[0158] Step S200, analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results;

[0159] Step S300, calculating an actual evaluation value based on the data structure characteristics and the data type characteristics, comparing the actual evaluation value with preset evaluation values of a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0160] Step S400, generating an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0161] Step S500: Divide the initial execution workflow into several subtasks, calculate the task complexity corresponding to the several subtasks, allocate several processing nodes based on the task complexity, monitor the real-time load conditions corresponding to the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result;

[0162] Step S600: executing the computing power allocation result through each processing node, monitoring the operating status of each processing node in real time, collecting operating data, and calculating the actual health score of the initial execution workflow based on the operating data;

[0163] Step S700 : Compare the actual health score with the preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0164] Specifically, a method for an intelligent feature exploration system for structured data provided by an embodiment of the present invention can be applied to the above-mentioned intelligent feature exploration system for structured data to achieve the same technical effect, which will not be repeated here.

[0165] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0166] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An intelligent feature exploration system for structured data, characterized by: include: The preprocessing module is used to obtain the initial data set uploaded by the user, preprocess the initial data set to obtain a structured data set; a feature acquisition module, connected to the preprocessing module, for analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results; an algorithm determination module, connected to the feature acquisition module, for calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result; A generation module, connected to the algorithm determination module, for generating an initial execution workflow according to the preset operation rules of the execution algorithm unit; an allocation module, connected to the generation module, configured to divide the initial execution workflow into a plurality of subtasks, calculate the task complexity corresponding to the plurality of subtasks, allocate a plurality of processing nodes based on the task complexity, monitor the real-time load conditions corresponding to the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result; a monitoring module, connected to the allocation module, configured to execute the computing power allocation result through each processing node, monitor the operating status of each processing node in real time, collect operating data, and calculate the actual health score of the initially executed workflow based on the operating data; an adjustment module, connected to the monitoring module, for comparing the actual health score with a preset health score, obtaining a comparison result, and adjusting the initial execution workflow based on the comparison result to obtain a target execution workflow; The feature acquisition module includes: a type determination unit, configured to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types; an identification unit, configured to identify data distribution characteristics and data relationship characteristics of the structured data set; a structure determination unit, connected to the identification unit, for determining the structure feature based on the data distribution feature and the data relationship feature; The identification unit includes: A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set; a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness, and the kurtosis; a correlation determination subunit, configured to calculate actual correlation coefficients corresponding to various fields in the structured data set based on a correlation analysis algorithm, and determine the data relationship characteristics based on a plurality of actual correlation coefficients; The algorithm determination module includes: a first calculation unit, configured to calculate an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness, and the kurtosis; a second calculation unit, configured to determine an actual correlation coefficient evaluation value according to the actual correlation coefficients; A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

2. The intelligent feature exploration system for structured data according to claim 1, characterized in that: The pre-processing module comprises: an integrity detection unit, configured to detect the integrity of the initial data set and obtain an integrity detection result; a correctness detection unit, configured to detect the correctness of the initial data set and obtain a correctness detection result; a calibration unit, configured to calibrate the initial data set according to the integrity test result and the correctness test result to obtain a calibration data set; A processing unit is connected to the calibration unit and is used to perform structured processing on the calibration data set to obtain the structured data set.

3. The intelligent feature exploration system for structured data according to claim 2, characterized in that: The generation module includes: A first determining unit, configured to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit; A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

4. The intelligent feature exploration system for structured data according to claim 3, characterized in that: The allocation module includes: A division unit, configured to divide the initial execution workflow according to a plurality of work nodes to obtain a plurality of subtasks; a complexity calculation unit, configured to determine an actual data volume and an actual data type of the subtask, compare the actual data volume and the actual data type with a historical execution database, obtain a historical average processing time and a historical resource consumption, and determine the task complexity based on the historical average processing time and the historical resource consumption; a node allocation unit, connected to the complexity calculation unit, for allocating the subtasks to corresponding processing nodes according to the task complexity; The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load conditions of the processing nodes in real time, allocate computing power to each processing node based on the real-time load conditions, and obtain a computing power allocation result.

5. The intelligent feature exploration system for structured data according to claim 4, characterized in that: The monitoring module includes: A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor; An operation monitoring unit is used to monitor the performance parameters of the task in real time, wherein the performance parameters include average response time and resource utilization; A score calculation unit is used to determine a task completion rate based on the running status, and calculate an actual health score based on the task completion rate, average response time and resource utilization.

6. The intelligent feature exploration system for structured data according to claim 5, characterized in that: The adjustment module includes: a comparing unit, configured to compare the actual health score with the preset health score to obtain a comparison result; a classification unit, configured to determine that the initial workflow execution has failed when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a plurality of processing nodes, and classify the initial workflow execution failure into a temporary failure and a permanent failure based on the analysis results; The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain a target workflow.

7. A method for applying the intelligent feature exploration system for structured data according to any one of claims 1 to 6, characterized in that: include: Obtain the initial data set uploaded by the user, preprocess the initial data set to obtain a structured data set; Analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results; Calculating an actual evaluation value based on the data structure characteristics and the data type characteristics, comparing the actual evaluation value with preset evaluation values in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result; Generate an initial execution workflow according to the preset operation rules of the execution algorithm unit; Dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result; Executing the computing power allocation result through each processing node, monitoring the operating status of each processing node in real time, collecting operating data, and calculating the actual health score of the initial execution workflow based on the operating data; The actual health score is compared with a preset health score to obtain a comparison result, and the initial execution workflow is adjusted based on the comparison result to obtain a target execution workflow.

Images