Intelligent feature exploration system and method for structured data

Abstract

The invention relates to the technical field of data processing, in particular to an intelligent feature exploration system and method for structured data, and the system comprises a preprocessing module which obtains a structured data set; the feature acquisition module analyzes the structured data set and acquires structural features and type features; the algorithm determination module calculates an actual evaluation value based on the data structure features and the data type features, compares the actual evaluation value with a preset evaluation value, and determines an execution algorithm unit; the generation module generates an initial execution workflow according to a preset operation rule; the distribution module divides an initial execution workflow, calculates task complexity, distributes a plurality of processing nodes, and monitors a real-time load condition to perform calculation power distribution; the monitoring module executes a computing power distribution result, collects operation data and calculates an actual health score; and the adjustment module compares the actual health score with a preset health score, and adjusts the initial execution workflow to obtain a target execution workflow. According to the invention, the efficiency of the data analysis processing process is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent feature exploration system and method for structured data. Background Art

[0002] In the field of machine learning, data preprocessing and feature engineering are crucial, especially when dealing with structured data. However, this process usually faces many challenges. First, the original data sets uploaded by users often cannot directly meet specific analysis needs and need to go through a series of complex processing steps, including data cleaning, conversion, feature engineering, and synthetic data generation. Secondly, faced with complex structured data, AI data engineers usually find it difficult to intuitively judge which are effective features. They need to try a large number of different types of data feature processing algorithms and evaluate the effectiveness of each method to determine the final feature set for deep learning.

[0003] The patent document with Chinese patent application number CN112579582A discloses a data exploration method and system for a data analysis engine, the method comprising collecting external big data and cleaning and structuring the external big data to obtain a structured data set; characterizing the structured data in the structured data set to obtain the characteristic value of each structured data and form a characteristic value set; performing data exploration on the characteristic value set based on a deep learning data model to obtain data exploration results.

[0004] The existing technology requires writing and managing a large number of processing programs based on feature algorithms. For different data sets or different features in the same data set, program parameters need to be frequently adjusted, and a lot of time is spent on modifying the input parameters of existing algorithms, which reduces the efficiency of the data feature exploration process. Summary of the invention

[0005] To this end, the present invention provides an intelligent feature exploration system and method for structured data, which can solve the problem of low feature exploration efficiency.

[0006] To achieve the above object, the present invention provides an intelligent feature exploration system for structured data, the system comprising:

[0007] A preprocessing module is used to obtain the initial data set uploaded by the user, preprocess the initial data set, and obtain a structured data set;

[0008] A feature acquisition module, connected to the preprocessing module, for analyzing the structured data set to acquire structural features and type features of the structured data set according to the analysis results;

[0009] an algorithm determination module, connected to the feature acquisition module, for calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0010] A generation module, connected to the algorithm determination module, for generating an initial execution workflow according to a preset operation rule of the execution algorithm unit;

[0011] an allocation module, connected to the generation module, for dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result;

[0012] A monitoring module, connected to the allocation module, for executing the computing power allocation result through each processing node, monitoring the operation status of each processing node in real time, collecting operation data, and calculating the actual health score of the initial execution workflow based on the operation data;

[0013] The adjustment module is connected to the monitoring module and is used to compare the actual health score with the preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0014] Furthermore, the preprocessing module includes:

[0015] An integrity detection unit, used to detect the integrity of the initial data set and obtain an integrity detection result;

[0016] A correctness detection unit, used to detect the correctness of the initial data set and obtain a correctness detection result;

[0017] a calibration unit, configured to calibrate the initial data set according to the integrity detection result and the correctness detection result to obtain a calibration data set;

[0018] A processing unit is connected to the calibration unit and is used to perform structured processing on the calibration data set to obtain the structured data set.

[0019] Furthermore, the feature acquisition module includes:

[0020] a type determination unit, configured to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types;

[0021] an identification unit, used to identify data distribution characteristics and data relationship characteristics of the structured data set;

[0022] The structure determination unit is connected to the identification unit and is used to determine the structure feature based on the data distribution feature and the data relationship feature.

[0023] Furthermore, the identification unit includes:

[0024] A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set;

[0025] a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness and the kurtosis;

[0026] The correlation determination subunit is used to calculate the actual correlation coefficient corresponding to each field in the structured data set based on the correlation analysis algorithm, and determine the data relationship characteristics based on a number of actual correlation coefficients.

[0027] Furthermore, the algorithm determination module includes:

[0028] A first calculation unit, used for calculating an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness and the kurtosis;

[0029] A second calculation unit, used for determining an actual correlation coefficient evaluation value according to the actual correlation coefficients;

[0030] A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

[0031] Furthermore, the generating module comprises:

[0032] A first determining unit, used to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit;

[0033] A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

[0034] Furthermore, the allocation module includes:

[0035] A division unit, used for dividing the initial execution workflow according to a number of work nodes to obtain a number of subtasks;

[0036] A complexity calculation unit, used to determine the actual data volume and actual data type of the subtask, compare the actual data volume and actual data type with the historical execution database, obtain the historical average processing time and historical resource consumption, and determine the task complexity based on the historical average processing time and historical resource consumption;

[0037] A node allocation unit, connected to the complexity calculation unit, for allocating the subtask to a corresponding processing node according to the task complexity;

[0038] The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load status of the processing node in real time, and allocate computing power to each processing node based on the real-time load status to obtain a computing power allocation result.

[0039] Furthermore, the monitoring module includes:

[0040] A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor;

[0041] An operation monitoring unit, used to monitor the performance parameters of the task in real time during operation, wherein the performance parameters include average response time and resource utilization;

[0042] A score calculation unit is used to determine the task completion rate based on the running state, and calculate the actual health score based on the task completion rate, the average response time and the resource utilization rate.

[0043] Furthermore, the adjustment module includes:

[0044] A comparison unit, used to compare the actual health score with the preset health score to obtain a comparison result;

[0045] a classification unit, configured to determine that the initial execution workflow fails when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a number of processing nodes, and classify the initial execution workflow failure into a temporary failure and a permanent failure according to the analysis result;

[0046] The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain the target workflow.

[0047] On the other hand, the present invention also provides a method based on an intelligent feature exploration system for structured data, the method comprising:

[0048] Obtain the initial data set uploaded by the user, preprocess the initial data set, and obtain a structured data set;

[0049] Analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results;

[0050] Calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0051] Generate an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0052] Dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result;

[0053] Execute the computing power allocation result through each processing node, monitor the operation status of each processing node in real time, collect operation data, and calculate the actual health score of the initial execution workflow based on the operation data;

[0054] The actual health score is compared with a preset health score to obtain a comparison result, and the initial execution workflow is adjusted based on the comparison result to obtain a target execution workflow.

[0055] Compared with the prior art, the beneficial effects of the present invention are that, by setting the preprocessing module to preprocess the initial data set uploaded by the user, the uniformity, integrity and cleanliness of the data format are ensured, a solid foundation is laid for subsequent data analysis, and it is helpful to improve the efficiency and accuracy of subsequent processing steps; by setting the feature acquisition module to analyze the preprocessed structured data set, key structural features and type features are extracted, which provides an important basis for the selection and customization of subsequent algorithms, and helps to more accurately identify patterns and information in the data, thereby enhancing the pertinence and effectiveness of data exploration; by setting the algorithm determination module to calculate the actual evaluation value and compare it with the preset evaluation value in the preset algorithm unit, the algorithm unit that best suits the current data characteristics and needs is intelligently selected, which improves the flexibility and efficiency of data processing, and ensures the accuracy and efficiency of algorithm application; by setting the generation module to preset the operating rules according to the selected algorithm unit, the initial execution workflow is automatically generated, which simplifies the operation flow. The program also ensures the consistency and repeatability of the workflow, provides clear guidance for subsequent task allocation and execution, and realizes dynamic optimization of computing resources by setting the allocation module to divide subtasks, calculate task complexity and allocate computing power based on real-time load conditions, ensuring that each processing node can still maintain efficient operation under high load, avoiding waste of resources and the emergence of bottlenecks, and significantly improving the overall performance and stability of the system. By setting the monitoring module to monitor the operating status of each processing node in real time and collect operating data, potential problems can be discovered and handled in a timely manner. The actual health score calculated based on the operating data provides a scientific basis for the evaluation and adjustment of the system health status, ensuring the efficient and stable operation of the system. By setting the adjustment module to compare the actual health score with the preset health score, and adjusting the initial execution workflow according to the comparison result, a more optimized target execution workflow is obtained, which can continuously optimize the design and execution of the workflow and improve the quality and efficiency of data processing.

[0056] In particular, by setting the integrity detection unit to detect the integrity of the initial data set, it is ensured that there is no missing or omitted data, and necessary preparation is provided for subsequent data processing to avoid errors or deviations caused by incomplete data in subsequent steps. By setting the correctness detection unit to verify whether the data in the initial data set conforms to the expected format, range and logical rules, it is helpful to identify abnormal values, wrong formats or data items that do not conform to business logic in the data, ensure the accuracy and reliability of the data, and provide an accurate data basis for subsequent data analysis and feature extraction. By setting the calibration unit to calibrate the initial data set according to the integrity detection results and the correctness detection results, including filling in missing values, correcting erroneous data, adjusting abnormal values ​​and other measures, to ensure the consistency and accuracy of the data set, and provide an accurate data source for subsequent data analysis and model training. By setting the processing unit to perform structured processing on the calibrated data set, it is converted into a structured data set suitable for subsequent analysis, and the data is organized into a unified format and structure to facilitate subsequent feature extraction and algorithm application, thereby improving the availability and accessibility of the data and improving the efficiency of subsequent data exploration and analysis.

[0057] In particular, by setting the type determination unit to identify the field type of each field in the structured data set, and determining the type characteristics based on these field types, a basis is provided for selecting algorithms in subsequent data processing and analysis, which helps to improve the efficiency and accuracy of data processing. By setting the identification unit to identify data distribution characteristics and data relationship characteristics, an important basis is provided for subsequent data analysis and model building. By setting the structure determination unit to further determine the structural characteristics of the structured data set based on data distribution characteristics and data relationship characteristics, a basis is provided for subsequent data processing and feature engineering, which helps to design a more efficient and accurate algorithm model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A structural block diagram of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0059] Figure 2 A structural block diagram of a preprocessing module of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0060] Figure 3 A structural block diagram of a feature acquisition module of an intelligent feature exploration system for structured data provided by an embodiment of the present invention;

[0061] Figure 4 A flowchart of a method based on an intelligent feature exploration system for structured data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0064] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present invention.

[0065] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] See also Figure 1 As shown, an embodiment of the present invention provides an intelligent feature exploration system for structured data, the system comprising:

[0067] A preprocessing module 10 is used to obtain an initial data set uploaded by a user, and preprocess the initial data set to obtain a structured data set;

[0068] A feature acquisition module 20, connected to the preprocessing module 10, for analyzing the structured data set to acquire structural features and type features of the structured data set according to the analysis results;

[0069] an algorithm determination module 30, connected to the feature acquisition module 20, for calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0070] A generation module 40, connected to the algorithm determination module 30, for generating an initial execution workflow according to a preset operation rule of the execution algorithm unit;

[0071] The allocation module 50 is connected to the generation module 40, and is used to divide the initial execution workflow into a number of subtasks, calculate the task complexity corresponding to the subtasks, allocate a number of processing nodes based on the task complexity, monitor the real-time load conditions corresponding to the processing nodes in real time, and allocate computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result;

[0072] A monitoring module 60, connected to the allocation module 50, for executing the computing power allocation result through each processing node, monitoring the operation status of each processing node in real time, collecting operation data, and calculating the actual health score of the initial execution workflow based on the operation data;

[0073] The adjustment module 70 is connected to the monitoring module 60 and is used to compare the actual health score with the preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0074] Specifically, the embodiment of the present invention sets the preprocessing module to preprocess the initial data set uploaded by the user, thereby ensuring the uniformity of the format, integrity and cleanliness of the data, laying a solid foundation for subsequent data analysis, and helping to improve the efficiency and accuracy of subsequent processing steps. By setting the feature acquisition module to analyze the preprocessed structured data set, key structural features and type features are extracted, which provides an important basis for the selection and customization of subsequent algorithms, and helps to more accurately identify patterns and information in the data, thereby enhancing the pertinence and effectiveness of data exploration. By setting the algorithm determination module to calculate the actual evaluation value and compare it with the preset evaluation value in the preset algorithm unit, the algorithm unit that best suits the current data characteristics and needs is intelligently selected, thereby improving the flexibility and efficiency of data processing and ensuring the accuracy and efficiency of algorithm application. By setting the generation module to preset the operating rules according to the selected algorithm unit, the initial execution workflow is automatically generated, the operation process is simplified, and the accuracy and efficiency of the algorithm application are ensured. The consistency and repeatability of the workflow are ensured, and clear guidance is provided for subsequent task allocation and execution. By setting the allocation module to divide subtasks, calculate task complexity, and allocate computing power based on real-time load conditions, dynamic optimization of computing resources is achieved, ensuring that each processing node can still maintain efficient operation under high load, avoiding waste of resources and the emergence of bottlenecks, and significantly improving the overall performance and stability of the system. By setting the monitoring module to monitor the operating status of each processing node in real time and collect operating data, potential problems can be discovered and handled in a timely manner. The actual health score calculated based on the operating data provides a scientific basis for the evaluation and adjustment of the system health status, ensuring the efficient and stable operation of the system. By setting the adjustment module to compare the actual health score with the preset health score, and adjusting the initial execution workflow according to the comparison result, a more optimized target execution workflow is obtained, which can continuously optimize the design and execution of the workflow and improve the quality and efficiency of data processing.

[0075] See also Figure 2 As shown, the preprocessing module 10 includes:

[0076] An integrity detection unit 11 is used to detect the integrity of the initial data set and obtain an integrity detection result;

[0077] A correctness detection unit 12, used to detect the correctness of the initial data set and obtain a correctness detection result;

[0078] A calibration unit 13, configured to calibrate the initial data set according to the integrity detection result and the correctness detection result to obtain a calibration data set;

[0079] The processing unit 14 is connected to the calibration unit 13 and is used to perform structured processing on the calibration data set to obtain the structured data set.

[0080] Specifically, the embodiment of the present invention detects the integrity of the initial data set including detecting missing data and duplicate data of the initial data set;

[0081] Detecting the correctness of the initial data set includes detecting garbled data and abnormal data of the initial data.

[0082] Specifically, the processing unit described in the embodiment of the present invention can perform structured processing on the calibration data set through the pandas library in Python.

[0083] Specifically, the detection of missing data in the embodiment of the present invention can be achieved by counting the number of null values ​​in each field, and the detection of duplicate data is completed by comparing the unique identifiers of each record in the data set. For the detection of garbled data, the system can use a specific encoding rule verification method, such as checking whether the characters comply with the UTF-8 encoding standard. The detection of abnormal data involves judging the rationality of the data value, such as identifying outliers or data that does not comply with business rules by setting thresholds.

[0084] Specifically, the embodiment of the present invention calibrates the initial data set according to the integrity detection result and the correctness detection result, including:

[0085] Fill in missing data, where the filling strategy can use the mean, median, mode or estimated value based on the prediction model of the field;

[0086] Delete duplicate data to ensure the uniqueness of each record in the calibration data set;

[0087] Correct the garbled data and repair or replace the characters that do not meet the standards through the coding rule verification method;

[0088] Correct abnormal data and adjust outliers or data that does not comply with the rules based on business rules and data distribution.

[0089] Specifically, the embodiment of the present invention detects the integrity of the initial data set by setting the integrity detection unit to ensure that there is no missing or omitted data, provides necessary preparation for subsequent data processing, and avoids errors or deviations caused by incomplete data in subsequent steps. By setting the correctness detection unit to verify whether the data in the initial data set conforms to the expected format, range and logical rules, it helps to identify abnormal values, wrong formats or data items that do not conform to business logic in the data, ensure the accuracy and reliability of the data, and provide an accurate data basis for subsequent data analysis and feature extraction. By setting the calibration unit, the initial data set is calibrated according to the integrity detection results and the correctness detection results, including filling missing values, correcting erroneous data, adjusting abnormal values ​​and other measures to ensure the consistency and accuracy of the data set, and provide an accurate data source for subsequent data analysis and model training. By setting the processing unit, the calibrated data set is structured and converted into a structured data set suitable for subsequent analysis, and the data is organized into a unified format and structure to facilitate subsequent feature extraction and algorithm application, thereby improving the availability and accessibility of the data and improving the efficiency of subsequent data exploration and analysis.

[0090] Specifically, the embodiment of the present invention performs structured processing on the calibration data set by:

[0091] See also Figure 3 As shown, the feature acquisition module 20 includes:

[0092] A type determination unit 21, used to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types;

[0093] an identification unit 22, used to identify data distribution characteristics and data relationship characteristics of the structured data set;

[0094] The structure determination unit 23 is connected to the identification unit 22 and is used to determine the structure feature based on the data distribution feature and the data relationship feature.

[0095] Specifically, the field type described in the embodiment of the present invention may include type characteristics including numeric type (such as integer, floating point number), character type (such as string), date type (such as date, timestamp), Boolean type (true / false), etc. For example, in a user information data set, the age field may be an integer type, the name field may be a string type, and the registration date field may be a date type;

[0096] Identifying the types of several fields in the structured data set can be done through Pandas library, Matplotlib library, etc.

[0097] Specifically, the embodiment of the present invention sets the type determination unit to identify the field type of each field in the structured data set, and determines the type characteristics based on these field types, thereby providing a basis for selecting algorithms in subsequent data processing and analysis, and helping to improve the efficiency and accuracy of data processing. By setting the identification unit to identify data distribution characteristics and data relationship characteristics, an important basis is provided for subsequent data analysis and model building. By setting the structure determination unit to further determine the structural characteristics of the structured data set based on data distribution characteristics and data relationship characteristics, a basis is provided for subsequent data processing and feature engineering, which helps to design a more efficient and accurate algorithm model.

[0098] Specifically, the identification unit includes:

[0099] A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set;

[0100] a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness and the kurtosis;

[0101] The correlation determination subunit is used to calculate the actual correlation coefficient corresponding to each field in the structured data set based on the correlation analysis algorithm, and determine the data relationship characteristics based on a number of actual correlation coefficients.

[0102] Specifically, the correlation analysis algorithm described in the embodiment of the present invention may be the Pearson correlation coefficient, the Spearman rank correlation coefficient, etc., wherein the value of the Pearson correlation coefficient is between -1 and 1, close to 1 indicates a positive correlation, close to -1 indicates a negative correlation, and close to 0 indicates no obvious linear correlation.

[0103] Specifically, the mean value in the embodiment of the present invention can be obtained by summing all the values ​​in the field and then dividing by the number of values;

[0104] The median is obtained by sorting the data from small to large and finding the number in the middle. If the amount of data is an odd number, the median is the number in the middle; if it is an even number, the median is the average of the two numbers in the middle.

[0105] The standard deviation is calculated by calculating the difference between each value and the mean, averaging the squares of these differences, and then taking the square root;

[0106] Skewness was calculated by using the standardized form of the third-order central moment, with positive values ​​indicating right skewness (positive skewness) and negative values ​​indicating left skewness (negative skewness);

[0107] Kurtosis is measured using the standardized form of the fourth-order central moment. A kurtosis value greater than 0 indicates a more peaked distribution, while a kurtosis value less than 0 indicates a flatter distribution compared to a normal distribution.

[0108] Specifically, the embodiment of the present invention calculates the statistical indicators of each field to reveal the central tendency and dispersion of the data set, providing basic information for subsequent data analysis. For example, the mean and median are used to reflect the central position of the data set, the standard deviation is used to measure the volatility of the data, and the skewness and kurtosis are used to describe the shape of the data distribution.

[0109] Specifically, the algorithm determination module 30 includes:

[0110] A first calculation unit, used for calculating an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness and the kurtosis;

[0111] A second calculation unit, used for determining an actual correlation coefficient evaluation value according to the actual correlation coefficients;

[0112] A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

[0113] Specifically, the actual structure evaluation value of the embodiment of the present invention = k1×[(mean-mean of means) / standard deviation of mean]+k2×[(median-mean of medians) / standard deviation of median]+k3×[(standard deviation-mean of standard deviation) / standard deviation of standard deviation]+k4×[(skewness-mean of skewness) / standard deviation of skewness]+k5×[(kurtosis-mean of kurtosis) / standard deviation of kurtosis], wherein k1, k2, k3, k4, k5 are weight values, k1+k2+k3+k4+k5=1. The weight values ​​k1 to k5 can be dynamically adjusted according to the specific requirements and business scenarios of the data set. For example, in some business scenarios, the mean and median of the data set are more important than the standard deviation, skewness and kurtosis. At this time, the weights of k1 and k2 are appropriately increased, and the weights of k3, k4 and k5 are reduced.

[0114] The actual correlation coefficient evaluation value is the average of several actual correlation coefficients;

[0115] An algorithm unit having the smallest corresponding sum of the first difference and the second difference is selected as the execution algorithm unit.

[0116] Specifically, the preset structural evaluation value in the embodiment of the present invention is calculated based on the statistical indicators of the historical data set;

[0117] The preset correlation coefficient evaluation value is calculated based on the historical correlation coefficient between fields in the historical data set.

[0118] Specifically, the embodiment of the present invention selects the most suitable algorithm unit based on the difference between the actual structure evaluation value and the actual correlation coefficient evaluation value and the preset value. If the sum of the first difference and the second difference is small, it means that the structure of the actual data set and the relationship between fields are closer to the preset ideal state. At this time, the selected algorithm unit can better adapt to the characteristics of the current data set, thereby improving the accuracy and efficiency of data analysis.

[0119] Specifically, the generating module 40 includes:

[0120] A first determining unit, used to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit;

[0121] A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

[0122] Specifically, the preset execution order and preset dependency relationship described in the embodiment of the present invention can be preset according to the characteristics of the data processing flow and the algorithm unit. For example, in a data cleaning and preprocessing workflow, the missing value processing unit is executed first, then the outlier detection unit is executed, and finally the data standardization unit is executed. The execution of each unit depends on the completion of the previous unit, ensuring that the data has reached the expected quality standard before entering the next processing stage.

[0123] Specifically, the workflow of the embodiment of the present invention may be a directed acyclic graph (DAG), in which nodes represent execution units and edges represent dependencies between execution units.

[0124] Specifically, the allocation module 50 includes:

[0125] A division unit, used for dividing the initial execution workflow according to a number of work nodes to obtain a number of subtasks;

[0126] A complexity calculation unit, used to determine the actual data volume and actual data type of the subtask, compare the actual data volume and actual data type with the historical execution database, obtain the historical average processing time and historical resource consumption, and determine the task complexity based on the historical average processing time and historical resource consumption;

[0127] A node allocation unit, connected to the complexity calculation unit, for allocating the subtask to a corresponding processing node according to the task complexity;

[0128] The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load status of the processing node in real time, and allocate computing power to each processing node based on the real-time load status to obtain a computing power allocation result.

[0129] Specifically, the task complexity described in the embodiment of the present invention = α1×historical average processing time+α2×historical resource consumption.

[0130] Specifically, an embodiment of the node allocation unit of the embodiment of the present invention can be to evaluate the available processing nodes, obtain the current load status, processing capacity and status of the assigned tasks of each processing node;

[0131] For subtasks with high task complexity, the node allocation unit will give priority to processing nodes with strong processing capabilities and relatively low current load for allocation;

[0132] For subtasks with lower task complexity, they are assigned to processing nodes with relatively weaker processing capabilities or slightly higher current loads while ensuring load balancing.

[0133] Specifically, an embodiment of the computing power allocation unit described in an embodiment of the present invention includes:

[0134] The workflow execution engine maintains a sliding window to record the resource usage of each node's task type in the last N (configurable) executions, from which it extracts average performance parameters (including: average resource utilization, average task progress, average idleness, historical platform load, and average downstream node demand);

[0135] Utilization deviation = current actual utilization - average resource utilization, schedule deviation = real-time idleness - average idleness, load deviation = platform load - average platform load, downstream demand deviation = current downstream node demand - average downstream node demand;

[0136] New quota = current quota × (1 + adjustment coefficient), adjustment coefficient = w1 × utilization deviation + w2 × progress deviation + w3 × queue backlog deviation + w4 × system load deviation + w5 × downstream demand deviation. Weights w1 to w5 can be adjusted dynamically according to specific scenarios. The default scenario is: w1>w5>w2>w3>w4, and the sum of w1 to w5 is 1.

[0137] Specifically, the embodiment of the present invention divides the initial execution workflow into multiple subtasks by setting the division unit, and decomposes the complex workflow into smaller and easier to manage units, thereby improving the flexibility and scalability of the system. Each subtask can be executed independently or work in conjunction with other subtasks to complete the entire data processing flow, thereby improving the efficiency of data processing. By setting the complexity calculation unit, the processing difficulty and resource requirements of each subtask can be accurately evaluated by comparing the data in the historical execution database. The historical average processing time and historical resource consumption provide an important reference basis for the evaluation, making task allocation more accurate and reasonable. By setting the node allocation unit, subtasks are reasonably allocated to different computing nodes according to the calculated task complexity, thereby optimizing task allocation, making full use of computing resources, and avoiding the situation where some nodes are overloaded while other nodes are idle, thereby achieving balanced utilization of resources and improving overall processing efficiency.

[0138] Specifically, the monitoring module 60 includes:

[0139] A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor;

[0140] An operation monitoring unit, used to monitor the performance parameters of the task in real time during operation, wherein the performance parameters include average response time and resource utilization;

[0141] A score calculation unit is used to determine the task completion rate based on the running state, and calculate the actual health score based on the task completion rate, the average response time and the resource utilization rate.

[0142] Specifically, a possible instance of the task status monitoring unit described in an embodiment of the present invention is: making a preliminary judgment based on a workflow status monitor, which is a custom Kubernetes controller built into the workflow execution engine, monitoring the execution status of each task node in the entire workflow to track the status of each task node, including startup, running, completion or failure.

[0143] Specifically, a possible example of the operation detection unit in the embodiment of the present invention is: implemented by a performance monitoring agent integrated in the workflow execution engine, which is responsible for collecting and analyzing performance data during task runtime, such as CPU usage, memory usage, disk I / O, and network transmission rate;

[0144] The average response time is obtained by measuring and averaging the response times of multiple tasks. For example, a task is operated 10 times, and the response time of each operation is 100ms, 120ms, 90ms, 110ms, 105ms, 95ms, 115ms, 100ms, 125ms, and 100ms respectively. Then the average response time = (100+120+90+110+105+95+115+100+125+100) / 10 = 105ms;

[0145] Resource utilization = CPU utilization × memory occupancy rate. Assuming that the CPU utilization rate is 70% and the memory occupancy rate is 60%, then resource utilization = 70% × 60% = 42%.

[0146] Specifically, a possible example of the scoring calculation unit in the embodiment of the present invention is: task completion rate = total number of tasks / number of completed tasks × 100%. Assuming that the total number of tasks is 100 and the number of completed tasks is 80, then the task completion rate is 80 / 100 × 100% = 80%;

[0147] The calculation formula of the actual health score S is: S = 0.5 × task completion rate + 0.3 × average response time + 0.2 × resource utilization rate;

[0148] Assuming the task completion rate is 80% (converted to 0.8), the average response time is 105ms (normalized to 0.7, assuming the normalized range is 0-1, 105ms is at a medium level), and the resource utilization rate is 42% (converted to 0.42), then the actual health score S = 0.5×0.8+0.3×0.7+0.2×0.42=0.4+0.21+0.084=0.694.

[0149] Specifically, the adjustment module 70 includes:

[0150] A comparison unit, used to compare the actual health score with the preset health score to obtain a comparison result;

[0151] a classification unit, configured to determine that the initial execution workflow fails when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a number of processing nodes, and classify the initial execution workflow failure into a temporary failure and a permanent failure according to the analysis result;

[0152] The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain the target workflow.

[0153] Specifically, the embodiment of the present invention formulates a processing strategy according to the failure category of the initial execution workflow, including:

[0154] For temporary failures, the processing strategies include: 1. Automatic retry mechanism: implement a retry strategy with gradually increasing intervals (exponential backoff), set the maximum number of retries, and avoid infinite retries that lead to resource waste. 2. Dynamic resource adjustment: before retrying, moderately increase the resources allocated to the failed task (such as CPU, memory), monitor resource usage, and dynamically adjust to alleviate resource pressure. 3. Load balancing: If the failure is caused by node overload, try to reschedule the task to a node with a lighter load. 4. Network fault handling: For temporary network-related failures, implement a short waiting strategy before trying to reconnect.

[0155] The processing strategies for permanent failures include: 1. Stop retrying immediately: After identifying a permanent failure, stop automatic retrying immediately to avoid wasting resources. 2. Error reporting and log aggregation: Generate detailed error reports, including the cause of failure, related logs and environmental information, aggregate the error information and send it to the relevant development team or operation and maintenance personnel. 3. Rollback mechanism: If the failure is caused by recent changes, start the automatic rollback process and restore to the last known stable version. 4. Resource release and cleanup: Ensure that all resources occupied by failed tasks are released to prevent resource leakage. 5. Update the failure knowledge base: Add the newly identified permanent failure mode to the failure knowledge base for future rapid identification and processing.

[0156] See also Figure 4 As shown, an embodiment of the present invention further provides a method for an intelligent feature exploration system for structured data, the method comprising:

[0157] Step S100, obtaining an initial data set uploaded by a user, and preprocessing the initial data set to obtain a structured data set;

[0158] Step S200, analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results;

[0159] Step S300, calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result;

[0160] Step S400, generating an initial execution workflow according to the preset operation rules of the execution algorithm unit;

[0161] Step S500, dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result;

[0162] Step S600, executing the computing power allocation result through each processing node, monitoring the operating status of each processing node in real time, collecting operating data, and calculating the actual health score of the initial execution workflow based on the operating data;

[0163] Step S700, comparing the actual health score with the preset health score to obtain a comparison result, and adjusting the initial execution workflow based on the comparison result to obtain a target execution workflow.

[0164] Specifically, a method for an intelligent feature exploration system for structured data provided by an embodiment of the present invention can be applied to the above-mentioned intelligent feature exploration system for structured data to achieve the same technical effect, which will not be repeated here.

[0165] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent feature exploration system for structured data, characterized in that: include: A preprocessing module is used to obtain the initial data set uploaded by the user, preprocess the initial data set, and obtain a structured data set; A feature acquisition module, connected to the preprocessing module, for analyzing the structured data set to acquire structural features and type features of the structured data set according to the analysis results; an algorithm determination module, connected to the feature acquisition module, for calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result; A generation module, connected to the algorithm determination module, for generating an initial execution workflow according to a preset operation rule of the execution algorithm unit; an allocation module, connected to the generation module, for dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result; A monitoring module, connected to the allocation module, for executing the computing power allocation result through each processing node, monitoring the operation status of each processing node in real time, collecting operation data, and calculating the actual health score of the initial execution workflow based on the operation data; The adjustment module is connected to the monitoring module and is used to compare the actual health score with the preset health score to obtain a comparison result, and adjust the initial execution workflow based on the comparison result to obtain a target execution workflow.

2. The intelligent feature exploration system for structured data according to claim 1, characterized in that: The preprocessing module comprises: An integrity detection unit, used to detect the integrity of the initial data set and obtain an integrity detection result; A correctness detection unit, used to detect the correctness of the initial data set and obtain a correctness detection result; a calibration unit, configured to calibrate the initial data set according to the integrity detection result and the correctness detection result to obtain a calibration data set; A processing unit is connected to the calibration unit and is used to perform structured processing on the calibration data set to obtain the structured data set.

3. The intelligent feature exploration system for structured data according to claim 2, characterized in that: The feature acquisition module comprises: a type determination unit, configured to identify field types of a plurality of fields in the structured data set, and determine the type characteristics based on the plurality of field types; an identification unit, used to identify data distribution characteristics and data relationship characteristics of the structured data set; The structure determination unit is connected to the identification unit and is used to determine the structure feature based on the data distribution feature and the data relationship feature.

4. The intelligent feature exploration system for structured data according to claim 3, characterized in that: The identification unit comprises: A calculation subunit, used to calculate the mean, median, standard deviation, skewness and kurtosis corresponding to each field in the structured data set; a distribution determination subunit, connected to the calculation subunit, for determining the data distribution characteristics based on the mean, the median, the standard deviation, the skewness and the kurtosis; The correlation determination subunit is used to calculate the actual correlation coefficient corresponding to each field in the structured data set based on the correlation analysis algorithm, and determine the data relationship characteristics based on a number of actual correlation coefficients.

5. The intelligent feature exploration system for structured data according to claim 4, characterized in that: The algorithm determination module comprises: A first calculation unit, used for calculating an actual structure evaluation value according to the mean, the median, the standard deviation, the skewness and the kurtosis; A second calculation unit, used for determining an actual correlation coefficient evaluation value according to the actual correlation coefficients; A third calculation unit is connected to the first calculation unit and the second calculation unit, and is used to calculate a first difference between the actual structure evaluation value and the preset structure evaluation value, calculate a second difference between the actual correlation coefficient evaluation value and the preset correlation coefficient evaluation value, and determine a corresponding execution algorithm unit based on the calculation results of the first difference and the second difference.

6. The intelligent feature exploration system for structured data according to claim 5, characterized in that: The generation module comprises: A first determining unit, used to determine a preset execution order and a preset dependency relationship of each execution unit in the execution algorithm unit; A generating unit is connected to the first determining unit and is used to generate an initial execution workflow according to the preset execution order and the preset dependency relationship.

7. The intelligent feature exploration system for structured data according to claim 6, characterized in that: The allocation module comprises: A division unit, used for dividing the initial execution workflow according to a number of work nodes to obtain a number of subtasks; A complexity calculation unit, used to determine the actual data volume and actual data type of the subtask, compare the actual data volume and actual data type with the historical execution database, obtain the historical average processing time and historical resource consumption, and determine the task complexity based on the historical average processing time and historical resource consumption; A node allocation unit, connected to the complexity calculation unit, for allocating the subtask to a corresponding processing node according to the task complexity; The computing power allocation unit is connected to the node allocation unit and is used to detect the real-time load status of the processing node in real time, and allocate computing power to each processing node based on the real-time load status to obtain a computing power allocation result.

8. The intelligent feature exploration system for structured data according to claim 7, characterized in that: The monitoring module comprises: A task status monitoring unit, used to monitor the running status of tasks on each processing node based on the status monitor; An operation monitoring unit, used to monitor the performance parameters of the task in real time during operation, wherein the performance parameters include average response time and resource utilization; A score calculation unit is used to determine the task completion rate based on the running state, and calculate the actual health score based on the task completion rate, the average response time and the resource utilization rate.

9. The intelligent feature exploration system for structured data according to claim 8, characterized in that: The adjustment module comprises: A comparison unit, used to compare the actual health score with the preset health score to obtain a comparison result; a classification unit, configured to determine that the initial execution workflow fails when the actual health score is less than the preset health score, analyze error logs and resource usage corresponding to a plurality of processing nodes, and classify the initial execution workflow failure into a temporary failure and a permanent failure according to the analysis result; The adjustment unit is connected to the classification unit and is used to formulate a processing strategy according to the failure category of the initial execution workflow, adjust the initial execution workflow based on the processing strategy, and obtain the target workflow.

10. A method for applying an intelligent feature exploration system for structured data as claimed in any one of claims 1 to 9, characterized in that: include: Obtain the initial data set uploaded by the user, preprocess the initial data set, and obtain a structured data set; Analyzing the structured data set to obtain structural features and type features of the structured data set according to the analysis results; Calculating an actual evaluation value based on the data structure feature and the data type feature, comparing the actual evaluation value with preset evaluation values ​​in a plurality of preset algorithm units, and determining an execution algorithm unit according to the comparison result; Generate an initial execution workflow according to the preset operation rules of the execution algorithm unit; Dividing the initial execution workflow into a number of subtasks, calculating the task complexity corresponding to the subtasks, allocating a number of processing nodes based on the task complexity, monitoring the real-time load conditions corresponding to the processing nodes in real time, and allocating computing power to each processing node based on the real-time load conditions to obtain a computing power allocation result; Execute the computing power allocation result through each processing node, monitor the operation status of each processing node in real time, collect operation data, and calculate the actual health score of the initial execution workflow based on the operation data; The actual health score is compared with a preset health score to obtain a comparison result, and the initial execution workflow is adjusted based on the comparison result to obtain a target execution workflow.

Images