Intelligent analysis and management method and system for enterprise data

Abstract

The invention relates to the technical field of business process management, in particular to an enterprise data intelligent analysis management method and system, and the method comprises the following steps: obtaining a business data timestamp, an input source, an operation type and a transmission path, analyzing a data flow node, and generating a business data flow dependency matrix. According to the method, abnormal data is screened based on stability calculation, the data quality is improved, error interference is reduced, task execution pressure and dependency are combined, task sorting is optimized, the scheduling sequence is dynamically adjusted, conflicts are reduced, execution efficiency is improved, the priority level is adjusted according to the execution progress and urgency degree, and efficient hierarchical scheduling is achieved; by combining resource demand, consumption value calculation, calculation node matching, resource allocation optimization, utilization rate improvement, waste reduction, data analysis accuracy improvement, task execution strategy optimization, resource matching efficiency improvement and business management intelligence level enhancement are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of business process management, and in particular, to an enterprise data intelligent analysis and management method and system. Background Art

[0002] The technical field of business process management includes related methods and systems for planning, executing, monitoring, and optimizing internal and cross-departmental business activities within an enterprise. The core content includes the standardized management and optimization of business activities through structured process modeling, process automation, process monitoring, and data analysis means. This technical field mainly involves aspects such as process definition, task allocation, data flow, and business rule management, and usually relies on computer systems to complete process logic control, task coordination, and data processing. With the development of information technology, this field is widely applied in multiple scenarios such as enterprise resource management, supply chain management, and customer relationship management, improving business operation efficiency through digital means.

[0003] Among them, the enterprise data intelligent analysis and management method refers to the specific method of using data acquisition, data preprocessing, data analysis, and visualization technologies to analyze and manage enterprise operation data. It covers data collection and storage, performs structured data storage through a database management system, and establishes an analysis model based on data modeling technology; it covers data preprocessing, and uses data cleaning, anomaly detection, and feature extraction means to ensure data quality; it covers data analysis, and extracts data features through pattern recognition, classification analysis, and clustering calculation methods; it covers data visualization, and uses graphics rendering, statistical plotting, and interactive interface design methods to display analysis results. This method provides quantitative support for the operation of an enterprise through intelligent analysis of business process data.

[0004] In the prior art, during the process of business data flow parsing, there is a lack of in-depth analysis of data path dependencies, resulting in an unclear data flow chain during task execution, making it difficult to accurately grasp the relevance of business data, and affecting the monitoring and analysis of data stability. Abnormal data screening usually adopts a simple threshold determination method, which is difficult to accurately identify data deviation, and some abnormal data cannot be removed in time, affecting the accuracy of data calculation. Task scheduling mainly relies on static rules and fails to dynamically analyze task pressure, dependency relationships, and urgency, resulting in inflexible adjustment of task execution order, and there may be situations where tasks are congested or low-priority tasks occupy too many resources. The resource allocation strategy is relatively fixed and fails to fully combine task resource requirements and the load conditions of computing nodes for dynamic matching, resulting in some computing nodes being in a high-load state for a long time, while other nodes are in an inefficient operating state, reducing the overall utilization rate of computing resources. The prior art has deficiencies in aspects such as data flow analysis, abnormal data processing, task scheduling optimization, and dynamic resource allocation, affecting the accuracy of business data management and the efficiency of business process execution. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an enterprise data intelligent analysis management method and system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a method for intelligent analysis and management of enterprise data, comprising the following steps: S1: Obtain business data timestamp, input source, operation type, and transmission path, parse data flow nodes, and generate a business data flow dependency matrix; S2: Based on the business data flow dependency matrix, extract the task execution nodes, calculate the data stability, analyze the data change range to determine the data abnormality offset, and generate an abnormal business data screening table; S3: Call the abnormal business data screening table to remove abnormal data, extract task trigger factors to calculate task execution pressure, parse task dependencies to calculate task load levels, adjust task execution order according to load levels, compare task urgency values, call task execution to optimize task sorting, calculate execution conflict rate to adjust task scheduling, and generate a task priority sorting table; S4: Based on the task priority ranking table, extract the task execution progress and calculate the remaining time, compare the task urgency value, adjust the execution priority level, and generate a task hierarchical scheduling list; S5: Based on the task hierarchical scheduling list, extract the task resource requirements to calculate the resource consumption value, analyze the computing node load to determine the task adaptation node, and generate an enterprise data analysis management plan.

[0007] As a further solution of the present invention, the business data flow dependency matrix includes data transmission links, data association weights, and business node mapping relationships; the abnormal business data screening table includes abnormal offset thresholds, abnormal impact factors, and data stability indicators; the task priority sorting table includes task load levels, task execution priorities, and task scheduling conflict coefficients; the task hierarchical scheduling list includes task execution status identification, task level classification, and task scheduling adjustment parameters; the enterprise data analysis management plan includes computing resource allocation parameters, task execution matching degree, and task migration adaptability indicators.

[0008] As a further solution of the present invention, the specific steps of obtaining the business data timestamp, input source, operation type, transmission path, parsing the data flow node, and generating the business data flow dependency matrix are as follows: S101: obtaining the timestamp, input source, operation type and transmission path of the business data, extracting the associated attributes of the data record, sorting the business data according to the timestamp, calculating the interval between adjacent timestamps, marking the association between the data source and the operation type, and combining the transmission path record flow order to obtain the data flow sequence; S102: Based on the data flow sequence, analyze the data interaction situation of the flow nodes, calculate the data reception volume and output volume of the nodes, extract the transmission paths of the input sources, count the number of times the data from different sources flows through each node, filter the paths with the number of flows exceeding the set threshold, calculate the interaction frequency of the nodes, extract the main channels of data circulation, and obtain the data flow node correlation degree; S103: According to the data flow node correlation degree, construct a data dependency matrix, calculate the input and output dependency degrees of the nodes, filter the key nodes with the dependency degree exceeding the threshold, establish a data flow network, and generate a business data flow dependency matrix.

[0009] As a further solution of the present invention, the specific steps for extracting task execution nodes based on the business data flow dependency matrix, calculating data stability, analyzing the data change range to judge the data abnormal offset amount, and generating an abnormal business data screening table are as follows: S201: Obtain the business data flow dependency matrix, extract task execution nodes, count the number of data interactions of the nodes, filter the nodes with the data reception and transmission volume exceeding the threshold, and obtain a task execution node list; S202: Based on the task execution node list, calculate data stability, count the data input fluctuation range of the nodes, calculate the change rate of adjacent data points, filter the nodes with the change rate exceeding the threshold, calculate the mean value and standard deviation of the change range, compare the deviation degree of the data fluctuation value from the mean value, filter the stable nodes, and obtain the data stability coefficient; S203: According to the data stability coefficient, analyze the data change range, calculate the data offset amount, filter the data with the offset amount exceeding the threshold, mark the abnormal data, and generate an abnormal business data screening table.

[0010] As a further solution of the present invention, the specific steps for calling the abnormal business data screening table to eliminate abnormal data, extracting task trigger factors to calculate task execution pressure, analyzing task dependency relationships to calculate task load levels, adjusting task execution orders according to load levels, comparing task urgency values, calling task execution to optimize task sorting, calculating execution conflict rates to adjust task scheduling, and generating a task priority sorting table are as follows: S301: Call the abnormal business data screening table, eliminate abnormal data, filter task records, extract task trigger factors, and calculate task execution pressure to obtain a task execution pressure coefficient; S302: Based on the task execution pressure coefficient, analyze task dependency relationships, analyze the number of task triggers, count the correlation degree between tasks, filter the tasks with the dependency degree exceeding the threshold, identify the load levels of the tasks, filter the tasks with the load levels exceeding the limit, adjust the task execution order, calculate task urgency values, call task execution to optimize task sorting, and obtain task execution priorities; S303: Calculate the execution conflict rate according to the task execution priority, filter tasks with a conflict rate exceeding the threshold, adjust the task scheduling, update the task sorting, and generate a task priority sorting table.

[0011] As a further solution of the present invention, the specific formula for the task urgency value is: ; represents task and task the degree of association between them, represents task at the th trigger execution time, represents task at the th trigger execution time, represents the total number of trigger times of the task, indicates the summation calculation for all trigger times, represents task and task at the th trigger absolute value of the execution time difference, represents task square root of the sum of squares of trigger times, represents task square root of the sum of squares of trigger times, represents the dependency threshold, represents task CPU resource occupancy ratio of, represents task storage resource occupancy ratio of, represents task bandwidth resource occupancy ratio of, represents the load threshold, represents task remaining time of, represents task urgency value of.

[0012] As a further solution of the present invention, based on the task priority sorting table, the specific steps for extracting the task execution progress to calculate the remaining duration, comparing the task urgency value, adjusting the execution priority level, and generating a task hierarchical scheduling list are: S401: Call the task priority sorting table, extract the task execution progress, calculate the remaining duration of the task, filter tasks with an execution progress lower than the threshold, and obtain a task remaining time matrix; S402: Based on the remaining task time matrix, compare the task urgency values, calculate the task priority adjustment amplitude value, screen tasks with urgency exceeding the threshold, count the execution dependencies of the adjusted tasks, analyze the impact of task priority changes on the dependency relationship, and adjust the task execution order. Screen high-dependency tasks and optimize the sorting to obtain the task priority levels. S403: According to the task priority levels, adjust the task scheduling, allocate task execution batches, screen scheduling conflict tasks, update the task execution order, and generate a task hierarchical scheduling list.

[0013] As a further solution of the present invention, the formula for the priority adjustment amplitude value is as follows: ; represents the priority adjustment amplitude value of task , represents the degree of dependence between task and task , represents the urgency value of task , represents the urgency value of task , represents the summation operation on all tasks that depend on task , represents task and the absolute value of the difference in urgency values between task , represents the square root of the sum of the squares of the degrees of dependence of the tasks that depend on task represents the threshold for task priority adjustment.

[0014] As a further solution of the present invention, based on the task hierarchical scheduling list, the specific steps for extracting task resource requirements, calculating resource consumption values, parsing the load of computing nodes, and determining task-adapted nodes to generate an enterprise data analysis and management solution are as follows: S501: Invoke the task hierarchical scheduling list, extract task resource requirements, count the consumption of computing, storage, and bandwidth, calculate the resource occupancy ratio of tasks, and obtain the task resource consumption matrix; S502: Based on the task resource consumption matrix, parse the load of computing nodes, count the node resource occupancy ratio, calculate the task adaptation degree, screen tasks with a matching degree lower than the threshold, analyze the task distribution balance degree, screen nodes with overloaded loads, adjust the task allocation strategy, optimize the task scheduling order, and obtain the task-adapted node list; S503: Adapt the node list according to the task, adjust resource allocation, screen scheduling conflict tasks, optimize the task execution order, update the task allocation plan, and generate an enterprise data analysis management plan.

[0015] An enterprise data intelligent analysis and management system, comprising: The data flow module obtains the business data timestamp, input source, operation type, and transmission path, analyzes the data flow nodes, verifies the data accuracy, and generates a business data flow dependency matrix; The exception screening module extracts the task execution nodes based on the business data flow dependency matrix, analyzes the data change range to judge the data abnormal offset, and generates an abnormal business data screening table; The task scheduling module calls the abnormal business data screening table to eliminate abnormal data, extracts the task trigger factors to calculate the task execution pressure, analyzes the task dependency relationship to calculate the task load level, adjusts the task execution order according to the load level, compares the task urgency values, calculates the execution conflict rate to adjust the task scheduling, and generates a task priority ranking table; The execution management module extracts the task execution progress based on the task priority ranking table to calculate the remaining duration, compares the task urgency values, and generates a task hierarchical scheduling list; The resource optimization module extracts the task resource requirements based on the task hierarchical scheduling list to calculate the resource consumption value, verifies the accuracy of the resource consumption value, analyzes the computing node load to judge the task adaptation nodes, and generates an enterprise data analysis management plan.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In the present invention, by screening abnormal data based on stability calculation, the data quality is improved, the error interference is reduced, the task execution pressure and dependency relationship are combined, the task sorting is optimized, the scheduling order is dynamically adjusted, the conflict is reduced, the execution efficiency is improved, the priority level is adjusted according to the execution progress and urgency, the efficient hierarchical scheduling is realized, the resource consumption value is calculated in combination with the resource requirements, the computing nodes are matched, the resource allocation is optimized, the utilization rate is improved, the waste is reduced, the data analysis accuracy is improved, the task execution strategy is optimized, the resource matching efficiency is improved, and the intelligent level of business management is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic diagram of the step flow of the present invention; Figure 2 It is the flowchart of step S1 of the present invention; Figure 3 It is the flowchart of step S2 of the present invention; Figure 4 It is the flowchart of step S3 of the present invention; Figure 5 It is the flowchart of step S4 of the present invention; Figure 6 It is the flowchart of step S5 of the present invention; Figure 7 It is the system module diagram of the present invention. Detailed implementation manners

[0019] Next, in conjunction with the accompanying drawings, the technical solutions in the present invention will be described.

[0020] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of the word "example" is intended to present concepts in a specific manner. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two can be selected.

[0021] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same. "(of)", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same.

[0022] In the embodiments of the present invention, sometimes subscripts such as W 1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meanings they express are the same.

[0023] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments.

[0024] Please refer to Figure 1 , an enterprise data intelligent analysis and management method, including the following steps: S1: Obtain the business data timestamp, input source, operation type, transmission path, analyze the data flow transfer nodes, and generate a business data flow transfer dependency matrix; S2: Based on the business data flow dependency matrix, extract task execution nodes, calculate data stability, analyze the data change range to determine the data abnormal offset, and generate a screening table for abnormal business data; S3: Invoke the screening table for abnormal business data to eliminate abnormal data, extract task trigger factors to calculate task execution pressure, analyze task dependency relationships to calculate task load levels, adjust the task execution order according to the load level, compare the task urgency values, invoke the task execution optimization for task sorting, calculate the execution conflict rate to adjust task scheduling, and generate a task priority sorting table; S4: Based on the task priority sorting table, extract the task execution progress to calculate the remaining duration, compare the task urgency values, and adjust the execution priority level to generate a task hierarchical scheduling list; S5: Based on the task hierarchical scheduling list, extract the task resource requirements to calculate the resource consumption value, analyze the load of computing nodes to determine task adaptation nodes, and generate an enterprise data analysis management plan.

[0025] The business data flow dependency matrix includes data transmission links, data association weights, and business node mapping relationships. The screening table for abnormal business data includes abnormal offset thresholds, abnormal impact factors, and data stability indicators. The task priority sorting table includes task load levels, task execution priorities, and task scheduling conflict coefficients. The task hierarchical scheduling list includes task execution status identifiers, task level classifications, and task scheduling adjustment parameters. The enterprise data analysis management plan includes computing resource allocation parameters, task execution matching degrees, and task migration adaptability indicators.

[0026] Please refer to Figure 2 , the specific steps of S1 are: S101: Obtain the timestamp, input source, operation type, and transmission path of business data, extract the associated attributes of data records, sort the business data according to the timestamp, calculate the adjacent timestamp intervals, mark the association relationships between data sources and operation types, and combine the transmission path records to obtain the data flow sequence; To obtain the timestamp of business data, it is necessary to extract the field corresponding to the event occurrence time from the data record. This field is usually stored with millisecond or second-level precision. The input source involves the origin of the data at the time of collection, which can be a database table, a log file, an API interface, etc. The operation type refers to different processing methods such as creating, modifying, and deleting data records. The transmission path is associated with the flow direction of the data from the input source to the target storage location. First, it is necessary to preprocess the data to standardize the data from different sources so that fields such as timestamps and operation types can be stored uniformly. Then, sort the data according to the timestamp. The specific operation is to arrange all records in ascending order according to the timestamp field and calculate the time interval between adjacent timestamps. The time interval calculation method is to subtract the timestamp of the previous record from the timestamp of the current record to obtain the data change frequency. Suppose the timestamp data are 1700000000, 1700000020, and 1700000055 respectively, then the time intervals are calculated as 20 seconds and 35 seconds in sequence. To mark the association relationship between the data source and the operation type, a mapping needs to be established. Combine the data source field and the operation type field to form a unique identifier. For example, if a certain record comes from the "log file" and its operation type is "modify", then the unique identifier "log file - modify" can be formed. Combining the transmission path record transfer sequence, it is necessary to trace multiple intermediate nodes of the data from the source to the target storage according to the data storage path. For example, the data is transmitted from the "collection server" to the "transfer database" and then to the "data warehouse". Each step records the transfer timestamp, path, and data content change situation, and finally obtains the data transfer sequence.

[0027] S102: Based on the data transfer sequence, analyze the data interaction situation of the transfer nodes, calculate the data reception volume and output volume of the nodes, extract the transmission path of the input source, count the transfer times of the data from different sources at each node, filter the paths whose transfer times exceed the set threshold, calculate the interaction frequency of the nodes, extract the main channels of data circulation, and obtain the data transfer node correlation degree; Based on the data transfer sequence to analyze the data interaction of transfer nodes, first, it is necessary to extract the data reception volume and output volume of each node. The data reception volume refers to the number of data items received by a node from upstream nodes, and the data output volume refers to the number of data items transmitted by a node to downstream nodes. Suppose a node N1 receives 100 data items and outputs 90 to N2, then its reception volume is 100 and the output volume is 90. To extract the transmission path of the input source, it is necessary to record the upstream node for each piece of data. For example, if data A comes from node N1 and is transmitted to node N2, then record its path as "N1→N2". To count the transfer times of data from different sources at each node, that is, to count the occurrence frequency of different source data at the current node. Suppose 80 of the data received by N2 come from N1 and 20 come from N3, then the transfer times of N1-N2 is 80, and the transfer times of N3-N2 is 20. Filter the paths with transfer times exceeding the set threshold. Suppose the threshold is set to 50, then only keep the N1-N2 path. Calculate the interaction frequency of the node, that is, count the number of data transmissions in each time period of the current node. Suppose N2 has 50 data exchanges in one minute, then the interaction frequency is 50 times / minute. To extract the main channel of data circulation, it is necessary to sort according to the interaction frequency and transfer times of each node, and filter out the main transfer paths. For example, if the transfer times of the path N1→N2→N4 are the highest, then this path becomes the main channel of data circulation, and finally, the data transfer node correlation is obtained.

[0028] S103: Based on the data transfer node correlation, construct a data dependency matrix, calculate the input and output dependency degrees of the nodes, filter out the key nodes with dependency degrees exceeding the threshold, establish a data transfer network, and generate a business data transfer dependency matrix; Construct a data dependency matrix according to the data transfer node correlation. The data dependency matrix represents the dependency relationship between nodes. Each matrix element represents the data transfer intensity between two nodes, and the calculation method is the historical data transfer times between nodes. Suppose the data transfer times from node N1 to N2 is 100, and the data transfer times from N2 to N3 is 80, then the corresponding elements in the dependency matrix are 100 and 80 respectively. To calculate the input and output dependency degrees of the nodes, it is necessary to normalize the input transfer volume and output transfer volume of each node. For example, if a node receives 1000 data items and outputs 800 data items, then the input dependency degree is 1000 / (1000 + 800) = 0.56, and the output dependency degree is 800 / (1000 + 800) = 0.44. Filter out the key nodes with dependency degrees exceeding the threshold. Suppose the threshold is set to 0.5, then filter out the nodes with input or output dependency degrees exceeding 0.5. To establish a data transfer network, it is necessary to establish a directed graph structure according to the data transfer relationship in the dependency matrix, where the nodes represent data processing units, the edges represent data transfer paths, and the weights are the transfer times. Finally, generate a business data transfer dependency matrix.

[0029] Please refer to Figure 3 , and the specific steps of S2 are as follows: S201: Obtain the business data flow dependency matrix, extract the task execution nodes, count the number of data interactions of the nodes, filter out the nodes with data reception and transmission volume exceeding the threshold, and obtain the task execution node list; To obtain the business data flow dependency matrix, it is necessary to extract the historical data transmission relationships between each node from the data flow network, construct the data flow times of each node into a matrix form, where each matrix element corresponds to the total data transmission volume between two nodes. To extract the task execution nodes, it is necessary to count the number of data interactions of each node from this matrix. The number of interactions is equal to the input data volume plus the output data volume of the node. Suppose the input data of a certain node N1 is 1500 pieces and the output data is 1200 pieces, then its number of interactions is 2700 times. To filter out the nodes with data reception and transmission volume exceeding the threshold, it is necessary to set the threshold T. If T is set to 2000, then the nodes with the number of interactions higher than 2000 are filtered out as task execution nodes. Suppose the number of interactions of N1, N2, and N3 are 2700, 1900, and 3100 respectively, then only N1 and N3 meet the screening conditions, and finally the task execution node list is obtained.

[0030] S202: Based on the task execution node list, calculate the data stability, count the data input fluctuation range of the nodes, calculate the change rate of adjacent data points, filter out the nodes with the change rate exceeding the threshold, calculate the mean value and standard deviation of the change range, compare the deviation degree of the data fluctuation value from the mean value, filter out the stable nodes, and obtain the data stability coefficient; First, it is necessary to count the data input fluctuation range of each task execution node. The calculation method of the fluctuation range is the difference between the maximum input volume and the minimum input volume within a certain period. For example, the maximum input data of N1 within five minutes is 1600 pieces, and the minimum is 1300 pieces, then the fluctuation range is 300 pieces. To calculate the change rate of adjacent data points, it is necessary to obtain the data input values at each time point and calculate the change amplitude between adjacent time points. Divide the change value by the data value at the previous time point to obtain the change rate. For example, the input data of a certain node at time t1 is 1200 pieces, and at time t2 is 1400 pieces, then the change rate = (1400 - 1200) / 1200 = 16.67%. To filter out the nodes with the change rate exceeding the threshold, it is necessary to set the change rate threshold R. If R is set to 15%, then the nodes with the change rate exceeding 15% are filtered out. Calculate the mean value and standard deviation of the change range. The mean value calculation method is the average value of all fluctuation values within a period of time. The standard deviation calculation method is to calculate the square root of the mean value of the sum of the squared deviations of each fluctuation value from the mean value. For example, the fluctuation values of N1 are 300, 250, 280, 310, then the mean value is (300 + 250 + 280 + 310) / 4 = 285, and the standard deviation calculation formula is ; Substituting the numerical value, we get the standard deviation of 25.3. Comparing the degree of deviation between the data fluctuation value and the mean, the calculation method is whether the difference between the fluctuation value of a data point and the mean exceeds the standard deviation multiple threshold. If the fluctuation value of a node is 320, the deviation threshold is set to 1.5 times the standard deviation, that is, 1.5×25.3=37.95, then 320-285=35 is less than 37.95, and the data fluctuation value of the node does not exceed the threshold. Screening stable nodes means retaining nodes whose data fluctuation does not exceed the threshold, and finally obtaining the data stability coefficient.

[0031] S203: according to the data stability coefficient, analyze the data change range, calculate the data offset, filter the data with the offset exceeding the threshold, mark the abnormal data, and generate the abnormal business data filtering table; According to the data stability coefficient, the data change range is analyzed. It is necessary to extract the historical data input fluctuations of each task execution node and calculate the data offset. The offset is calculated as the difference between the current data value and the historical mean. For example, the historical input data mean of a node is 1400 and the current data value is 1550. The offset is 1550-1400=150. To filter out data with an offset exceeding the threshold, a threshold D needs to be set. If D is set to 200, node data with an offset less than 200 will not be filtered. If the input data of a node is 1700 and the historical mean is 1400, the offset is 300. If it is greater than 200, the data is filtered as abnormal data. To mark abnormal data, the filtered data needs to be marked and stored in the abnormal data record table, and finally an abnormal business data filtering table is generated.

[0032] See also Figure 4 , the specific steps of S3 are: S301: calling the abnormal business data screening table, removing abnormal data, screening task records, extracting task trigger factors, calculating task execution pressure, and obtaining task execution pressure coefficient; The abnormal service data screening table needs to read data from the screened abnormal data records. The screened data fields include task ID, timestamp, task input data volume, task execution status, etc. To eliminate abnormal data, it is necessary to compare the historical data patterns of the tasks and remove the data points where the input data fluctuation exceeds the set threshold. Suppose the average value of the historical input data of a certain task is 1500 pieces, and the current input data reaches 1900 pieces. If the set abnormal threshold is 20%, that is, 1500×1.2 = 1800, then the current data point is eliminated. Screening task records needs to be classified according to the task ID. Extracting the task trigger factor requires analyzing the task trigger conditions, including timed trigger, event trigger, dependent task trigger, etc. Based on the timestamp in the task log record, calculate the task execution pressure, that is, the amount of data processed by the task per unit time. If a certain task T1 is executed 3 times within one minute and 500 pieces of data are processed each time, then the task execution pressure is 500×3 = 1500 pieces / minute, and finally the task execution pressure coefficient is obtained.

[0033] S302: Based on the task execution pressure coefficient, analyze the task dependency relationship, analyze the task trigger times, count the correlation degree between tasks, screen the tasks with a dependency degree exceeding the threshold, identify the load level of the tasks, screen the tasks with an over-limit load level, adjust the task execution order, calculate the task urgency value, call the task execution optimization task sorting, and obtain the task execution priority; The specific formula for calculating the task urgency value is: ; represents the task urgency value, represents the task at the th trigger time, represents the task at the th trigger time, represents the total number of trigger times of the task, indicates the summation calculation for all trigger times, represents the task and the task at the th trigger time difference of the absolute value, represents the task square root of the sum of squares of the trigger times, represents the task square root of the sum of squares of the trigger times, represents the dependency threshold, represents the task CPU resource occupancy ratio, represents the task storage resource occupancy ratio, The bandwidth resource occupancy ratio representing the task , the load threshold represents the task remaining time represents the task urgency value

[0034] First, it is necessary to obtain the trigger time data recorded during the execution of the task, classify all tasks according to the unique task ID, extract the task trigger time series, and record the trigger time of each task as , where represents the task number represents the th trigger of the task. When calculating the correlation degree between tasks, it is necessary to calculate the trigger frequency of different tasks within the same time period and calculate the absolute value of the time interval. The trigger time points of task are 10:00, 10:05, 10:15, and the trigger time points of task are 10:02, 10:07, 10:20. Then the corresponding time intervals are 2min, 2min, 5min respectively. When calculating the task dependency, it is necessary to introduce the calculation formula of the task trigger time difference

[0035] To screen tasks with dependency exceeding the threshold, a threshold needs to be set. Assuming , then all task pairs with a dependency are screened out and used for subsequent load analysis. When identifying the load level of tasks, it is necessary to calculate the CPU resource consumption ratio , storage resource occupancy ratio , bandwidth resource occupancy ratio of each task. The calculation method is to divide the resources consumed by the task by the total resource pool. For example, if the CPU requirement of task is 8 cores and the total CPU resources are 64 cores, then . To screen tasks with load level exceeding the limit, a load threshold needs to be set. Assuming , then all tasks with a load exceeding this value are re-prioritized. When adjusting the task execution order, it is necessary to calculate the priority execution level of the task in combination with the task urgency value. The urgency value is calculated as follows ; where is the remaining time of the task, in minutes. For example, if the expected completion time of a task is 12:00 and the current time is 11:45, then min, and substituting it into the formula to calculate the urgency value is , tasks with high urgency values are executed first, and finally the task execution optimization task sorting is called to obtain the task execution priority.

[0036] The given task trigger time series is as follows: ; ; Calculate The value is as follows: min min min Calculate the sum of squares: ; ; ; ; Calculate the degree of association: ; At this time is below the threshold , so the task and have a low degree of dependence.

[0037] Given the task The resource occupancy is as follows: ; ; ; Calculate the urgency value: min ; At this time is lower than other tasks, so the priority is lower.

[0038] By calculating the absolute time difference of the task trigger time and combining the sum of squares of the task execution time for normalized calculation of the dependence degree, the correlation between different tasks can be measured by the relative time difference degree, avoiding misjudgments that may be caused by traditional simple trigger frequency statistics. At the same time, by introducing task load calculation and urgency calculation, high-load and high-priority tasks can be accurately screened to optimize the task execution order and make resource scheduling more reasonable. The result shows that the tasks and have a low correlation, and the task The computing resource consumption is low and the urgency is low, so the tasks and can be executed in parallel, and their priorities need to be lowered.

[0039] S303: According to the task execution priorities, calculate the execution conflict rate, filter out the tasks with conflict rate exceeding the threshold, adjust the task scheduling, update the task sorting, and generate a task priority sorting table; Calculating the execution conflict rate according to the task execution priorities requires checking the concurrent execution of multiple tasks within the same time window. Filtering out the tasks with conflict rate exceeding the threshold requires setting a concurrent conflict threshold C. Assuming C is set to 40%, if the conflict rate of a certain task group is calculated as 45%, it exceeds the threshold. Adjusting the task scheduling requires reasonably allocating the task execution time within the same time period to avoid high-priority tasks competing for resources with other tasks. Updating the task sorting requires rearranging the task execution order according to the adjusted task scheduling table. Finally, a task priority sorting table is generated.

[0040] Please refer to Figure 5 , and the specific steps of S4 are as follows: S401: Call the task priority sorting table, extract the task execution progress, calculate the remaining task duration, filter out the tasks with execution progress lower than the threshold, and obtain a task remaining time matrix; Calling the task priority sorting table requires extracting task information from the generated task priority data. Each task record contains fields such as task ID, current execution progress, and estimated completion time. Extracting the task execution progress requires calculating the ratio of the completed task volume to the total task volume from the task log. For example, if the total processing data volume of a certain task T1 is 10,000 pieces and 7,000 pieces have been completed currently, the execution progress is 7,000 / 10,000 = 70%. Calculating the remaining task duration requires obtaining the difference between the current time and the estimated completion time. For example, if the estimated completion time of T1 is 12:00 and the current time is 11:30, the remaining duration is 30 minutes. Filtering out the tasks with execution progress lower than the threshold requires setting a progress threshold P. If P is set to 75%, the tasks with execution progress lower than 75% are filtered out. Assuming the progress of task T2 is 60%, then T2 meets the filtering conditions. Finally, the remaining time records of all tasks are formed into a matrix to obtain the task remaining time matrix.

[0041] S402: Based on the task remaining time matrix, compare the task urgency values, calculate the task priority adjustment amplitude value, filter out the tasks with urgency exceeding the threshold, count the execution dependencies of the adjusted tasks, analyze the impact of the task priority change on the dependency relationship, and adjust the task execution order. Filter out the high-dependency tasks and optimize the sorting to obtain the task priority levels; The formula for the priority adjustment amplitude value is as follows: ; Represents the priority adjustment amplitude value of the task , Represents the degree of dependence between task and task Represents the urgency value of the task , Represents the urgency value of the task , Represents the sum operation for all tasks that task depends on The absolute value of the difference in urgency values between task and task Represents the square root of the sum of the squares of the degrees of dependence of the tasks that task depends on Represents the threshold for task priority adjustment

[0042] When calculating the amount of change in task priority, it is necessary to normalize the urgency values of all tasks to ensure that the calculation results are not affected by the scale of the total number of tasks. When screening tasks with urgency exceeding the threshold, a threshold needs to be set. If is set , then all tasks need to have their task priorities adjusted. To count the execution dependencies of the adjusted tasks, it is necessary to traverse the task dependency matrix to find all directly or indirectly associated task sets. When analyzing the impact of task priority changes on dependencies, it is necessary to check whether the execution order of the tasks has changed. For example, if task depends on task but is executed first due to the high urgency of task , then it may be necessary to adjust the priority of

[0043] Ultimately, adjust the task execution order, screen high-dependency tasks and optimize the sorting to obtain the task priority levels The dependency relationship matrix of task execution needs to be obtained. The degree of dependence between each task and other tasks is calculated by statistically analyzing the triggering sequence and resource occupancy between tasks. The urgency value of each task

[0044] The given task data is as follows: Task Dependent task and , , The task urgency is calculated as follows: Remaining task time min Remaining task time min Remaining task time min Calculate the urgency value: ; ; ; Calculate the priority adjustment range: ; Calculate the sum of squares: ; Calculate the adjustment range value: ; ; Since is less than the set threshold , so the task does not need to adjust the priority, while other tasks need to be further screened. By calculating the difference in the task urgency value and performing normalization calculations in combination with the task dependency matrix, the adjustment of task priority is based on the actual execution dependency relationship rather than a simple urgency comparison. At the same time, using the square root of the sum of squares of task dependencies as the normalization factor can avoid the calculation result being amplified in the case of a large number of dependent tasks, thus ensuring that the adjustment of task priority is more accurate. The result shows that the priority adjustment range value of task is lower than the threshold and does not require priority adjustment, while other tasks need to be further screened. Task may need to increase the priority due to its high urgency. After adjusting the task priority, recalculate the task dependency relationship to optimize the task execution order.

[0045] S403: According to the task priority level, adjust the task scheduling, allocate the task execution batches, screen the scheduling conflict tasks, update the task execution order, and generate the task hierarchical scheduling list; To adjust task scheduling according to task priority levels, it is necessary to reallocate execution time based on task dependencies and priorities. To allocate task execution batches, it is necessary to divide task execution batches into multiple batches based on task priorities. For example, set tasks with priorities 1-3 as the first batch to be executed, and tasks with priorities 4-6 as the second batch to be executed. To filter out scheduling conflicting tasks, it is necessary to detect whether there are multiple tasks competing for the same computing resources in the same time period. For example, if both T5 and T6 need to use the same database interface and are planned to be executed at the same time, a scheduling conflict will occur. To update the task execution order, it is necessary to adjust the execution time of the conflicting tasks, stagger the tasks in the time windows, and finally generate a hierarchical task scheduling list.

[0046] See also Figure 6 , the specific steps of S5 are: S501: calling the task hierarchical scheduling list, extracting the task resource requirements, counting the computing, storage and bandwidth consumption, calculating the resource occupancy ratio of the task, and obtaining the task resource consumption matrix; Calling the task hierarchical scheduling list requires reading the task's computing, storage, and bandwidth requirement information from the task scheduling table. Each task contains a task ID, execution node, and resource consumption. To extract the task resource requirement, it is necessary to calculate the required CPU cores, memory usage, disk IO throughput, and network bandwidth usage based on the task execution logic. For example, the computing requirement of a task T1 is a 4-core CPU, the storage requirement is 16GB RAM, and the bandwidth requirement is 100Mbps. To calculate the computing, storage, and bandwidth consumption, it is necessary to calculate the total resource usage of all tasks separately. Assuming that there are 5 tasks in the system, and their computing resource requirements are 2, 4, 6, 8, and 10 cores respectively, the total resource consumption is calculated to be 2+4+6+8+10=30 cores. To calculate the resource usage ratio of a task, it is necessary to obtain the ratio of the resources consumed by a single task to the entire system resource pool. For example, if the total computing resources are 64 cores, the computing resource usage ratio of T1 is 4 / 64=6.25%, and the storage usage ratio is 16GB / 256GB=6.25%. Finally, all task resource consumption ratios are stored in a matrix to obtain the task resource consumption matrix.

[0047] S502: Based on the task resource consumption matrix, analyze and calculate the node load, count the node resource occupancy ratio, calculate the task adaptability, filter out tasks with a matching degree lower than a threshold, analyze the task distribution balance, filter out nodes with excessive load, adjust the task allocation strategy, optimize the task scheduling sequence, and obtain a task adaptation node list; Parsing the computing node load based on the task resource consumption matrix requires counting the resource occupancy of each node. The calculation method is the total resource consumption of all tasks on the current node divided by the maximum resource capacity of the node. For example, if the total CPU capacity of a computing node C1 is 32 cores and the total CPU demand of the currently allocated tasks is 24 cores, then the CPU occupancy ratio is 24 / 32 = 75%. To count the node resource occupancy ratio, it is necessary to calculate the storage and bandwidth occupancy ratios, and calculate the storage occupancy ratio and bandwidth occupancy ratio in the same way. Calculating the task fitness requires comparing the resource requirements of the task with the remaining available resources of the node. The calculation method is (node available resources - task requirements) / task requirements. For example, if the available CPU resources of C1 are 8 cores and the requirement of a task T2 is 6 cores, then the fitness is (8 - 6) / 6 = 33.3%. Screening tasks with a matching degree lower than the threshold requires setting a fitness threshold T. Assuming T is set to 20%, tasks with a fitness lower than 20% need to adjust the execution node. Analyzing the task distribution balance requires calculating the resource occupancy variance of all nodes. The calculation method is the standard deviation of the resource occupancy ratios of each node. Screening overloaded nodes requires setting a load threshold L. Assuming L is set to 80%, nodes with an occupancy ratio exceeding 80% are screened. Adjusting the task allocation strategy requires reallocating low-fitness tasks to nodes with lower loads. Optimizing the task scheduling order requires adjusting the allocation order of tasks according to task priorities, and finally obtaining a list of task-adapted nodes.

[0048] S503: According to the list of task-adapted nodes, adjust the resource allocation, screen scheduling conflict tasks, optimize the task execution order, update the task allocation plan, and generate an enterprise data analysis and management plan; Adjusting the resource allocation according to the list of task-adapted nodes requires reallocating computing, storage, and bandwidth resources based on task requirements. Screening scheduling conflict tasks requires checking whether multiple tasks compete for the same resources during the same time period. For example, both T3 and T4 are planned to run during the same time period and both need to use 20GB of available space on the storage node S1, while S1 currently only has 30GB remaining, then a scheduling conflict occurs. Optimizing the task execution order requires giving priority to arranging high-priority tasks for execution and separating tasks with high resource requirement overlap in time sequence. Updating the task allocation plan requires synchronizing the adjusted task execution time and resource allocation information to the task scheduling table, and finally generating an enterprise data analysis and management plan.

[0049] Please refer to Figure 7 , an enterprise data intelligent analysis and management system, including: The data flow module obtains the business data timestamp, input source, operation type, and transmission path, parses the data flow nodes, verifies the data accuracy, and generates a business data flow dependency matrix; The anomaly screening module extracts task execution nodes based on the business data flow dependency matrix, analyzes the data change range to determine the data anomaly offset, and generates an anomaly business data screening table; The task scheduling module calls the anomaly business data screening table to eliminate anomaly data, extracts task trigger factors to calculate the task execution pressure, analyzes the task dependency relationship to calculate the task load level, adjusts the task execution order according to the load level, compares the task urgency values, calculates the execution conflict rate to adjust the task scheduling, and generates a task priority ranking table; The execution management module extracts the task execution progress based on the task priority ranking table to calculate the remaining duration, compares the task urgency values, and generates a task hierarchical scheduling list; The resource optimization module extracts the task resource requirements based on the task hierarchical scheduling list to calculate the resource consumption value, verifies the accuracy of the resource consumption value, analyzes the computing node load to determine the task adaptation node, and generates an enterprise data analysis management solution.

[0050] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for intelligent analysis and management of enterprise data, characterized in that: The following steps are involved: S1: Obtain business data timestamp, input source, operation type, and transmission path, parse data flow nodes, and generate a business data flow dependency matrix; S2: Based on the business data flow dependency matrix, extract the task execution nodes, calculate the data stability, analyze the data change range to determine the data abnormality offset, and generate an abnormal business data screening table; S3: Call the abnormal business data screening table to remove abnormal data, extract task trigger factors to calculate task execution pressure, parse task dependencies to calculate task load levels, adjust task execution order according to load levels, compare task urgency values, call task execution to optimize task sorting, calculate execution conflict rate to adjust task scheduling, and generate a task priority sorting table; S4: Based on the task priority ranking table, extract the task execution progress and calculate the remaining time, compare the task urgency value, adjust the execution priority level, and generate a task hierarchical scheduling list; S5: Based on the task hierarchical scheduling list, extract the task resource requirements to calculate the resource consumption value, analyze the computing node load to determine the task adaptation node, and generate an enterprise data analysis management plan.

2. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: The business data flow dependency matrix includes data transmission links, data association weights, and business node mapping relationships; the abnormal business data screening table includes abnormal offset thresholds, abnormal impact factors, and data stability indicators; the task priority sorting table includes task load levels, task execution priorities, and task scheduling conflict coefficients; the task hierarchical scheduling list includes task execution status identification, task level classification, and task scheduling adjustment parameters; the enterprise data analysis management plan includes computing resource allocation parameters, task execution matching degree, and task migration adaptability indicators.

3. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: The specific steps to obtain the business data timestamp, input source, operation type, transmission path, parse the data flow nodes, and generate the business data flow dependency matrix are as follows: S101: obtaining the timestamp, input source, operation type and transmission path of the business data, extracting the associated attributes of the data record, sorting the business data according to the timestamp, calculating the interval between adjacent timestamps, marking the association between the data source and the operation type, and combining the transmission path record flow order to obtain the data flow sequence; S102: Based on the data flow sequence, analyze the data interaction of the flow nodes, calculate the data receiving and output of the nodes, extract the transmission path of the input source, count the number of flows of data from different sources at each node, filter the paths whose flow times exceed the set threshold, calculate the interaction frequency of the nodes, extract the main channel of data circulation, and obtain the correlation degree of data flow nodes; S103: construct a data dependency matrix according to the data flow node association, calculate the node input and output dependencies, screen key nodes with dependencies exceeding a threshold, establish a data flow network, and generate a business data flow dependency matrix.

4. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: Based on the business data flow dependency matrix, extract the task execution nodes, calculate the data stability, analyze the data change range to determine the data abnormal offset, and generate the abnormal business data screening table in the following specific steps: S201: Obtain the business data flow dependency matrix, extract task execution nodes, count the number of node data interactions, filter out nodes whose data transmission and reception volume exceeds a threshold, and obtain a task execution node list; S202: Based on the task execution node list, calculate data stability, count the fluctuation range of node data input, calculate the change rate of adjacent data points, filter out nodes whose change rate exceeds the threshold, calculate the mean and standard deviation of the change range, compare the deviation degree of data fluctuation value from the mean, filter out stable nodes, and obtain data stability coefficient; S203: According to the data stability coefficient, analyze the data variation range, calculate the data offset, filter the data with the offset exceeding the threshold, mark the abnormal data, and generate an abnormal business data filter table.

5. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: The specific steps of calling the abnormal business data screening table to remove abnormal data, extracting task trigger factors to calculate task execution pressure, parsing task dependencies to calculate task load levels, adjusting task execution order according to load levels, and comparing task urgency values, calling task execution to optimize task sorting, calculating execution conflict rates to adjust task scheduling, and generating a task priority sorting table are as follows: S301: calling the abnormal business data screening table, removing abnormal data, screening task records, extracting task triggering factors, calculating task execution pressure, and obtaining task execution pressure coefficient; S302: Based on the task execution pressure coefficient, analyze the task dependency, analyze the number of task triggering times, count the correlation between tasks, filter out tasks with dependency exceeding the threshold, identify the load level of tasks, filter out tasks with load level exceeding the limit, adjust the task execution order, calculate the task urgency value, call task execution optimization task sorting, and obtain the task execution priority; S303: Calculate the execution conflict rate according to the task execution priority, filter out tasks with conflict rates exceeding a threshold, adjust task scheduling, update task ranking, and generate a task priority ranking table.

6. The enterprise data intelligent analysis and management method according to claim 5, characterized in that: The calculation formula of the task urgency value is specifically: ； Representative tasks With the task The degree of correlation between Representative tasks In the The execution time of the trigger. Representative tasks In the The execution time of the trigger. Represents the total number of times the task is triggered. Indicates the sum of all trigger times. Representative tasks With the task In the The absolute value of the execution time difference between the triggers. Representative tasks The square root of the sum of the squares of the trigger times, Representative tasks The square root of the sum of the squares of the trigger times, represents the dependency threshold, Representative tasks CPU resource usage ratio, Representative tasks Storage resource usage ratio, Representative tasks The bandwidth resource utilization ratio is represents the load threshold, Representative tasks The remaining time, Representative tasks The urgency value of .

7. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: Based on the task priority ranking table, extract the task execution progress to calculate the remaining time, compare the task urgency value, adjust the execution priority level, and generate the task hierarchical scheduling list in the following specific steps: S401: calling the task priority sorting table, extracting the task execution progress, calculating the task remaining time, screening the tasks whose execution progress is lower than the threshold, and obtaining the task remaining time matrix; S402: Based on the task remaining time matrix, compare the task urgency values, calculate the task priority adjustment amplitude value, filter out tasks with urgency exceeding the threshold, statistically adjust the execution dependencies of the tasks, analyze the impact of task priority changes on the dependency relationship, adjust the task execution order, filter out high-dependency tasks and optimize the order, and obtain the task priority level; S403: According to the task priority level, adjust the task scheduling, assign task execution batches, screen the scheduling conflicting tasks, update the task execution order, and generate a task hierarchical scheduling list.

8. The enterprise data intelligent analysis and management method according to claim 7, characterized in that: The priority adjustment amplitude value formula is as follows: ； Representative tasks The priority adjustment value of Representative tasks and tasks The degree of dependence between Representative tasks The urgency value of Representative tasks The urgency value of Representatives on the task All dependent tasks Perform a sum operation, Representative tasks With the task The absolute value of the difference in urgency value, Representative tasks The square root of the sum of the squares of the dependencies of the dependent tasks, Represents the threshold for adjusting task priority.

9. The enterprise data intelligent analysis and management method according to claim 1, characterized in that: Based on the task hierarchical scheduling list, extract the task resource requirements, calculate the resource consumption value, analyze the computing node load to determine the task adaptation node, and generate the enterprise data analysis management plan in the following specific steps: S501: calling the task hierarchical scheduling list, extracting task resource requirements, counting computing, storage and bandwidth consumption, calculating the resource occupancy ratio of the task, and obtaining a task resource consumption matrix; S502: Based on the task resource consumption matrix, analyze and calculate the node load, count the node resource occupancy ratio, calculate the task adaptability, filter out tasks with a matching degree lower than a threshold, analyze the task distribution balance, filter out nodes with excessive load, adjust the task allocation strategy, optimize the task scheduling sequence, and obtain a task adaptation node list; S503: According to the task adaptation node list, adjust resource allocation, screen scheduling conflict tasks, optimize task execution order, update task allocation plan, and generate enterprise data analysis management plan.

10. An enterprise data intelligent analysis and management system, characterized in that: According to an enterprise data intelligent analysis management method according to any one of claims 1 to 9, the system comprises: The data flow module obtains the business data timestamp, input source, operation type, and transmission path, parses the data flow nodes, verifies the data accuracy, and generates a business data flow dependency matrix; The abnormal screening module extracts the task execution nodes based on the business data flow dependency matrix, analyzes the data change range to determine the data abnormal offset, and generates an abnormal business data screening table; The task scheduling module calls the abnormal business data screening table to remove abnormal data, extracts task trigger factors to calculate task execution pressure, analyzes task dependencies to calculate task load levels, adjusts task execution order according to load levels, compares task urgency values, calculates execution conflict rates to adjust task scheduling, and generates a task priority ranking table; The execution management module extracts the task execution progress and calculates the remaining time based on the task priority ranking table, compares the task urgency value, and generates a hierarchical task scheduling list; The resource optimization module extracts the task resource requirements based on the task hierarchical scheduling list, calculates the resource consumption value, verifies the accuracy of the resource consumption value, analyzes the computing node load to determine the task adaptation node, and generates an enterprise data analysis management plan.

Images