Intelligent data management method for data extraction

By evaluating and real-time monitoring of the historical log information of the data processing node, calculating the data conversion rate and matching degree, and setting the threshold dynamically, the accuracy and real-time problems of data extraction exception monitoring in the existing technology are solved, and efficient exceptions are achieved Status judgment and monitoring.

CN119272202AInactive Publication Date: 2025-01-07ONE ENTERPRISE SERVICE (SHENZHEN) TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411783024.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-06
Publication Date
2025-01-07
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing data extraction exception monitoring methods are prone to ignore small exceptions in macro monitoring, and real-time monitoring has a large data flow to highly integrated systems, resulting in high computing power costs and inability to meet real-time requirements.

Method used

By obtaining the historical log information of the data processing node, dividing the data packet into units, performing evaluation and analysis and real-time monitoring, calculating data conversion rate and matching degree, dynamically setting thresholds to judge abnormal states, reducing system computing pressure, and improving monitoring accuracy and Real-time.

Benefits of technology

It realizes accurate abnormal monitoring of data extraction processing, reduces system computing pressure, improves monitoring stability and real-time performance, and ensures the accuracy and flexibility of abnormal state monitoring under different load states.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119272202A_ABST
    Figure CN119272202A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent data management method for data extraction, and relates to the technical field of data management, and the method comprises the following steps: S100, obtaining all data extraction processing historical log information of each data processing node of a system, and dividing a data packet in the historical log information into data units; step S200, evaluating the data packet extraction processing operation, and analyzing predicted evaluation results of each data processing node for extracting and processing each data unit; s300, monitoring the data extraction processing information of each data processing node, and carrying out abnormality judgment on the data extraction processing operation of each data processing node; s400, data extraction processing operation of the abnormal state data processing node is traced back, the total data conversion rate of the data packet is analyzed, and abnormal data extraction processing operation is positioned; and step S500, calculating the matching degree of the data packet and each data processing node in the abnormal data extraction processing operation, and performing abnormal state feedback on each data processing node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to an intelligent data management method for data extraction. Background Art

[0002] Data extraction achieves centralized data management, supports decision-making, and improves business efficiency through operations such as data collection, organization, and analysis. Therefore, abnormal monitoring of the data extraction process is an indispensable part of the overall business of the system. In the existing data extraction anomaly monitoring, there are usually several technical solutions: first, through macro-monitoring of the entire system, analyzing system characteristics, and using clustering and other methods to determine whether anomalies occur; second, through real-time monitoring of each data extracted by the system, analyzing the characteristics of each data item, and determining whether anomalies occur in the data extraction process; However, the above technical solutions also have the following technical defects: on the one hand, by macro-monitoring the characteristic state of the overall system, the control of some nodes is relatively broad, and it is easy to ignore some small-scale data anomalies. If the abnormal warning threshold is increased, the warning signal will be triggered due to normal fluctuations; on the other hand, by monitoring the characteristics of each data in the data extraction process to judge the anomaly, in a highly integrated system, due to the large data flow, the computing cost of monitoring each data is extremely large, and in scenarios with high real-time requirements, it cannot meet the requirements; Therefore, an intelligent data management method for data extraction is needed to make up for the above technical deficiencies. Summary of the invention

[0003] The object of the present invention is to provide an intelligent data management method for data extraction to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solutions: An intelligent data management method for data extraction, the method comprising the following analysis steps: Step S100: Acquire all data extraction and processing history log information of each data processing node in the system, and divide all extracted and processed data packets in the history log information into data units; Step S200: evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information, and analyzing the estimated evaluation results of the extraction and processing of each data unit by each data processing node; Step S300: monitor the data extraction and processing information of each data processing node in real time, and make abnormal judgment on the data extraction and processing operation of each data processing node; Step S400: tracing back the data extraction and processing operations of the abnormal data processing node, calculating the total data conversion rate of the data packets in each data extraction and processing operation, and locating the abnormal data extraction and processing operation of the data processing node; Step S500: Calculate and analyze the matching degree between the data packet and each data processing node in the abnormal data extraction and processing operation of the data processing node, and perform abnormal status monitoring and feedback on each data processing node in the system according to the calculation result.

[0005] According to the above technical solution, step S100 includes the following contents: Filter the historical log information of each data processing node for data extraction and processing in all system historical logs, identify the representation function of each data, and divide the comprehensive data in the system data flow into different data units according to different representation functions; By accurately dividing the data units in each data packet, we ensure the subsequent evaluation of the expected processing results of each data processing node when performing data extraction and processing operations, and also provide a data basis for the subsequent calculation of the matching degree between the data packet and the data processing node.

[0006] According to the above technical solution, the method for evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information in step S200 is as follows: Obtaining attribute information of each data packet and manual annotation evaluation score information of each data packet extraction and processing; wherein the data packet attribute information includes the overall size of each data packet, the size of each data unit in the data packet, and the data packet extraction and processing time; For any data packet a, extract the evaluation result of the processing operation on data packet a The calculation formula is as follows: ; in, Manually annotate the evaluation score for data packet a, and are the size of data packet a and the extraction processing time respectively, and k is the evaluation coefficient of data packet a.

[0007] According to the above technical solution, the method for analyzing the estimated evaluation results of each data processing node extracting and processing each data unit in step S200 is as follows: The evaluation results of all data packets in the historical log information are classified according to the assigned data processing nodes. For any data processing node n, the calculation formula for the estimated evaluation result of each data unit extracted and processed by data processing node n is as follows: ; in, extracting the estimated evaluation result of processing data unit u for data processing node n, is the number of data packets processed by data processing node n in the historical log information, i is the number of data packets processed by data processing node n, is the size of data unit u, is the size of data packet i, Evaluate the result of the extraction processing operation for data packet i; The estimated evaluation results of each data processing node when extracting and processing different data units are calculated by weighted comprehensive calculation based on the proportion of data units, providing a reliable data basis for the positioning and judgment of subsequent abnormal data extraction and processing operations.

[0008] Calculate the estimated evaluation results of all data units extracted by data processing node n and construct the estimated evaluation result vector , denoted as , x is the number of all data units, Extract processing data units for data processing node n expected evaluation results.

[0009] According to the above technical solution, step S300 includes the following contents: The data conversion rate of each data unit extracted and processed by the data processing node of the measurement system at different load rates is calculated by weight according to the proportion of each data unit processed by the data processing node in the historical log information. The load rate-data conversion rate curve is obtained by curve fitting. The data conversion rate refers to the ratio of the amount of data output after processing by the data acquisition node to the amount of data extracted. Monitor the load rate of each data processing node in real time, and use the load rate-data conversion rate curve to match the curve to obtain the estimated data conversion rate of the data processing node , and set the conversion rate threshold , when the real-time data conversion rate of the data processing node is less than or equal to the conversion rate threshold, the data processing node is determined to be in an abnormal state; wherein, is the conversion rate deviation coefficient, k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve; Curve fitting is used to analyze the data conversion rate at different load rates of data processing nodes, and the conversion rate threshold is dynamically set to achieve adaptive comprehensive judgment on abnormal data extraction and processing operations of data processing nodes. Compared with the overall macro-monitoring of the system, the accuracy and flexibility of system data extraction and processing exceptions are improved. Compared with the precise monitoring of each data, the system computing pressure of data extraction and processing operation monitoring is reduced, the real-time and stability of the system are improved, and the universality and scientificity of the system are improved at different demand levels.

[0010] According to the above technical solution, step S400 includes the following contents: Record the time point when the abnormal state of the data processing node is detected as the abnormal time point of the data processing node, and set the abnormal backtracking window , trace back to the abnormal time point forward All data extraction and processing operations performed by the data processing node within the time period, and the data packets in each data extraction and processing operation are sorted in the order of extraction and processing time, and the total data conversion rate of each data packet extracted and processed in the current data processing node is analyzed in turn; The total conversion rate in each data packet is less than or equal to the conversion rate threshold The extraction and processing operation record is the abnormal data extraction and processing operation of the data processing node at the current abnormal time point; wherein, is the conversion rate deviation coefficient, and k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve.

[0011] According to the above technical solution, step S500 includes the following contents: For any data packet in an abnormal data extraction and processing operation, obtain the extraction and processing time point of the data packet, synchronously obtain the real-time load rate and the amount of data to be processed of each data processing node at that time point, calculate and analyze the matching degree between the data packet and each data processing node, and record all data processing nodes whose matching degree calculation result with the data packet is greater than that of the current data processing node as the data processing nodes to be checked for the data packet; The matching degree between the data packet and the current data processing node and all the data processing nodes to be checked, as well as the load rate and the amount of data to be processed of the current data processing node and all the data processing nodes to be checked at the time of data packet extraction and processing are recorded, and a data processing node troubleshooting signal is sent to the supervisor.

[0012] According to the above technical solution, the method for calculating and analyzing the matching degree between the data packet and each data processing node in step S500 is as follows: The data packet is split into data units, and the real-time load rate and the amount of data to be processed of each data processing node at the time point of data packet extraction and processing are obtained. The matching degree calculation formula for any data packet a and data processing node n is as follows: ; ; ; in, Extract the expected evaluation of processing data packet a for data processing node n, j is the data unit number, x is the number of all data units, is the size of data unit j, is the size of data packet a, extracting the estimated evaluation result of processing data unit j for data processing node n, is the load assessment factor, is the natural logarithmic exponential function, is the upper threshold of the data processing node load rate, Extract the real-time load rate of data processing node n at the processing time point for data packet a, is the matching degree between data packet a and data processing node n, is the data backlog coefficient, The amount of data waiting to be processed in the data processing node n at the time point of extracting the processing of the data packet a; the upper limit threshold of the load rate of the data processing node Refers to the load rate at which the data processing node load is in a critical congestion state.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention aims to determine whether there is an abnormality in the system data extraction and processing by real-time monitoring and analyzing the data conversion rate of the system data processing nodes, and then verify the abnormal judgment signal of each data processing node according to the matching degree calculation result between the data packet and the data processing node; wherein, by monitoring each data processing node, the system operation pressure brought by the precise monitoring of each data is avoided, and the stability and real-time performance of the overall operation of the system and abnormal monitoring are improved; in addition, the present invention also analyzes the expected data conversion rate of each data processing node at different load rates, thereby setting a dynamic conversion rate threshold for each data processing node, reducing the erroneous judgment of the abnormal state of data extraction and processing caused by the macro-monitoring of the system state, and ensuring the accuracy and flexibility of the abnormal state monitoring of each data processing node under different load states. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The figure is a flow chart of the intelligent data management method for data extraction of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Example: See Figure 1 , the present invention provides a technical solution, like Figure 1 As shown, the present invention provides an intelligent data management method for data extraction, the method comprising the following analysis steps: Step S100: Acquire all data extraction and processing history log information of each data processing node in the system, and divide all extracted and processed data packets in the history log information into data units; Step S200: evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information, and analyzing the estimated evaluation results of the extraction and processing of each data unit by each data processing node; Step S300: monitor the data extraction and processing information of each data processing node in real time, and make abnormal judgment on the data extraction and processing operation of each data processing node; Step S400: tracing back the data extraction and processing operations of the abnormal data processing node, calculating the total data conversion rate of the data packets in each data extraction and processing operation, and locating the abnormal data extraction and processing operation of the data processing node; Step S500: Calculate and analyze the matching degree between the data packet and each data processing node in the abnormal data extraction and processing operation of the data processing node, and perform abnormal status monitoring and feedback on each data processing node in the system according to the calculation result.

[0017] The step S100 includes the following contents: Filter the historical log information of each data processing node for data extraction and processing in all system historical logs, identify the representation function of each data, and divide the comprehensive data in the system data flow into different data units according to different representation functions; In specific implementation, since current systems often perform distributed processing on data to improve overall efficiency and effectiveness, the data packets formed by the integration of comprehensive data are divided into single data to facilitate subsequent detailed analysis of the data conversion rate of the data processing nodes.

[0018] The method for evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information in step S200 is as follows: Obtaining attribute information of each data packet and manual annotation evaluation score information of each data packet extraction and processing; wherein the data packet attribute information includes the overall size of each data packet, the size of each data unit in the data packet, and the data packet extraction and processing time; For any data packet a, extract the evaluation result of the processing operation on data packet a The calculation formula is as follows: ; in, Manually annotate the evaluation score for data packet a, and are the size of data packet a and the extraction processing time, respectively, and k is the evaluation coefficient of data packet a; In specific implementation, the manual annotation evaluation of data packets can be comprehensively evaluated based on parameters that measure the effectiveness of data extraction and processing operations, such as data integrity in the data packet and data comprehensiveness in data packet processing. The introduction of the evaluation coefficient aims to scale the numerical value of the ratio of data packet size to extraction and processing time to facilitate system calculation cache.

[0019] The method for analyzing the estimated evaluation results of each data processing node in step S200 for extracting and processing each data unit is as follows: The evaluation results of all data packets in the historical log information are classified according to the assigned data processing nodes. For any data processing node n, the calculation formula for the estimated evaluation result of each data unit extracted and processed by data processing node n is as follows: ; in, extracting the estimated evaluation result of processing data unit u for data processing node n, is the number of data packets processed by data processing node n in the historical log information, i is the number of data packets processed by data processing node n, is the size of data unit u, is the size of data packet i, Evaluate the result of the extraction processing operation for data packet i; Calculate the estimated evaluation results of all data units extracted by data processing node n and construct the estimated evaluation result vector , denoted as , x is the number of all data units, Extract processing data units for data processing node n the expected evaluation results; In specific implementation, due to the integrated processing of data extraction and processing, the data packets that need to be extracted and processed by different data processing nodes often contain data types with less frequent processing frequency. Therefore, it is necessary to analyze and evaluate the expected effectiveness of each data node in extracting and processing different data units to provide a data basis for the subsequent calculation of the matching degree between the data packet and the data processing node.

[0020] The step S300 includes the following contents: The data conversion rate of each data unit extracted and processed by the data processing node of the measurement system at different load rates is calculated by weight according to the proportion of each data unit processed by the data processing node in the historical log information. The load rate-data conversion rate curve is obtained by curve fitting. The data conversion rate refers to the ratio of the amount of data output after processing by the data acquisition node to the amount of data extracted. Monitor the load rate of each data processing node in real time, and use the load rate-data conversion rate curve to match the curve to obtain the estimated data conversion rate of the data processing node , and set the conversion rate threshold , when the real-time data conversion rate of the data processing node is less than or equal to the conversion rate threshold, the data processing node is determined to be in an abnormal state; wherein, is the conversion rate deviation coefficient, k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve; In the specific implementation, the historical data of data conversion rate during data extraction and processing of each data processing node is combined, and the expected data conversion rate of each data processing node is weighted and analyzed. The dynamic threshold is calculated according to the slope of the fitting curve to ensure the data interpretability of abnormal judgment of data processing nodes at different load rates.

[0021] The step S400 includes the following contents: Record the time point when the abnormal state of the data processing node is detected as the abnormal time point of the data processing node, and set the abnormal backtracking window , trace back to the abnormal time point forward All data extraction and processing operations performed by the data processing node within the time period, and the data packets in each data extraction and processing operation are sorted in the order of extraction and processing time, and the total data conversion rate of each data packet extracted and processed in the current data processing node is analyzed in turn; The total conversion rate in each data packet is less than or equal to the conversion rate threshold The extraction and processing operation record is the abnormal data extraction and processing operation of the data processing node at the current abnormal time point; wherein, is the conversion rate deviation coefficient, and k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve.

[0022] The step S500 includes the following contents: For any data packet in an abnormal data extraction and processing operation, obtain the extraction and processing time point of the data packet, synchronously obtain the real-time load rate and the amount of data to be processed of each data processing node at that time point, calculate and analyze the matching degree between the data packet and each data processing node, and record all data processing nodes whose matching degree calculation result with the data packet is greater than that of the current data processing node as the data processing nodes to be checked for the data packet; Record the matching degree between the data packet and the current data processing node and all the data processing nodes to be checked, as well as the load rate and the amount of data to be processed of the current data processing node and all the data processing nodes to be checked at the time of data packet extraction and processing, and send a data processing node troubleshooting signal to the supervisor; In specific implementation, under normal circumstances, data packets will be allocated to data processing nodes that are expected to have better processing results. However, if the load rate of the data processing node is high, the processing efficiency of the data processing node will be reduced, resulting in a backlog of data to be processed at the node. When the data packet is allocated, the priority of the data processing node will be lowered, that is, the matching calculation result will become smaller. If an abnormal situation occurs and the system process is not processed according to the above allocation method, it may happen that the data packet is not allocated to the data processing node with the highest matching degree. Therefore, each node is supervised by sorting the matching degree.

[0023] The method for calculating and analyzing the matching degree between the data packet and each data processing node in step S500 is as follows: The data packet is split into data units, and the real-time load rate and the amount of data to be processed of each data processing node at the time point of data packet extraction and processing are obtained. The matching degree calculation formula for any data packet a and data processing node n is as follows: ; ; ; in, Extract the expected evaluation of processing data packet a for data processing node n, j is the data unit number, x is the number of all data units, is the size of data unit j, is the size of data packet a, extracting the estimated evaluation result of processing data unit j for data processing node n, is the load assessment factor, is the natural logarithmic exponential function, is the upper threshold of the data processing node load rate, Extract the real-time load rate of data processing node n at the processing time point for data packet a, is the matching degree between data packet a and data processing node n, is the data backlog coefficient, The amount of data waiting to be processed in the data processing node n at the time point of extracting the processing of the data packet a; the upper limit threshold of the load rate of the data processing node Refers to the load rate at which the data processing node load is in a critical congestion state; For the above calculation formula, the following calculation example is given: assuming that the calculated value of the size of data packet a is 100, and data packet a contains 3 data units; among them, the calculated value of the size of data unit 1 is 50, and the expected evaluation result is 9, the calculated value of the size of data unit 2 is 30, and the expected evaluation result is 8, the calculated value of the size of data unit 3 is 20, and the expected evaluation result is 8, the upper limit threshold of the load rate of data processing node n is 95%, the real-time load rate of data processing node n is 30%, the amount of data to be processed by data processing node n is 1, and the data backlog coefficient is 1. It can be calculated that the evaluation expectation of data processing node n extracting and processing data packet a is 8.5, and the load evaluation coefficient in the matching formula of data packet a and data processing node is calculated to be 0.496. Therefore, it can be calculated that the matching result of data packet a and data processing node n is 2.108.

[0024] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An intelligent data management method for data extraction, characterized in that: The method comprises the following analysis steps: Step S100: Acquire all data extraction and processing history log information of each data processing node in the system, and divide all extracted and processed data packets in the history log information into data units; Step S200: evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information, and analyzing the estimated evaluation results of the extraction and processing of each data unit by each data processing node; Step S300: monitor the data extraction and processing information of each data processing node in real time, and make abnormal judgment on the data extraction and processing operation of each data processing node; Step S400: tracing back the data extraction and processing operations of the abnormal data processing node, calculating the total data conversion rate of the data packets in each data extraction and processing operation, and locating the abnormal data extraction and processing operation of the data processing node; Step S500: Calculate and analyze the matching degree between the data packet and each data processing node in the abnormal data extraction and processing operation of the data processing node, and perform abnormal status monitoring and feedback on each data processing node in the system according to the calculation result.

2. The intelligent data management method for data extraction according to claim 1, characterized in that: The step S100 includes the following contents: The historical log information of each data processing node for data extraction and processing is screened in all system historical logs, the representation function of each data is identified, and the comprehensive data in the system data flow is divided into different data units according to different representation functions.

3. The intelligent data management method for data extraction according to claim 1, characterized in that: The method for evaluating and analyzing the extraction and processing operations of all data packets in the data extraction and processing history log information in step S200 is as follows: Obtaining attribute information of each data packet and manual annotation evaluation score information of each data packet extraction and processing; wherein the data packet attribute information includes the overall size of each data packet, the size of each data unit in the data packet, and the data packet extraction and processing time; For any data packet a, extract the evaluation result of the processing operation on data packet a The calculation formula is as follows: ; in, Manually annotate the evaluation score for data packet a, and are the size of data packet a and the extraction processing time respectively, and k is the evaluation coefficient of data packet a.

4. The intelligent data management method for data extraction according to claim 1, characterized in that: The method for analyzing the estimated evaluation results of each data processing node in step S200 for extracting and processing each data unit is as follows: The evaluation results of all data packets in the historical log information are classified according to the assigned data processing nodes. For any data processing node n, the calculation formula for the estimated evaluation result of each data unit extracted and processed by data processing node n is as follows: ; in, extracting the estimated evaluation result of processing data unit u for data processing node n, is the number of data packets processed by data processing node n in the historical log information, i is the number of data packets processed by data processing node n, is the size of data unit u, is the size of data packet i, Evaluate the result of the extraction processing operation for data packet i; Calculate the estimated evaluation results of all data units extracted by data processing node n and construct the estimated evaluation result vector , denoted as , x is the number of all data units, Extract processing data units for data processing node n expected evaluation results.

5. The intelligent data management method for data extraction according to claim 1, characterized in that: The step S300 includes the following contents: The data conversion rate of each data unit extracted and processed by the data processing node of the measurement system at different load rates is calculated by weight according to the proportion of each data unit processed by the data processing node in the historical log information. The load rate-data conversion rate curve is obtained by curve fitting. The data conversion rate refers to the ratio of the amount of data output after processing by the data acquisition node to the amount of data extracted. Monitor the load rate of each data processing node in real time, and use the load rate-data conversion rate curve to match the curve to obtain the estimated data conversion rate of the data processing node , and set the conversion rate threshold , when the real-time data conversion rate of the data processing node is less than or equal to the conversion rate threshold, the data processing node is determined to be in an abnormal state; wherein, is the conversion rate deviation coefficient, and k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve.

6. The intelligent data management method for data extraction according to claim 5, characterized in that: The step S400 includes the following contents: Record the time point when the abnormal state of the data processing node is detected as the abnormal time point of the data processing node, and set the abnormal backtracking window , trace back to the abnormal time point forward All data extraction and processing operations performed by the data processing node within the time period, and the data packets in each data extraction and processing operation are sorted in the order of extraction and processing time, and the total data conversion rate of each data packet extracted and processed in the current data processing node is analyzed in turn; The total conversion rate in each data packet is less than or equal to the conversion rate threshold The extraction and processing operation record is the abnormal data extraction and processing operation of the data processing node at the current abnormal time point; wherein, is the conversion rate deviation coefficient, and k is the slope of the corresponding point of the real-time load rate of the data processing node in the load rate-data conversion rate curve.

7. The intelligent data management method for data extraction according to claim 6, characterized in that: The step S500 includes the following contents: For any data packet in an abnormal data extraction and processing operation, obtain the extraction and processing time point of the data packet, synchronously obtain the real-time load rate and the amount of data to be processed of each data processing node at that time point, calculate and analyze the matching degree between the data packet and each data processing node, and record all data processing nodes whose matching degree calculation result with the data packet is greater than that of the current data processing node as the data processing nodes to be checked for the data packet; The matching degree between the data packet and the current data processing node and all the data processing nodes to be checked, as well as the load rate and the amount of data to be processed of the current data processing node and all the data processing nodes to be checked at the time of data packet extraction and processing are recorded, and a data processing node troubleshooting signal is sent to the supervisor.

8. The intelligent data management method for data extraction according to claim 6, characterized in that: The method for calculating and analyzing the matching degree between the data packet and each data processing node in step S500 is as follows: The data packet is split into data units, and the real-time load rate and the amount of data to be processed of each data processing node at the time point of data packet extraction and processing are obtained. The matching degree calculation formula for any data packet a and data processing node n is as follows: ; ; ; in, Extract the expected evaluation of processing data packet a for data processing node n, j is the data unit number, x is the number of all data units, is the size of data unit j, is the size of data packet a, extracting the estimated evaluation result of processing data unit j for data processing node n, is the load assessment factor, is the natural logarithmic exponential function, is the upper threshold of the data processing node load rate, Extract the real-time load rate of data processing node n at the processing time point for data packet a, is the matching degree between data packet a and data processing node n, is the data backlog coefficient, The amount of data waiting to be processed in the data processing node n at the time point of extracting the processing of the data packet a; the upper limit threshold of the load rate of the data processing node Refers to the load rate at which the data processing node load is in a critical congestion state.

Citation Information

Patent Citations

  • Power consumption information acquisition system acquisition quality analyzing method and device thereof

    CN109785606A

  • Intelligent early warning and disposal method based on historical monitoring data

    CN117827608A

  • Self-adaptive production scheduling optimization method based on industrial digital twinning

    CN118132996A

  • Engineering data management system and method based on digital construction

    CN118195326A

  • Short-term operation optimization method of electric power system including large-scale wind power

    US20160169202A1