Network monitoring data quality evaluation method, device, equipment and medium

Abstract

The invention relates to a network monitoring data quality evaluation method, device and equipment and a medium. The method comprises the following steps: based on an evaluation period and a key data list, extracting records generated during equipment operation at corresponding time from a source database to obtain an original data snapshot; performing basic quality assessment about validity and completeness on the original data snapshot to obtain a data quality basic assessment report; based on the data quality basic evaluation report, quantifying the quality of the data into corresponding scores, and obtaining a data quality quantitative score table; inputting the data quality basic evaluation report into a root cause diagnosis model to obtain an intelligent diagnosis report; generating a comprehensive evaluation report based on the data quality quantitative score table and the intelligent diagnosis report; the comprehensive evaluation report is used for assisting engineers to diagnose and troubleshoot the equipment. By adopting the method, data missing can be effectively identified, data relevance abnormity can be found, and data quality can be accurately evaluated.

Description

Technical Field

[0001] This invention belongs to the field of data quality assessment, and in particular relates to methods, devices, equipment and media for assessing the data quality of wired network monitoring. Background Technology

[0002] In silicon wafer dicing, multi-wire dicing has become the mainstream process. This technology relies on multiple diamond wires simultaneously performing high-speed, precision dicing on a silicon rod. The process involves complex mechanical tension, slurry flow, and vibration changes, leading to the development of advanced methods that utilize dense sensor networks for real-time monitoring and data analysis of the dicing process. Traditional techniques handle wire network monitoring data in a relatively crude manner. When analyzing critical anomalies such as wire skips or breaks to optimize the process, data is directly collected using data acquisition devices. The raw data is then aggregated and statistically analyzed, and fixed thresholds are set based on experience to determine anomalies, generating probability ranking charts for various dimensions. This traditional method has significant limitations. It cannot effectively identify data loss and numerical drift caused by sensor malfunctions, network interruptions, or signal interference; it cannot distinguish between individual equipment differences and actual process anomalies; and it struggles to detect complex correlations between different parameters. Ranking charts generated from such data may be severely distorted, preventing production managers from accurately pinpointing the root cause of problems and potentially leading to incorrect maintenance decisions or process adjustments, thus hindering further improvements in dicing yield and equipment efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, device, equipment, and medium for assessing the quality of network monitoring data that can effectively identify missing data and numerical drift, and detect abnormal correlations, in order to address the aforementioned technical problems.

[0004] Firstly, this application provides a method for evaluating the quality of network monitoring data, including:

[0005] Based on the evaluation cycle and key data list, records generated during the corresponding time period of equipment operation are extracted from the source database to obtain the original data snapshot; the key data list represents the data required to generate business charts;

[0006] A basic quality assessment of the validity and completeness of the original data snapshot is conducted to obtain a basic data quality assessment report;

[0007] Based on the basic data quality assessment report, the quality of the data is quantified into corresponding scores to obtain a data quality quantification scoring table.

[0008] Input the basic data quality assessment report into the root cause diagnosis model to obtain an intelligent diagnosis report;

[0009] Based on the data quality quantitative scoring table and intelligent diagnostic report, a comprehensive evaluation report is generated; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting equipment.

[0010] Furthermore, a basic quality assessment regarding the validity and completeness of the original data snapshot is conducted, resulting in a basic data quality assessment report, including:

[0011] Traverse the records in the original data snapshot, count the number of records for each device within the evaluation period, and obtain a time completeness report;

[0012] Iterate through the records in the original data snapshot, count the records with missing key fields, and obtain a report of missing key information;

[0013] Based on the original data snapshot, the records are verified for value domain compliance to obtain a value domain violation report;

[0014] The code is compared with the corresponding authoritative data list in the original data snapshot. Codes that are not in the authoritative data list are identified, and an external correlation anomaly report is obtained. The authoritative data list includes valid operators, wire batch numbers and groove wheel numbers.

[0015] Business logic verification is performed on the original data snapshot to obtain a logic conflict report; and the time completeness report, key information missing report, value range violation report, external correlation anomaly report and logic conflict report are integrated to obtain a basic data quality assessment report.

[0016] Furthermore, based on the original data snapshot, value range compliance is verified on the records to obtain a value range violation report, including:

[0017] Based on predefined static rules, records in the original data snapshot that violate the static rules are marked to obtain a static rule violation list; static rules include at least one of the following: a valid dictionary, format matching, and absolute range.

[0018] Based on the historical data of each device, the dynamic baseline of each operating parameter included in the record is calculated using the following formula:

[0019] UCL=μ+kσ

[0020] UCL=μ-kσ

[0021] Where UCL is the upper control limit, LCL is the lower control limit, μ is the mean, σ is the standard deviation, and k is the control limit multiplier factor;

[0022] Extract operational parameters that are not in the static rule violation list; operational parameters characterize the operating status of the device.

[0023] Compare the operating parameters with the corresponding dynamic baseline; mark the operating parameters that are not within the dynamic baseline as abnormal and generate a dynamic baseline abnormality report;

[0024] By integrating the static rule violation list and the dynamic baseline anomaly report, a value range violation report is obtained.

[0025] Furthermore, based on the basic data quality assessment report, the data quality is quantified into corresponding scores, resulting in a data quality quantification scoring table, including:

[0026] Based on the evaluation objectives, core quality dimensions are determined; and weights are assigned to these core quality dimensions to obtain a data quality dimension weight table; the core quality dimensions include completeness and effectiveness.

[0027] Based on the data quality dimension weight table and the basic data quality assessment report, the quantitative scores of each dimension are calculated to obtain the initial scores of each dimension;

[0028] Based on the data quality dimension weight table, the initial scores of each dimension are weighted and summed to obtain the comprehensive data quality index;

[0029] The comprehensive data quality index and each initial score are mapped to the corresponding level to obtain the data quality quantitative scoring table.

[0030] Furthermore, based on the data quality dimension weight table and the basic data quality assessment report, the quantitative scores for each dimension are calculated to obtain the initial scores for each dimension, including:

[0031] Based on the basic data quality assessment report, the completeness dimension score in the data quality dimension weight table is calculated using the following formula:

[0032]

[0033] Among them, S c For the completeness dimension score, N missing N represents the total number of missing data records. empty T represents the number of records with missing key fields. records F represents the total number of records in the static snapshot of the original data. critical T represents the number of key fields included in each record. devices T represents the total number of devices. intervals To record the number of intervals;

[0034] Based on the basic data quality assessment report, the validity dimension score in the data quality dimension weight table is calculated using the following formula:

[0035]

[0036] Among them, S vFor the validity dimension score, N invalid N represents the number of invalid value records. orphan N represents the number of externally associated abnormal records. logic T represents the number of business logic conflict records. records F represents the total number of records in the static snapshot of the original data. validated The number of fields in each record that need to be validated;

[0037] The scores for completeness and effectiveness were used as the initial scores.

[0038] Furthermore, based on the data quality quantitative scoring table and intelligent diagnostic report, a comprehensive evaluation report is generated, including:

[0039] Based on the data quality quantitative scoring table and intelligent diagnostic report, the overall data quality is rated to obtain the rating result; and based on the rating result, data usage suggestions are generated to obtain the data usage suggestion letter.

[0040] Based on the data usage proposal and the original data snapshot, generate business charts; the business charts include at least one of the following: personnel jump line probability chart, equipment jump line probability chart, steel wire jump line probability chart, grooved wheel jump line probability chart, and grooved wheel position jump line probability chart;

[0041] Based on the intelligent diagnostic report, corresponding improvement measures are matched; and the improvement measures are prioritized to obtain a data quality improvement priority list.

[0042] By integrating the data quality improvement priority list, business charts, and data usage recommendations, a comprehensive evaluation report is obtained.

[0043] Furthermore, based on the data usage proposal and the original data snapshot, business charts are generated, including:

[0044] Based on the rating results in the data usage recommendation, the rigor of data cleaning is determined, and a strategy specification is obtained.

[0045] Based on the strategy specification, data cleaning was performed on the problem records in the original data snapshot to obtain a high-quality dataset.

[0046] Based on the dimensions of the business charts, the data in the high-quality dataset is grouped to obtain data groups; and the jump probability is calculated for each data group to obtain a summary table of business indicators.

[0047] Generate basic business charts based on the business indicator summary table; and add explanatory text to the basic business charts to obtain the business charts.

[0048] Secondly, this application also provides a network monitoring data quality assessment device, comprising:

[0049] The snapshot module is used to extract records generated during the equipment operation period at the corresponding time from the source database based on the evaluation cycle and key data list to obtain the original data snapshot; the key data list represents the data required to generate business charts;

[0050] The basic module is used to perform a basic quality assessment on the validity and completeness of the original data snapshot, and to obtain a basic data quality assessment report;

[0051] The quantification module is used to quantify the quality of data into corresponding scores based on the basic data quality assessment report, and obtain a data quality quantification scoring table.

[0052] The root cause module is used to input the basic data quality assessment report into the root cause diagnosis model to obtain an intelligent diagnosis report.

[0053] The reporting module is used to generate a comprehensive evaluation report based on the data quality quantitative scoring table and intelligent diagnostic report; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting equipment.

[0054] Thirdly, this application also provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any step of the method provided in the first aspect of this application.

[0055] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any step of the method provided in the first aspect of this application.

[0056] The aforementioned network monitoring data quality assessment method, device, equipment, and media, based on the assessment period and key data list, extract records generated during equipment operation at the corresponding time from the source database to obtain a raw data snapshot; the key data list represents the data required to generate business charts; a basic quality assessment of the validity and completeness of the raw data snapshot is performed to obtain a basic data quality assessment report; based on the basic data quality assessment report, the data quality is quantified into corresponding scores to obtain a data quality quantification scoring table; the basic data quality assessment report is input into a root cause diagnosis model to obtain an intelligent diagnosis report; based on the data quality quantification scoring table and the intelligent diagnosis report, a comprehensive assessment report is generated; the comprehensive assessment report is used to assist engineers in diagnosing and troubleshooting equipment. It can effectively identify missing data, trace the root causes of data quality problems, and generate a data quality report that can assist managers in making maintenance decisions or process adjustments. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of the process of a network monitoring data quality assessment method provided in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the structure of a wired network monitoring data quality assessment device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In one embodiment, such as Figure 1 As shown, a method for assessing the quality of network monitoring data is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0062] Step 101: Based on the evaluation period and key data list, extract the records generated during the corresponding time period of equipment operation from the source database to obtain the original data snapshot; the key data list represents the data required to generate business charts.

[0063] Specifically, the evaluation period is a predefined time range that specifies the start and end times for data extraction from the source database. It defines the time window for data extraction, ensuring the evaluation targets equipment operation data within a specific time period. The key data list is a list of data fields or data types necessary for generating business charts. Examples include equipment number, timestamp, operating parameters, operator information, and wire batch number, defining business requirements and ensuring the extracted data covers the core elements for chart generation. The source database is a database system that stores raw equipment operation data; it is a historical database containing various records generated during equipment operation, such as sensor readings and event logs. The raw data snapshot is a collection of equipment operation data extracted from the source database within the evaluation period. It is a raw, unprocessed dataset containing all field records specified in the key data list, serving as the basis for subsequent quality assessments. The terminal performs data queries or extractions from the source database based on the time range determined by the evaluation period. Using a key data list as a filter, it extracts data fields relevant to business charts, avoiding interference from irrelevant data. It calls the database query language, with the time condition corresponding to the start and end dates of the evaluation period. Field selection is based on the key data list. The extraction process ensures data integrity, without any cleaning or transformation, preserving the original state. The terminal parses the evaluation period and key data list, constructs the query statement, and after query execution, the data is extracted to temporary storage or memory, forming a snapshot of the original data. This process is automated, triggered periodically or by receiving trigger conditions, ensuring timely data updates.

[0064] Step 102: Conduct a basic quality assessment on the validity and completeness of the original data snapshot to obtain a basic data quality assessment report.

[0065] In this context, validity refers to whether the data conforms to predefined rules, logic, and scope, ensuring that data values ​​are reasonable and reliable. Completeness refers to whether the data is complete and intact, including the continuity of the time series and the fullness of key fields. Basic quality assessment is a multifaceted inspection process that evaluates the validity and completeness of raw data snapshots. The basic data quality assessment report is a comprehensive report summarizing the results of the basic quality assessment and includes multiple sub-reports. The report presents issue details in a structured format; for example, it may list the number of abnormal records, a list of affected devices, etc. The terminal system calculates the number of records each device should have and the actual number of records within the assessment period, calculates the missing rate, generates a time completeness report, scans key fields, counts the number of records with missing values, generates a key information missing report, marks invalid value records based on static rules and dynamic baselines, generates a violation list by applying static rules, calculates dynamic baselines for non-violation data, compares operating parameters, generates a dynamic baseline anomaly report, and finally integrates them into a value range violation report. It also compares the codes in the data with the valid code library, identifies non-existent codes, generates an external association anomaly report, checks whether records violate business rules, generates a logical conflict report, and integrates all sub-reports into a unified data quality basic assessment report.

[0066] Step 103: Based on the basic data quality assessment report, quantify the quality of the data into corresponding scores to obtain the data quality quantification scoring table.

[0067] Quantification involves converting quality assessment results into numerical scores, making quality measurable and comparable. This process involves mathematical calculations and weight allocation. The data quality quantification scoring table is a table containing scores for each quality dimension and a comprehensive data quality index, providing a concise numerical representation for rapid evaluation. The terminal determines core quality dimensions based on the assessment objectives and assigns weights to each dimension. Relevant indicators are extracted from the basic data quality assessment report, and initial scores are calculated for each dimension. A completeness score is calculated using formulas based on parameters such as the number of missing records, the number of missing records in key fields, and the total number of records. The formulas emphasize data coverage and completeness; higher scores indicate better completeness. A validity score is calculated using formulas based on the number of invalid value records, the number of records with external correlation anomalies, and the number of records with logical conflicts. This formula focuses on data correctness; higher scores indicate better validity. The initial scores for each dimension are weighted and summed to obtain the comprehensive data quality index. The scores are then mapped to predefined levels to generate the data quality quantification scoring table. This rule-driven approach ensures objectivity and consistency.

[0068] Step 104: Input the basic data quality assessment report into the root cause diagnosis model to obtain the intelligent diagnosis report.

[0069] Specifically, a root cause diagnostic model is an analytical model, possibly based on machine learning, statistical analysis, or rule engines, used to identify the root causes of data quality problems. The model analyzes problem patterns and infers underlying causes, such as equipment failure, configuration errors, or human error. The intelligent diagnostic report is a summary of the root cause analysis results, including identified primary causes, scope of impact, potential risks, and suggested investigation directions, presented in a structured format to provide intelligent insights. The endpoint takes the basic data quality assessment report as input and passes it to the root cause diagnostic model. The model parses the problem details in the report, including the distribution of abnormal records and temporal patterns, and performs root cause analysis based on historical data. For example, if multiple devices experience data loss, the model might identify network outages or sensor malfunctions; abnormally concentrated value ranges might point to calibration issues. The analysis process uses clustering, classification, or association rule algorithms, and the model performs automated diagnostics, generating an intelligent diagnostic report that includes a list of root causes, confidence scores, and relevant evidence, reducing the burden of manual analysis.

[0070] Step 105: Based on the data quality quantitative scoring table and intelligent diagnostic report, generate a comprehensive evaluation report; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting equipment.

[0071] Specifically, the comprehensive assessment report is a complete document that integrates data quality scores, diagnostic results, visualizations, and improvement suggestions. It provides engineers with a one-stop reference to assist in equipment diagnosis and decision-making. The terminal rates the overall data quality based on the data quality quantitative scoring table and the intelligent diagnostic report. Combined with original data snapshots, it determines the rigor of data cleaning, cleans problematic records, calculates jumper probabilities by business dimension, generates charts, matches improvement measures from the intelligent diagnostic report, prioritizes them based on the degree of problem impact, and integrates all elements into the comprehensive assessment report.

[0072] The network monitoring data quality assessment method provided in this embodiment, based on the assessment period and a key data list, extracts records generated during equipment operation at the corresponding time from the source database to obtain a raw data snapshot; the key data list represents the data required to generate business charts; a basic quality assessment of the validity and completeness of the raw data snapshot is performed to obtain a basic data quality assessment report; based on the basic data quality assessment report, the data quality is quantified into corresponding scores to obtain a data quality quantification scoring table; the basic data quality assessment report is input into a root cause diagnosis model to obtain an intelligent diagnosis report; based on the data quality quantification scoring table and the intelligent diagnosis report, a comprehensive assessment report is generated; the comprehensive assessment report is used to assist engineers in diagnosing and troubleshooting equipment. Through the above steps, it is possible to effectively identify data missing and invalid issues, trace the root causes of data quality problems, and generate a data quality report that can assist managers in making maintenance decisions or process adjustments.

[0073] In one embodiment, a basic quality assessment regarding the validity and completeness of the original data snapshot is performed to obtain a basic data quality assessment report, including:

[0074] Step 201: Traverse the records in the original data snapshot, count the number of records for each device during the evaluation period, and obtain the time completeness report.

[0075] The record count refers to the number of data entries actually present in the original data snapshots for a specific device within the evaluation period. The time completeness report is a report that primarily compares the expected number of records for each device within the evaluation period with the actual number of records present, revealing data gaps in the time dimension. Based on the evaluation period and preset data collection intervals, the terminal calculates the total number of records each device should theoretically generate. By traversing the original data snapshots, grouping by device, and actually counting the number of records for each device, the actual number is compared with the theoretical number to calculate the record missing rate, generating the time completeness report. For example, if the evaluation period is 1 hour and the collection frequency is once per minute, then one device should have 60 records. If the count shows only 55 records, it indicates that data is missing at 5 time points.

[0076] Step 202: Traverse the records in the original data snapshot, count the missing records of key fields, and obtain a key information missing report.

[0077] Specifically, key fields are those used for subsequent business analysis and chart generation. A missing record refers to a record where one or more key fields have empty values. The Key Information Missing Report is a report that primarily summarizes how many records in the original data snapshot have missing key fields, listing which fields are missing and identifying the missing records. The terminal then iterates through each record in the original data snapshot, focusing on the fields within each record. For each record, it checks every field in the predefined key field list to determine if its value is valid. If any key field value is found to be empty, the record is marked as a key information missing record. All missing records are then summarized and statistically analyzed.

[0078] Step 203: Based on the original data snapshot, perform value domain compliance verification on the records to obtain a value domain violation report.

[0079] Specifically, value range compliance verification refers to checking whether the value or content of a specific field in a data record conforms to a pre-defined reasonable range, format, or list of values. A value range violation report is a report that lists all data records found to be non-compliant during value range compliance verification, indicating which field in which record violated the rules, and the specific details of the violation. The terminal iterates through the original data snapshots, comparing the actual values ​​of the fields requiring verification with predefined compliance standards. These standards can be simple numerical ranges, format requirements, or valid lists of values. Any record containing a value that violates the rules is marked as a violation.

[0080] Step 204: Compare all codes in the original data snapshot with the corresponding authoritative data list, identify codes that are not in the authoritative data list, and obtain an external association anomaly report; the authoritative data list includes valid operators, wire batch numbers, and groove wheel numbers.

[0081] In this context, "encoding" refers to the identifier fields in the original data snapshot used to associate information about external entities. In this embodiment, it specifically refers to the values ​​of fields such as operator, wire batch number, and grooved wheel number. The authoritative data list is an independently maintained list considered the sole source of truth, containing all currently valid encodings. For example, it may include a list of valid operators, a list of registered wire batch numbers, and a list of currently used grooved wheel numbers. The external association anomaly report is a report listing encodings found in the original data snapshot that are not present in the authoritative data list, indicating that the external entity referred to by the encoding is invalid or unauthorized. The terminal extracts all values ​​of the specified encoding fields from the original data snapshot and compares these values ​​with the corresponding authoritative data list. If a encoding found in the original data cannot be found in the authoritative list, then this encoding is considered an invalid association, and the record containing it is marked as an anomaly.

[0082] Step 205: Perform business logic verification on the original data snapshot to obtain a logic conflict report; and integrate the time completeness report, key information missing report, value range violation report, external correlation anomaly report, and logic conflict report to obtain a basic data quality assessment report.

[0083] Business logic verification refers to checking whether the values ​​of multiple fields between or within data records conform to specific business rules and logical relationships, focusing on logical consistency across fields or records. The logic conflict report lists all data records that violate established business logic. The data quality baseline assessment report is a comprehensive report that integrates core findings and summaries of time completeness, missing key information, value range violations, external correlation anomalies, and logical conflicts, providing a comprehensive, multi-faceted assessment baseline for the overall quality of the original data snapshot. The terminal traverses and checks data based on predefined business rules, identifies records that violate these rules, and combines the contents of the five generated sub-reports for summarization and consolidation. The integrated report clearly shows the specific types of problems existing in the data in terms of completeness and validity, as well as the severity of each problem. For example, business logic verification may include ensuring that jumpers occur between the start and end of device operation.

[0084] This embodiment generates a basic data quality assessment report, which efficiently and comprehensively evaluates the validity and completeness of the data from multiple dimensions, providing detailed evidence for subsequent quantitative scoring of data quality and improving the accuracy of data quality assessment.

[0085] In one embodiment, based on the original data snapshot, value range compliance verification is performed on the records to obtain a value range violation report, including:

[0086] Step 301: Based on predefined static rules, mark the records in the original data snapshot that violate the static rules to obtain a static rule violation list; the static rules include at least one of the following: a valid dictionary, format matching, and absolute range.

[0087] Among them, predefined static rules are a set of fixed data validation standards set before evaluation, which do not change with the data itself or time, and are hard rules for judging the validity of data values. A valid dictionary is a predefined list containing all acceptable values. For example, for the device status field, the valid dictionary only includes three values: running, standby, and faulty; any value not in this list is a violation. Format matching refers to the string format requirements for data values. Absolute range refers to the minimum and maximum values ​​set for a numeric field that are physically or business-wise not allowed to be exceeded. Marking refers to the action of identifying or separately recording data records found to violate rules during data processing. A static rule violation list is a list that records all data records found to have violated any static rule in the original data snapshot. The list indicates which record, which field, and which specific rule was violated. The terminal application performs the first round of data filtering using fixed rules, removing obviously unreasonable or incorrectly formatted values. It then iterates through each record in the original data snapshot, comparing the actual values ​​of the fields to be checked with the predefined static rules. If any field value is found to be inconsistent with the rules, the record is immediately marked and its detailed information is added to the static rule violation list.

[0088] Step 302: Based on the historical data of each device, calculate the dynamic baseline of each operating parameter included in the record using the following formula:

[0089] UCL=μ+kσ

[0090] LCL=μ-kσ

[0091] Where UCL is the upper control limit, LCL is the lower control limit, μ is the mean, σ is the standard deviation, and k is the control limit multiplier.

[0092] Specifically, historical equipment data refers to a large amount of historical operating parameter data generated by the equipment during normal operation before the current evaluation period. This data is used to establish a behavioral benchmark for the equipment under normal conditions. Operating parameters are numerical indicators that directly characterize the operating status of the equipment. The dynamic baseline is a normal fluctuation range calculated individually for each operating parameter of each piece of equipment. It is dynamic, calculated based on the equipment's own historical data, and reflects the normal operating characteristics of the equipment in a personalized way, rather than a uniform fixed standard. The mean is the average value of a certain operating parameter calculated based on the equipment's historical data, representing the central location of the parameter's normal operation. The standard deviation is the degree of dispersion of a certain operating parameter calculated based on the equipment's historical data. The larger the standard deviation, the greater the historical volatility of the parameter. The control limit multiplier is a configurable multiplier used to adjust the looseness of the control limits. The upper and lower control limits are the upper and lower limits calculated by the formula, which together constitute the dynamic baseline. For a certain operating parameter, its normal value is expected to fall within the [LCL, UCL] interval. For each device, the terminal selects its specific historical dataset, calculates the average and standard deviation of each operating parameter that needs to be monitored in the historical data, and uses a given formula combined with a preset k value to calculate the upper and lower control limits of the parameter, thus obtaining the dynamic baseline of the parameter for that device.

[0093] Step 303: Extract the operating parameters that are not in the static rule violation list; the operating parameters characterize the operating status of the device.

[0094] Specifically, runtime parameters not listed in the static rule violation list refer to runtime parameter values ​​whose corresponding records have passed the static rule check and have not been marked as violations. These parameter values ​​are considered superficially compliant and eligible for more granular dynamic analysis. The terminal excludes all data records already listed in the static rule violation list from the original data snapshot, extracting the runtime parameter values ​​contained in the remaining records. This ensures that subsequent dynamic baseline analysis is based on clean data, avoiding interference from obviously erroneous data or leading to misjudgments.

[0095] Step 304: Compare the operating parameters with the corresponding dynamic baseline; mark the operating parameters that are not within the dynamic baseline as abnormal and generate a dynamic baseline abnormality report.

[0096] "Not within the dynamic baseline" refers to an actual value of an operating parameter that is lower than its corresponding lower control limit or higher than its upper control limit. The dynamic baseline anomaly report is a list recording all data points marked as abnormal, including device information, parameter name, abnormal value, and the corresponding dynamic baseline range. The terminal iterates through each extracted operating parameter value. For each value, it identifies the corresponding device and parameter type, retrieves the dynamic baseline calculated for that device and parameter type, and determines whether the actual value falls within the dynamic baseline range. If it is not within the range, the data point is marked as abnormal, and detailed information is recorded in the dynamic baseline anomaly report.

[0097] Step 305: Integrate the static rule violation list and the dynamic baseline anomaly report to obtain the value range violation report.

[0098] The value domain violation report is a comprehensive report that includes all problematic data discovered during value domain compliance verification, encompassing both violations of hard static rules and violations of personalized dynamic baselines. The terminal integrates the static rule violation list and dynamic baseline anomaly reports according to a specific structure to ultimately form a comprehensive value domain violation report.

[0099] This embodiment uses a two-step approach—static rules and dynamic baselines—to determine parameter value violations. It comprehensively reflects all issues related to the numerical reasonableness of the original data snapshot, balancing the rigidity of the rules with the dynamism of history. This provides crucial evidence of validity for overall data quality assessment, thereby improving the accuracy of data evaluation.

[0100] In one embodiment, based on the basic data quality assessment report, the quality of the data is quantified into a corresponding score to obtain a data quality quantification scoring table, including:

[0101] Step 401: Based on the evaluation objectives, determine the core quality dimensions; and assign weights to the core quality dimensions to obtain the data quality dimension weight table; the core quality dimensions include completeness and effectiveness.

[0102] The evaluation objective refers to the specific business purpose that this data quality evaluation aims to achieve, determining the focus of the evaluation. Core quality dimensions refer to different aspects or perspectives for measuring data quality. In this embodiment, the core dimensions of this evaluation are clearly stated as completeness and effectiveness, the two most fundamental dimensions of data quality. Weight allocation assigns an importance coefficient to each core quality dimension; the weight indicates the relative importance of that dimension to the overall data quality. The data quality dimension weight table is a simple mapping table that clearly lists each determined core quality dimension and its corresponding weight allocation. For example, if the goal is to generate accurate production reports, then data effectiveness is more important than completeness, therefore, effectiveness will be assigned a higher weight. Based on this analysis, completeness and effectiveness are formally determined as the core dimensions of this evaluation. Based on the importance of the goal, a weight value is assigned to each of the two dimensions, and the correspondence between the dimension weights is recorded in the data quality dimension weight table.

[0103] Step 402: Based on the data quality dimension weight table and the data quality basic assessment report, calculate the quantitative score of each dimension to obtain the initial score of each dimension.

[0104] Specifically, the data quality baseline assessment report details the specific issues found in the completeness and validity dimensions. The quantitative score for each dimension is a single score calculated for each core quality dimension, representing the data's quality level in that dimension. The initial scores are raw scores obtained after independent calculation for each dimension, without weighted adjustment. The terminal processes each core quality dimension separately. For the completeness dimension, relevant indicators are extracted from the data quality baseline assessment report, such as the total number of records, the number of missing records, and the number of missing key fields, and then substituted into a calculation formula specifically designed for completeness to calculate a score between 0 and 100; a higher score indicates better completeness. For the validity dimension, indicators such as the number of invalid value records and the number of logically conflicting records are extracted and substituted into their specific calculation formula to obtain an independent validity score. The calculation process strictly relies on the original evidence provided in the baseline assessment report.

[0105] Step 403: Based on the data quality dimension weight table, the initial scores of each dimension are weighted and summed to obtain the comprehensive data quality index.

[0106] Specifically, weighted summation is a mathematical method used to combine multiple values ​​into a single overall value. It involves multiplying the initial score of each dimension by its corresponding weight and then summing all the products. The overall data quality index is a single numerical value representing the overall data quality level after considering the importance of each dimension. The terminal reads the weight of each dimension from the data quality dimension weight table, takes the calculated initial score for each dimension, multiplies each initial score by its corresponding weight, and then sums them to obtain the overall data quality index. This ensures that dimensions with higher weights have a greater impact on the final total score.

[0107] Step 404: Map the comprehensive data quality index and each initial score to the corresponding level to obtain the data quality quantitative scoring table.

[0108] Mapping involves categorizing numerical scores into specific levels or categories according to a set of preset rules. Levels are predefined labels representing quality levels, more intuitive and easier to understand than numbers. The data quality quantification scoring table is a structured report or table that includes not only initial scores for each dimension and the overall data quality index, but also corresponding level labels. The terminal accesses the preset level mapping rules, compares the overall data quality index and the initial scores for completeness and validity against the rules, determines the level to which each score belongs, and integrates all information, including score values ​​and corresponding level labels, to generate the final data quality quantification scoring table.

[0109] This embodiment introduces weights to establish evaluation criteria and focuses for the entire quantification process, making the quantification score not a simple average, but a focused assessment that reflects business needs. It transforms qualitative assessment reports into quantitative scores, making data quality measurable and comparable, and providing a foundation for subsequent data quality assessments.

[0110] In one embodiment, based on the data quality dimension weight table and the data quality basic assessment report, a quantitative score for each dimension is calculated to obtain an initial score for each dimension, including:

[0111] Step 501: Based on the basic data quality assessment report, calculate the completeness dimension score in the data quality dimension weight table using the following formula:

[0112]

[0113] Among them, S c For the completeness dimension score, N missing N represents the total number of missing data records. empty T represents the number of records with missing key fields. records F represents the total number of records in the static snapshot of the original data. criticalT represents the number of key fields included in each record. devices T represents the total number of devices. intervals This is for recording the number of intervals.

[0114] Specifically, the Data Quality Foundation Assessment Report is a comprehensive report generated from the assessment process. It details specific issues related to data completeness and validity, including statistical information such as missing records, empty fields, and value range violations. It serves as the foundational input for this calculation. The completeness dimension score is a numerical score calculated using a formula, ranging from 0 to 100. It quantifies the completeness quality of the data; a higher score indicates more complete data and fewer missing records. The total number of missing data records refers to the total number of records that should have been generated by the device but did not actually appear in the original data snapshot during the assessment period, reflecting the data loss over time. The number of records with missing key fields refers to the number of records in the original data snapshot where the record itself exists, but one or more key fields have empty or invalid values, reflecting the lack of critical information within the record. The total number of records in the original static data snapshot refers to the total number of all data records extracted from the source database during the assessment period, representing the actual amount of data acquired. The number of key fields included in each record is a predefined constant, representing the number of fields considered key in each data record. Key fields are essential for business operations. The total number of devices refers to the total number of devices monitored within the assessment scope. The number of recording intervals refers to the total number of time intervals during which devices should theoretically generate records based on the preset data collection frequency within the assessment period. The terminal extracts relevant statistics from the basic data quality assessment report and substitutes these values ​​into a given mathematical formula for calculation. The numerator of the formula represents the sum of all completeness issues, including overall record missingness and missing key fields; the denominator is a normalization factor designed to measure the total chance or maximum probable problem point of data completeness.

[0115] Step 502: Based on the basic data quality assessment report, calculate the validity dimension score in the data quality dimension weight table using the following formula:

[0116]

[0117] Among them, S v For the validity dimension score, N invalid N represents the number of invalid value records. orphan N represents the number of externally associated abnormal records. logic T represents the number of business logic conflict records. records F represents the total number of records in the static snapshot of the original data. validated The number of fields that need to be validated in each record.

[0118] Specifically, the validity dimension score is a numerical score calculated using a formula, typically ranging from 0 to 100. It quantifies the validity quality of the data; a higher score indicates more accurate and compliant data with fewer invalidity issues. The number of invalid value records refers to the number of records in the original data snapshot that violate value range rules, meaning the records contain values ​​outside the reasonable range. The number of records with external correlation anomalies refers to the number of records in the original data snapshot whose codes do not exist in the authoritative data list, reflecting inconsistencies between the data and external systems. The number of records with business logic conflicts refers to the number of records in the original data snapshot that violate business logic rules. The total number of records in the original static data snapshot refers to the total number of records extracted. The number of fields in each record that need validity verification is a predefined constant, representing the number of fields in each data record that need to undergo validity checks. The terminal extracts relevant statistics from the data quality baseline assessment report and substitutes them into a given mathematical formula. The numerator of the formula represents the total number of records with all validity issues, covering value range, correlation, and logic aspects; the denominator represents the total number of fields in all records that need validity verification, reflecting the overall scope of validity checks.

[0119] Step 503: Determine the completeness dimension score and the validity dimension score as the initial scores.

[0120] The initial scores specifically refer to the calculated completeness and effectiveness scores, which are the raw quantitative values ​​of each quality dimension without weight adjustment. The terminal formally confirms the completeness and effectiveness scores as the initial scores for this assessment, mainly by storing or marking these two scores as initial scores for use in subsequent steps, ensuring that the scores of each dimension remain independent and clear before entering the comprehensive calculation.

[0121] This embodiment provides an objective metric by calculating a completeness score, enabling engineers to quickly understand the data's performance in terms of integrity. A high score indicates fewer missing data and higher reliability, while a low score suggests the need to check the data acquisition or transmission process. By calculating a validity score, the embodiment intuitively reflects the data's quality level in terms of correctness, relevance, and logical consistency. A high score indicates high data credibility, making it suitable for in-depth analysis, while a low score indicates that the data contains many errors and needs to be cleaned or verified, providing crucial input for subsequent weighted and comprehensive evaluation.

[0122] In one embodiment, a comprehensive evaluation report is generated based on a data quality quantitative scoring table and an intelligent diagnostic report, including:

[0123] Step 601: Based on the data quality quantitative scoring table and intelligent diagnostic report, the overall data quality is rated to obtain the rating result; and based on the rating result, data usage suggestions are generated to obtain the data usage suggestion letter.

[0124] The data quality quantification scoring table includes a comprehensive data quality index and initial scores and levels for each dimension. The intelligent diagnostic report, derived from the root cause diagnostic model, analyzes the root causes of data quality issues. The rating result is a qualitative label representing the quality level, determined based on the score range of the comprehensive data quality index. Data usage recommendations, based on the rating result and diagnostic information, provide specific guidance on how to use the current data. The data usage recommendation document, based on the rating result and insights from the intelligent diagnostic report, indicates the applicable scenarios for the current data. For example, data rated "excellent" can be directly used for detailed analysis and decision-making; data rated "acceptable" needs cleaning before being used for trend analysis; and data rated "poor" is not recommended for direct use, with explanations of the main reasons. The terminal reads the comprehensive data quality index from the data quality quantification scoring table and compares it with predefined rating standards to determine a clear rating result. Combining the rating result with the specific content of the intelligent diagnostic report, targeted usage recommendations are generated.

[0125] Step 602: Based on the data usage proposal and the original data snapshot, generate business charts; the business charts include at least one of the following: personnel jump line probability chart, equipment jump line probability chart, steel wire jump line probability chart, grooved wheel jump line probability chart, and grooved wheel position jump line probability chart.

[0126] Specifically, business charts refer to graphical representations used to intuitively display business analysis results. In this embodiment, they specifically refer to charts used to monitor and analyze the probability of a jumper event. Jumper probability refers to the frequency or likelihood of a jumper event occurring under a specific dimension or group. The personnel jumper probability chart displays the jumper probability for each operator, with the operator as the dimension. The equipment jumper probability chart displays the jumper probability for each piece of equipment, with the equipment as the dimension. The steel wire jumper probability chart displays the jumper probability for different batches of steel wire, with the steel wire batch as the dimension. The Geneva wheel jumper probability chart and Geneva wheel position jumper probability chart display the jumper probability based on the Geneva wheel and its specific position, respectively. The terminal performs appropriate cleaning and processing on the original data snapshot according to the guidance in the data usage recommendation, and then performs analysis and visualization. It reads the data usage recommendation, determines the data processing strategy based on the rating and recommendations therein, cleans the original data snapshot according to the strategy, and obtains a dataset that can be used for analysis. According to business needs, different dimensions are selected to group the cleaned data. For each group, the jump probability is calculated, and these probability data are plotted into corresponding business charts.

[0127] Step 603: Based on the intelligent diagnostic report, match the corresponding improvement measures; and prioritize the improvement measures to obtain a data quality improvement priority list.

[0128] Specifically, improvement measures refer to the planned actions to eliminate or mitigate the root causes of data quality issues identified in the intelligent diagnostic report. Prioritization refers to the action of ranking multiple improvement measures according to factors such as the scope of the problem's impact, severity, and difficulty of resolution. The data quality improvement priority list is a list that shows all matched improvement measures and ranks them by priority. The terminal parses the intelligent diagnostic report, extracts each root cause identified, and matches one or more specific improvement measures for each cause based on a pre-set knowledge base. Taking into account factors such as the severity and scope of the problem, all matched improvement measures are ranked. For example, measures affecting data acquisition from core equipment have high priority, while measures with less impact and high resolution costs have low priority.

[0129] Step 604: Integrate the data quality improvement priority list, business charts, and data usage recommendations to obtain a comprehensive evaluation report.

[0130] The comprehensive assessment report is a complete report that summarizes all key findings of the data quality assessment, visualization analysis results, usage guidance, and improvement plans. The client creates a report template, filling in the corresponding sections with a data quality improvement priority list, business charts, data usage recommendations, and other possible information. The integration process ensures the report content is coherent and logically clear. After showcasing the issues revealed by the business charts, it immediately presents the improvement priority list, forming a closed loop from problem identification to problem resolution.

[0131] This embodiment provides engineers with a one-stop information center by generating a comprehensive evaluation report, which includes all-round information from data quality status rating, visualized business insights, data usage precautions to specific improvement measures, greatly assisting in equipment diagnosis, troubleshooting and continuous optimization.

[0132] In one embodiment, a business chart is generated based on the data usage proposal and the original data snapshot, including:

[0133] Step 701: Based on the rating results in the data usage recommendation, determine the rigor of data cleaning and obtain the strategy specification.

[0134] The data usage recommendation includes a rating of the overall data quality and preliminary usage suggestions. The rigor of data cleaning refers to the level of strictness of the criteria used when processing problematic records in the raw data. The strategy specification is a detailed operational guide that clearly defines the appropriate level of cleaning strategy for the current data quality level and includes detailed rules. The terminal reads the rating result from the data usage recommendation and, based on this rating, selects and determines the corresponding level of data cleaning rigor from a predefined strategy library. For example, an excellent rating means very high data quality, and the cleaning strategy can be minimal cleaning or no cleaning at all, directly using the raw data; a satisfactory rating allows for standard cleaning strategies to correct or remove obvious errors; a poor rating initiates a strict cleaning strategy, broadly removing problematic data and recommending data re-collection, thus transforming the abstract rating into concrete, actionable data processing guidelines.

[0135] Step 702: Based on the strategy specification, perform data cleaning on the problem records in the original data snapshot to obtain a high-quality dataset.

[0136] Specifically, problem records are those identified as having various issues in the raw data snapshot by the data quality baseline assessment report. Data cleaning is a series of operations, such as correction, filling, or removal of problem records, according to established strategies to improve data quality. A high-quality dataset is a collection of data obtained after cleaning, which minimizes known data quality issues and can be used for precise business analysis and chart generation. The terminal loads the strategy specification and processes each problem record in the raw data snapshot according to the stringency specified therein. For example, for missing values, the record is filled or directly removed according to the strategy; for outliers, records exceeding the tolerance limit are removed or marked as invalid according to the strategy specification; for invalid codes or logical conflicts, they are corrected or removed. All cleaning operations strictly follow the guidance of the strategy specification to ensure consistency in processing methods.

[0137] Step 703: Based on the dimensions of the business chart, group the data in the high-quality dataset to obtain data groups; and calculate the jump probability for each data group to obtain a summary table of business indicators.

[0138] Specifically, the dimensions of a business chart are the classification perspectives used for analysis and visualization. Data groups are the subsets of data obtained after grouping operations. Jump probability is the frequency of jump events occurring for a given data group. The business metric summary table is a structured table listing the groups under each dimension and their corresponding jump probability calculations. The terminal determines the analysis dimensions based on the type of business chart to be generated. Then, according to this dimension, the records in the high-quality dataset are grouped, and each data group is traversed. Within each group, it counts the total number of records and the number of records marked as having experienced jump events, calculates the jump probability for that group using a formula, and organizes all groups and their probability values ​​into a clear table—the business metric summary table.

[0139] Step 704: Generate basic business charts based on the business indicator summary table; and add explanatory text to the basic business charts to obtain the business charts.

[0140] Basic business charts refer to the initial graphical representations generated directly from the data in the business indicator summary table. At this stage, the charts only have basic elements such as axes, data points, and legends. Explanatory text refers to supplementary text added to the charts to explain the content, highlight key findings, indicate data sources, or add analytical conclusions. Business charts are the final version of the basic business charts, with enhanced elements such as explanatory text added, and can be directly used for reports and presentations. The terminal reads the business indicator summary table, uses visualization libraries or tools to convert the tabular data into graphics, generates text annotations in natural language based on the analysis results, and adds text annotations to appropriate locations on the basic charts. This aims to make the charts more readable, provide more complete information, and help readers quickly understand the business significance revealed by the charts.

[0141] This embodiment generates business charts to intuitively display the results of data analysis. It also provides context and interpretation through explanatory text, which greatly enhances the persuasiveness and operability of the comprehensive evaluation report and can directly assist engineers in diagnosis and troubleshooting.

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0143] Based on the same inventive concept, this application also provides a network monitoring data quality assessment device for implementing the network monitoring data quality assessment method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more network monitoring data quality assessment device embodiments provided below can be found in the limitations of the network monitoring data quality assessment method described above, and will not be repeated here.

[0144] In one exemplary embodiment, such as Figure 2 As shown, a wired network monitoring data quality assessment device 800 is provided, comprising:

[0145] The snapshot module 801 is used to extract records generated during the operation of the equipment at the corresponding time from the source database based on the evaluation period and the key data list to obtain the original data snapshot; the key data list represents the data required to generate business charts.

[0146] Basic module 802 is used to perform a basic quality assessment on the validity and completeness of the original data snapshot, and to obtain a basic data quality assessment report;

[0147] The quantification module 803 is used to quantify the quality of data into corresponding scores based on the basic data quality assessment report, and obtain a data quality quantification scoring table.

[0148] The root cause module 804 is used to input the basic data quality assessment report into the root cause diagnosis model to obtain an intelligent diagnosis report.

[0149] The reporting module, 805, is used to generate a comprehensive evaluation report based on the data quality quantitative scoring table and intelligent diagnostic report; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting equipment.

[0150] Furthermore, the basic module 802 is also used for:

[0151] Traverse the records in the original data snapshot, count the number of records for each device within the evaluation period, and obtain a time completeness report;

[0152] Iterate through the records in the original data snapshot, count the records with missing key fields, and obtain a report of missing key information;

[0153] Based on the original data snapshot, the records are verified for value domain compliance to obtain a value domain violation report;

[0154] The code is compared with the corresponding authoritative data list in the original data snapshot. Codes that are not in the authoritative data list are identified, and an external correlation anomaly report is obtained. The authoritative data list includes valid operators, wire batch numbers and groove wheel numbers.

[0155] Business logic verification is performed on the original data snapshot to obtain a logic conflict report; and the time completeness report, key information missing report, value range violation report, external correlation anomaly report and logic conflict report are integrated to obtain a basic data quality assessment report.

[0156] Furthermore, the basic module 802 is also used for:

[0157] Based on predefined static rules, records in the original data snapshot that violate the static rules are marked to obtain a static rule violation list; static rules include at least one of the following: a valid dictionary, format matching, and absolute range.

[0158] Based on the historical data of each device, the dynamic baseline of each operating parameter included in the record is calculated using the following formula:

[0159] UCL=μ+kσ

[0160] LCL=μ-kσ

[0161] Where UCL is the upper control limit, LCL is the lower control limit, μ is the mean, σ is the standard deviation, and k is the control limit multiplier factor;

[0162] Extract operational parameters that are not in the static rule violation list; operational parameters characterize the operating status of the device.

[0163] Compare the operating parameters with the corresponding dynamic baseline; mark the operating parameters that are not within the dynamic baseline as abnormal and generate a dynamic baseline abnormality report;

[0164] By integrating the static rule violation list and the dynamic baseline anomaly report, a value range violation report is obtained.

[0165] Furthermore, the quantization module 803 is also used for:

[0166] Based on the evaluation objectives, core quality dimensions are determined; and weights are assigned to these core quality dimensions to obtain a data quality dimension weight table; the core quality dimensions include completeness and effectiveness.

[0167] Based on the data quality dimension weight table and the basic data quality assessment report, the quantitative scores of each dimension are calculated to obtain the initial scores of each dimension;

[0168] Based on the data quality dimension weight table, the initial scores of each dimension are weighted and summed to obtain the comprehensive data quality index;

[0169] The comprehensive data quality index and each initial score are mapped to the corresponding level to obtain the data quality quantitative scoring table.

[0170] Furthermore, the quantization module 803 is also used for:

[0171] Based on the basic data quality assessment report, the completeness dimension score in the data quality dimension weight table is calculated using the following formula:

[0172]

[0173] Among them, S c For the completeness dimension score, N missing N represents the total number of missing data records. empty T represents the number of records with missing key fields. records F represents the total number of records in the static snapshot of the original data. critical T represents the number of key fields included in each record. devices T represents the total number of devices. intervals To record the number of intervals;

[0174] Based on the basic data quality assessment report, the validity dimension score in the data quality dimension weight table is calculated using the following formula:

[0175]

[0176] Among them, S v For the validity dimension score, N invalid N represents the number of invalid value records. orphan N represents the number of externally associated abnormal records. logic T represents the number of business logic conflict records. records F represents the total number of records in the static snapshot of the original data. validated The number of fields in each record that need to be validated;

[0177] The scores for completeness and effectiveness were used as the initial scores.

[0178] Furthermore, report module 805 is also used for:

[0179] Based on the data quality quantitative scoring table and intelligent diagnostic report, the overall data quality is rated to obtain the rating result; and based on the rating result, data usage suggestions are generated to obtain the data usage suggestion letter.

[0180] Based on the data usage proposal and the original data snapshot, generate business charts; the business charts include at least one of the following: personnel jump line probability chart, equipment jump line probability chart, steel wire jump line probability chart, grooved wheel jump line probability chart, and grooved wheel position jump line probability chart;

[0181] Based on the intelligent diagnostic report, corresponding improvement measures are matched; and the improvement measures are prioritized to obtain a data quality improvement priority list.

[0182] By integrating the data quality improvement priority list, business charts, and data usage recommendations, a comprehensive evaluation report is obtained.

[0183] Furthermore, report module 805 is also used for:

[0184] Based on the rating results in the data usage recommendation, the rigor of data cleaning is determined, and a strategy specification is obtained.

[0185] Based on the strategy specification, data cleaning was performed on the problem records in the original data snapshot to obtain a high-quality dataset.

[0186] Based on the dimensions of the business charts, the data in the high-quality dataset is grouped to obtain data groups; and the jump probability is calculated for each data group to obtain a summary table of business indicators.

[0187] Generate basic business charts based on the business indicator summary table; and add explanatory text to the basic business charts to obtain the business charts.

[0188] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above-described method for assessing the quality of network monitoring data.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0190] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0191] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for evaluating the quality of wired network monitoring data, characterized in that, The method includes: Based on the evaluation cycle and key data list, records generated during the corresponding time period of equipment operation are extracted from the source database to obtain a raw data snapshot; the key data list represents the data required to generate business charts. A basic quality assessment of the validity and completeness of the original data snapshot is performed to obtain a basic data quality assessment report; Based on the aforementioned data quality basic assessment report, the quality of the data is quantified into corresponding scores to obtain a data quality quantification scoring table; Input the aforementioned data quality baseline assessment report into the root cause diagnosis model to obtain an intelligent diagnosis report; Based on the data quality quantification scoring table and the intelligent diagnostic report, a comprehensive evaluation report is generated; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting the equipment.

2. The method according to claim 1, characterized in that, The basic quality assessment of the original data snapshot regarding its validity and completeness is performed to obtain a basic data quality assessment report, including: Traverse the records in the original data snapshot, count the number of records existing for each device within the evaluation period, and obtain a time completeness report; Traverse the records in the original data snapshot, count the records with missing key fields, and obtain a key information missing report; Based on the original data snapshot, the value domain compliance of the record is verified to obtain a value domain violation report; The original data snapshot is compared with the corresponding authoritative data list to identify codes that are not present in the authoritative data list, thus generating an external association anomaly report; the authoritative data list includes valid operators, wire batch numbers, and grooved wheel numbers; The original data snapshot is validated using business logic to obtain a logic conflict report; and the time completeness report, the key information missing report, the value range violation report, the external association anomaly report, and the logic conflict report are integrated to obtain the basic data quality assessment report.

3. The method according to claim 2, characterized in that, The step of verifying the value range compliance of the records based on the original data snapshot to obtain a value range violation report includes: Based on predefined static rules, records in the original data snapshot that violate the static rules are marked to obtain a static rule violation list; the static rules include at least one of the following: a valid dictionary, format matching, and absolute range. Based on the historical data of each of the aforementioned devices, the dynamic baseline of each operating parameter contained in the record is calculated using the following formula: UCL=μ+kσ LCL=μ-kσ Where UCL is the upper control limit, LCL is the lower control limit, μ is the mean, σ is the standard deviation, and k is the control limit multiplier factor; Extract the operating parameters that are not in the static rule violation list; the operating parameters characterize the operating status of the device. The operating parameters are compared with the corresponding dynamic baseline; and the operating parameters that are not within the dynamic baseline are marked as abnormal, generating a dynamic baseline abnormality report. The value range violation report is obtained by integrating the static rule violation list and the dynamic baseline anomaly report.

4. The method according to claim 1, characterized in that, Based on the data quality fundamental assessment report, the quality of the data is quantified into corresponding scores to obtain a data quality quantification scoring table, including: Based on the evaluation objectives, core quality dimensions are determined; and weights are assigned to the core quality dimensions to obtain a data quality dimension weight table; the core quality dimensions include completeness and effectiveness. Based on the data quality dimension weight table and the data quality basic assessment report, the quantitative score of each dimension is calculated to obtain the initial score of each dimension; Based on the data quality dimension weight table, the initial scores of each dimension are weighted and summed to obtain the comprehensive data quality index. The comprehensive data quality index and each of the initial scores are mapped to the corresponding levels to obtain the data quality quantitative scoring table.

5. The method according to claim 4, characterized in that, The process involves calculating the quantitative score for each dimension based on the data quality dimension weight table and the data quality basic assessment report, to obtain the initial score for each dimension, including: Based on the aforementioned data quality fundamental assessment report, the completeness dimension score in the data quality dimension weight table is calculated using the following formula: Among them, S c For the completeness dimension score, N missing N represents the total number of missing data records. empty T represents the number of records with missing key fields. records F represents the total number of records in the static snapshot of the original data. critical T represents the number of key fields included in each record. devices T represents the total number of devices. intervals To record the number of intervals; Based on the aforementioned data quality fundamental assessment report, the validity dimension score in the data quality dimension weight table is calculated using the following formula: Among them, S v For the validity dimension score, N invalid N represents the number of invalid value records. orphan N represents the number of externally associated abnormal records. logic T represents the number of business logic conflict records. records F represents the total number of records in the static snapshot of the original data. validated The number of fields in each record that need to be validated; The completeness dimension score and the validity dimension score are determined as the initial scores.

6. The method according to claim 1, characterized in that, The process of generating a comprehensive evaluation report based on the data quality quantification scoring table and the intelligent diagnostic report includes: Based on the data quality quantitative scoring table and the intelligent diagnostic report, the overall data quality is rated to obtain a rating result; and based on the rating result, usage suggestions for the data are generated to obtain a data usage suggestion document. Based on the data usage proposal and the original data snapshot, the business chart is generated; the business chart includes at least one of the following: personnel jump line probability chart, equipment jump line probability chart, steel wire jump line probability chart, grooved wheel jump line probability chart, and grooved wheel position jump line probability chart; Based on the intelligent diagnostic report, corresponding improvement measures are matched; and the improvement measures are prioritized to obtain a data quality improvement priority list. The comprehensive evaluation report is obtained by integrating the data quality improvement priority list, the business charts, and the data usage recommendations.

7. The method according to claim 6, characterized in that, The process of generating the business chart based on the data usage proposal and the original data snapshot includes: Based on the rating results in the data usage recommendation, the rigor of data cleaning is determined, and a strategy specification is obtained. Based on the strategy specification, the problem records in the original data snapshot are cleaned to obtain a high-quality dataset. Based on the dimensions of the business chart, the data in the high-quality dataset is grouped to obtain data groups; and the jump probability is calculated for each data group to obtain a summary table of business indicators. A basic business chart is generated based on the business indicator summary table; explanatory text is added to the basic business chart to obtain the business chart.

8. A network monitoring data quality assessment device, characterized in that, The device includes: The snapshot module is used to extract records generated during the operation of the equipment at the corresponding time from the source database based on the evaluation period and the key data list to obtain the original data snapshot; the key data list represents the data required to generate business charts; The basic module is used to perform a basic quality assessment on the validity and completeness of the original data snapshot, and obtain a basic data quality assessment report; The quantification module is used to quantify the quality of the data into corresponding scores based on the data quality basic assessment report, and obtain a data quality quantification scoring table. The root cause module is used to input the data quality basic assessment report into the root cause diagnosis model to obtain an intelligent diagnosis report; The reporting module is used to generate a comprehensive evaluation report based on the data quality quantitative scoring table and the intelligent diagnostic report; the comprehensive evaluation report is used to assist engineers in diagnosing and troubleshooting the equipment.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images