Multi-source data quality evaluation method and system

Abstract

The invention discloses a multi-source data quality evaluation method and system, and the method comprises the steps: obtaining the operation state and related environment information of power transmission and transformation equipment from a plurality of data sources, and carrying out the preprocessing of the obtained data; performing feature extraction on the preprocessed data to obtain feature vectors of key data features; based on feature vectors obtained through feature extraction, training the extracted features through a deep learning algorithm, and constructing a data quality evaluation model; the built data quality evaluation model is used for monitoring newly input data in real time, deviation and abnormity in the data are recognized, and a real-time monitoring result is obtained; and timely feeding back a real-time monitoring result to the data acquisition and transmission system, and adjusting the data acquisition and transmission system according to feedback information to obtain a data quality comprehensive evaluation result. Dynamic optimization is carried out according to the characteristics and weights of different data sources, and the comprehensive evaluation capability of the data quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a multi-source data quality evaluation method and system. Background Art

[0002] In today's rapidly developing power system, the digital transformation of power transmission and transformation equipment has become the key to improving operational efficiency and safety. Digital twin technology uses data from physical equipment to create virtual models to monitor and analyze equipment status in real time. However, the effectiveness of digital twins depends on the quality of input data. At present, data collection for power transmission and transformation equipment mainly comes from multiple data sources such as sensors, monitoring systems, and historical records. The diversity of these data sources leads to large differences in data quality. Traditional data quality assessment methods are often based on a single data source, which cannot fully reflect the status of the equipment, and have obvious deficiencies in data integration and anomaly detection, which makes it difficult for the system to identify and respond to data quality issues in real time, increasing the risk of decision-making.

[0003] As the amount of equipment and data continues to increase, how to effectively manage and evaluate this data and ensure its accuracy, completeness and consistency has become a technical challenge that needs to be solved. Therefore, an innovative method is urgently needed to comprehensively evaluate the quality of multi-dimensional data from different data sources. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a multi-source data quality evaluation method and system to solve the problem that the existing data quality evaluation methods have deficiencies in data integration and anomaly detection, making it difficult for the system to identify and respond to data quality in real time.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a multi-source data quality evaluation method, comprising:

[0009] Obtain the operating status and related environmental information of power transmission and transformation equipment from multiple data sources, and pre-process the acquired data;

[0010] By extracting features from the preprocessed data, feature vectors of key data characteristics are obtained;

[0011] Based on the feature vectors obtained by feature extraction, the extracted features are trained through deep learning algorithms to build a data quality evaluation model;

[0012] Use the constructed data quality evaluation model to monitor the newly input data in real time, identify deviations and anomalies in the data, and obtain real-time monitoring results;

[0013] The real-time monitoring results are fed back to the data collection and transmission system in a timely manner. Based on the feedback information, the data collection and transmission system is adjusted to obtain a comprehensive evaluation result of the data quality.

[0014] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0015] The method of obtaining the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources and preprocessing the obtained data includes the following steps:

[0016] Obtain the operating status and related environmental information of power transmission and transformation equipment from multiple data sources;

[0017] Set initial quality targets;

[0018] Comprehensively clean and normalize the data using data cleaning, missing value filling, denoising, standardization and normalization;

[0019] The initial quality index is expressed as:

[0020] Q0=f(F,M,N)

[0021] Among them, F represents the conformity of the data format, M represents the number of missing values, and N represents the noise level.

[0022] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0023] The feature vectors of key data characteristics include timeliness, consistency, completeness and noise level;

[0024] The timeliness T is expressed as:

[0025] T=t current -t data

[0026] Among them, t current is the current timestamp, t data is the timestamp of the data record;

[0027] The consistency C is expressed as:

[0028]

[0029] Among them, n missing is the number of missing fields, n total is the total number of fields in the dataset;

[0030] The integrity I is expressed as:

[0031]

[0032] Among them, n missing is the number of missing fields, n total is the total number of fields in the dataset;

[0033] The noise level SNR is expressed as:

[0034]

[0035] Among them, P signal is the signal power, P noise is the noise power.

[0036] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0037] Based on the feature vector obtained by feature extraction, the extracted features are trained through a deep learning algorithm to build a data quality evaluation model, including the following steps:

[0038] Combine the extracted timeliness, consistency, completeness and noise level into a feature vector X;

[0039] Assign a quality score Q to each feature i ;

[0040] Calculate the weight w of each feature i ;

[0041] Use a deep learning algorithm to train the model, taking the feature vector X as input and the quality score of the annotations as output;

[0042] Use the trained model to predict new data and get the value of each feature;

[0043] Calculate the comprehensive score Q according to the weight;

[0044] The weight w of each feature is calculated i It is expressed as:

[0045]

[0046] Among them, Q i is the quality score of the i-th feature, and n is the total number of features;

[0047] The comprehensive score Q is expressed as:

[0048] Q=w1T+w2C+w3I+w4SNR.

[0049] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0050] The method of using the constructed data quality evaluation model to monitor the newly input data in real time, identify deviations and anomalies in the data, and obtain real-time monitoring results includes the following steps:

[0051] Use the constructed data quality evaluation model to conduct real-time evaluation of newly input data;

[0052] Set up real-time monitoring indicators Q t ;

[0053] Real-time monitoring of Q t When Q t When the drop exceeds the preset threshold, the system triggers an alarm and records the timestamp and feature value of the abnormal data;

[0054] Through the adaptive algorithm, the weight w of each feature is adjusted according to the actual situation i (t);

[0055] Use the new weight w i (t) Recalculate the quality score at the current moment.

[0056] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0057] The setting of real-time monitoring indicator Q t It is expressed as:

[0058] Q t =Q t-1 +ΔQ

[0059] Among them, ΔQ represents the difference in quality score between the current moment and the previous moment;

[0060] Adjust the weight w of each feature according to the actual situation i (t) is expressed as:

[0061] w i (t) = w i (t-1)+α·e(t)

[0062] Among them, w i (t) is the weight of the feature at time t, w i (t-1) is the weight of the feature at the previous time (t-1), α is the learning rate, which controls the amplitude of weight adjustment, and e(t) is the current error.

[0063] As a preferred solution of the multi-source data quality evaluation method of the present invention, wherein:

[0064] The method of adjusting the data collection and transmission system according to the feedback information to obtain a comprehensive evaluation result of the data quality includes the following steps:

[0065] Set up feedback adjustment model;

[0066] By combining the current data quality score with the quality of the processed data, a feedback loop is formed to optimize the data collection and processing process;

[0067] The setting feedback adjustment model is expressed as:

[0068] F m =f(Q t ,C t )

[0069] Among them, C t Represents the data quality score after feedback adjustment.

[0070] In a second aspect, the present invention provides a multi-source data quality evaluation system, comprising:

[0071] The data acquisition and preprocessing module is used to obtain the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources, and preprocess the acquired data;

[0072] A feature extraction module is used to extract features from the preprocessed data to obtain feature vectors of key data characteristics;

[0073] The data quality evaluation model building module is used to train the extracted features through a deep learning algorithm based on the feature vectors obtained by feature extraction to build a data quality evaluation model;

[0074] The real-time monitoring module is used to use the constructed data quality evaluation model to perform real-time monitoring on the newly input data, identify deviations and anomalies in the data, and obtain real-time monitoring results;

[0075] The quality feedback and adjustment module is used to promptly feed back the real-time monitoring results to the data acquisition and transmission system. According to the feedback information, the data acquisition and transmission system is adjusted to obtain a comprehensive evaluation result of the data quality.

[0076] In a third aspect, the present invention provides a computing device, comprising:

[0077] Memory, used to store programs;

[0078] The processor is used to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the multi-source data quality assessment method.

[0079] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising: when the program is executed by a processor, the steps of implementing the multi-source data quality assessment method are implemented.

[0080] Beneficial effects of the invention: The invention establishes a comprehensive evaluation framework covering multiple dimensions such as timeliness, consistency, completeness and noise level. Through this multi-dimensional analysis method, data quality issues can be fully identified to ensure that the model obtains high-quality data input. In addition, deep learning algorithms are used to automatically extract data features and identify potential quality issues. At the same time, statistical models are combined to detect data anomalies, thereby improving the accuracy of the model and enhancing its adaptability to complex data patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0082] Figure 1 A basic flow chart of a multi-source data quality evaluation method provided by one embodiment of the present invention;

[0083] Figure 2 A schematic diagram of a data quality evaluation model of a multi-source data quality evaluation method provided by an embodiment of the present invention;

[0084] Figure 3 A schematic diagram of a feedback adjustment model of a multi-source data quality evaluation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0086] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0087] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0088] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0089] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0090] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0091] Example 1

[0092] Reference Figure 1-3 and Table 1, which is an embodiment of the present invention, provides a multi-source data quality evaluation method, such as Figure 1 As shown, the following steps are included:

[0093] S1: Obtain the operating status and related environmental information of power transmission and transformation equipment from multiple data sources, and pre-process the acquired data;

[0094] In the embodiments of the present application, relevant data are first systematically collected from multiple data sources (such as sensors, monitoring systems, and historical records) to ensure the diversity and comprehensiveness of the data. The collected data may be affected by different formats, missing values, or noise, so comprehensive preprocessing is required. Table 1 shows the effects of different preprocessing methods, including the comparison of the advantages and disadvantages of standardization, denoising, and normalization.

[0095] Table 1 Comparison of data preprocessing methods

[0096]

[0097]

[0098] To quantify the initial quality of the data, an initial quality index Q0 is set:

[0099] Q0=f(F,M,N)

[0100] Among them, F represents the conformity of the data format, M represents the number of missing values, and N represents the noise level. By combining these factors, Q0 provides a baseline for subsequent feature extraction and helps identify potential problems with data quality.

[0101] In the embodiment of the present application, for the missing data, interpolation and other techniques are used to fill in to ensure the integrity of the data set. The existence of missing data may seriously affect the accuracy of the analysis results, so it must be solved. At the same time, in order to reduce the random noise in the data, a filtering algorithm (such as Kalman filtering) is used to improve the data quality. The goal of this step is to clean and normalize the data as much as possible so that it is suitable for subsequent in-depth analysis.

[0102] S2: Extract features from the preprocessed data to obtain feature vectors of key data characteristics;

[0103] In the present embodiment, the preprocessed data is deeply analyzed to extract key data features. These features will provide necessary information for subsequent data quality evaluation. The purpose of feature extraction is to capture the core features of the data, making the subsequent quality evaluation more accurate and effective. First, calculate the timeliness T as:

[0104] T=t current -t data

[0105] Among them, t current is the current timestamp, t data is the timestamp of data record. The above formula reflects the delay of data update. The higher the timeliness, the greater the practical value of the data, which can ensure that the digital twin model uses the latest data during operation.

[0106] In the embodiment of the present application, the evaluation consistency C used to quantify the degree of variation of different data sources under the same indicator is:

[0107]

[0108] Among them, σ is the standard deviation of the data sample, and μ is the mean of the data sample. The higher the consistency, the smaller the difference between data sources, thus improving the overall data quality.

[0109] In the embodiment of the present application, the integrity I is calculated by the following formula:

[0110]

[0111] Among them, n missing is the number of missing fields, n total is the total number of fields in the data set. The above formula reflects the integrity of the data set. The smaller the ratio, the less missing data and the better the integrity.

[0112] In the embodiment of the present application, the noise level SNR is calculated as:

[0113]

[0114] Among them, P signal is the signal power, P noise is the noise power. The signal-to-noise ratio (SNR) is used to evaluate the ratio of signal to noise in the data. The higher the SNR value, the smaller the noise impact in the data, thereby improving the data quality. The various data characteristics obtained in the feature extraction stage will provide the necessary input information for the subsequent data quality evaluation model construction.

[0115] S3: Based on the feature vectors obtained by feature extraction, the extracted features are trained through deep learning algorithms to build a data quality evaluation model;

[0116] In the embodiment of the present application, a data quality evaluation model is constructed, and the extracted features are trained through a deep learning algorithm to identify potential data quality issues. This process requires a large amount of training data so that the model can effectively learn the potential patterns and characteristics of the data. After feature extraction, the obtained characteristics such as timeliness, consistency, completeness and noise level will be used as model input.

[0117] In the embodiment of the present application, in order to quantify the importance of each characteristic, the weights of these characteristics in the formula can be set as follows:

[0118]

[0119] Among them, Q iis the quality score of the i-th feature, and n is the total number of features. The weight reflects the relative importance of each feature in the comprehensive score, ensuring that the model takes into account the contribution of each feature when evaluating data quality. The final score Q of the comprehensive evaluation model is:

[0120] Q=w1T+w2C+w3I+w4SNR

[0121] The above formula forms a comprehensive quantitative evaluation system by weighted integration of features, which can better reflect the data quality status. Figure 2 The components and input-output relationship of the data quality evaluation model are given.

[0122] It should be noted that the constructed model can not only identify the quality status of the current data, but also provide a benchmark and reference for subsequent dynamic monitoring and optimization.

[0123] S4: Use the constructed data quality evaluation model to monitor the newly input data in real time, identify deviations and anomalies in the data, and obtain real-time monitoring results;

[0124] In the embodiment of the present application, in the dynamic monitoring and optimization stage, real-time monitoring of data input is implemented to identify deviations and anomalies to ensure that data quality is continuously evaluated during the input process. The key to this stage is to be able to respond to changes in data quality in real time so that corresponding measures can be taken quickly.

[0125] In the embodiment of the present application, based on the constructed evaluation model, real-time monitoring can promptly discover the problem of data quality degradation and make corresponding adjustments. In real-time monitoring, real-time monitoring indicators can be set:

[0126] Q t =Q t-1 +ΔQ

[0127] Among them, ΔQ represents the difference in quality score between the current moment and the previous moment.

[0128] It should be noted that through such dynamic monitoring, the system can quickly identify problematic data and handle it in a timely manner.

[0129] In the embodiment of the present application, the weight of each characteristic is adjusted according to the actual situation through an adaptive algorithm:

[0130] w i (t) = w i (t-1)+α·e(t)

[0131] Among them, w i (t) is the weight of the feature at time t, w i(t-1) is the weight of the feature at the previous time (t-1), α is the learning rate, which controls the amplitude of weight adjustment, and e(t) is the current error, which reflects the difference between the model prediction and the actual.

[0132] It should be noted that this dynamic adjustment mechanism can effectively respond to changes in data sources and improve the adaptability of the evaluation system. The successful implementation of this process will further improve the real-time performance and reliability of the digital twin model.

[0133] S5: Feedback the real-time monitoring results to the data collection and transmission system in a timely manner. According to the feedback information, adjust the data collection and transmission system to obtain a comprehensive evaluation result of the data quality.

[0134] In the embodiment of the present application, a feedback mechanism is established to promptly feed back the data quality evaluation results to the data acquisition and transmission system. This mechanism ensures that data problems can be quickly identified and processed, avoiding the accumulation of data quality problems.

[0135] Based on the results of dynamic monitoring, the data collection and processing processes are adjusted in a timely manner to ensure that the digital twin model obtains high-quality data input.

[0136] In the embodiment of the present application, the feedback adjustment model F is set m :

[0137] F m =f(Q t ,C t )

[0138] Among them, C t Represents the data quality score after feedback adjustment. The feedback adjustment model forms a feedback loop by combining the current data quality score and the quality of the processed data, thereby optimizing the data collection and processing process and ensuring that the digital twin model obtains high-quality data input. Figure 3 Shows how the model adjusts after receiving feedback.

[0139] Through continuous quality feedback and adjustment, a closed-loop quality management system is eventually formed, laying the foundation for the effective operation and maintenance of power transmission and transformation equipment. This system can automatically adapt to different operating conditions and data source changes, providing stable and reliable data support for the digital twin system.

[0140] This embodiment also provides a multi-source data quality evaluation system, including:

[0141] The data acquisition and preprocessing module is used to obtain the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources, and preprocess the acquired data;

[0142] A feature extraction module is used to extract features from the preprocessed data to obtain feature vectors of key data characteristics;

[0143] The data quality evaluation model building module is used to train the extracted features through a deep learning algorithm based on the feature vectors obtained by feature extraction to build a data quality evaluation model;

[0144] The real-time monitoring module is used to use the constructed data quality evaluation model to perform real-time monitoring on the newly input data, identify deviations and anomalies in the data, and obtain real-time monitoring results;

[0145] The quality feedback and adjustment module is used to promptly feed back the real-time monitoring results to the data acquisition and transmission system. According to the feedback information, the data acquisition and transmission system is adjusted to obtain a comprehensive evaluation result of the data quality.

[0146] Furthermore, it also includes:

[0147] Memory, used to store programs;

[0148] A processor is used to load the program to execute the multi-source data quality assessment method.

[0149] This embodiment also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the multi-source data quality assessment method.

[0150] The storage medium proposed in this embodiment and the multi-source data quality evaluation method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0151] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0152] Example 2

[0153] This is an embodiment of the present invention, which provides a multi-source data quality evaluation system, including a data acquisition and preprocessing module, a feature extraction module, a data quality evaluation model construction module, a real-time monitoring module and a quality feedback and adjustment module;

[0154] In an embodiment of the present application, the data acquisition and preprocessing module includes acquiring the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources, and preprocessing the acquired data; ensuring that the collected data has high quality and integrity when entering the subsequent feature extraction and analysis stage, thereby improving the accuracy and reliability of the digital twin system of the power transmission and transformation equipment.

[0155] In the embodiment of the present application, the operating status and related environmental information of the power transmission and transformation equipment are obtained from multiple data sources including sensor data, monitoring system data, historical record data, maintenance and operation records and external environment data;

[0156] In the embodiments of the present application, sensor data is one of the most direct data sources, including the measured values ​​of physical quantities such as temperature, humidity, pressure, current, voltage, etc. These data are crucial for real-time monitoring of the operating status of the device and can help detect whether the device is in a normal operating range.

[0157] In the embodiments of the present application, the monitoring system data is not limited to the monitoring of physical parameters, but also includes non-traditional data such as video monitoring, acoustic monitoring, etc. These data can help identify whether the device has physical damage or abnormal behavior.

[0158] In the embodiment of the present application, the historical record data includes past maintenance records, fault reports, performance test results, etc. These historical data are very valuable for understanding the long-term performance trend of the equipment and predicting possible problems in the future.

[0159] In an embodiment of the present application, maintenance and operation records include records of daily maintenance activities of the equipment, operation logs of operators, etc. This information can provide insights into the maintenance status of the equipment and the compliance of operations.

[0160] In the embodiment of the present application, the external environmental data includes weather conditions, grid load conditions, etc. These external factors also have an important impact on the operation of the power transmission and transformation equipment.

[0161] In the embodiment of the present application, the feature extraction module includes extracting features from preprocessed data to obtain feature vectors of key data characteristics; extracting key data characteristics (such as statistical characteristics of equipment operating parameters, time series patterns, etc.) by in-depth analysis of the preprocessed data, which will be used for subsequent data quality evaluation. The key data characteristics can be extracted to provide necessary input information for the subsequent data quality evaluation model construction, thereby improving the accuracy and reliability of the digital twin system of power transmission and transformation equipment.

[0162] In an embodiment of the present application, the feature vectors of key data characteristics may include timeliness, consistency, completeness, and noise level. These feature vectors will provide necessary input information for the subsequent construction of data quality evaluation models. These feature vectors will be used for the subsequent construction of data quality evaluation models to help identify and evaluate potential quality problems in the data, thereby improving the accuracy and reliability of the digital twin system of power transmission and transformation equipment. In this way, key data characteristics can be systematically extracted and represented, providing a solid foundation for subsequent analysis and optimization.

[0163] In an embodiment of the present application, the data quality evaluation model construction module includes a feature vector obtained based on feature extraction, and the extracted features are trained by a deep learning algorithm to construct a data quality evaluation model; the extracted features are trained by a deep learning algorithm (such as a neural network) to construct a data quality evaluation model. The model can identify potential data quality issues, such as missing data, outliers, inconsistencies, etc. An effective data quality evaluation model can be constructed, which can not only identify the quality status of the current data, but also provide a benchmark and reference for subsequent dynamic monitoring and optimization.

[0164] In an embodiment of the present application, the real-time monitoring module includes using a constructed data quality evaluation model to perform real-time monitoring of newly input data, identify deviations and anomalies in the data, and obtain real-time monitoring results; this step ensures the consistency and accuracy of the data during collection, transmission and storage.

[0165] In the embodiment of the present application, the final comprehensive score Q is obtained after constructing and training the data quality evaluation model, using the model to predict new data, and calculating according to the weights. This comprehensive score Q provides a benchmark and reference for subsequent dynamic monitoring and optimization.

[0166] In the embodiment of the present application, real-time monitoring includes evaluating the data quality score Q in real time. t And the quality score Q of the previous moment t-1 Compare and identify deviations and anomalies in the data. When the drop exceeds the preset threshold, the system triggers an alarm and records the abnormal data for subsequent processing and optimization.

[0167] In the embodiment of the present application, the quality feedback and adjustment module includes timely feeding back the real-time monitoring results to the data acquisition and transmission system, adjusting the data acquisition and transmission system according to the feedback information, and obtaining a comprehensive evaluation result of the data quality, so that the system can make adjustments according to the feedback, such as recalibrating sensors, repairing data transmission problems, etc. This step forms a closed loop to ensure continuous optimization of data quality.

[0168] In the embodiment of the present application, the above modules can ensure that data quality issues can be quickly identified and processed, and ultimately form a closed-loop quality management system to provide reliable data support for the effective operation and maintenance of power transmission and transformation equipment and the stable operation of the digital twin system.

[0169] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-source data quality evaluation method, characterized in that: include: Obtain the operating status and related environmental information of power transmission and transformation equipment from multiple data sources, and pre-process the acquired data; By extracting features from the preprocessed data, feature vectors of key data characteristics are obtained; Based on the feature vectors obtained by feature extraction, the extracted features are trained through deep learning algorithms to build a data quality evaluation model; Use the constructed data quality evaluation model to monitor the newly input data in real time, identify deviations and anomalies in the data, and obtain real-time monitoring results; The real-time monitoring results are fed back to the data collection and transmission system in a timely manner. Based on the feedback information, the data collection and transmission system is adjusted to obtain a comprehensive evaluation result of the data quality.

2. The multi-source data quality evaluation method according to claim 1, characterized in that: The method of obtaining the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources and preprocessing the obtained data includes the following steps: Obtain the operating status and related environmental information of power transmission and transformation equipment from multiple data sources; Set initial quality targets; Comprehensively clean and normalize the data using data cleaning, missing value filling, denoising, standardization and normalization; The initial quality index is expressed as: Q0=f(F,M,N) Among them, F represents the conformity of the data format, M represents the number of missing values, and N represents the noise level.

3. The multi-source data quality assessment method according to claim 1 or 2, characterized in that: The feature vectors of key data characteristics include timeliness, consistency, completeness and noise level; The timeliness T is expressed as: T=t current -t data Among them, t current is the current timestamp, t data is the timestamp of the data record; The consistency C is expressed as: Among them, n missing is the number of missing fields, n total is the total number of fields in the dataset; The integrity I is expressed as: Among them, n missing is the number of missing fields, n total is the total number of fields in the dataset; The noise level SNR is expressed as: Among them, P signal is the signal power, P noise is the noise power.

4. The multi-source data quality evaluation method according to claim 3, characterized in that: Based on the feature vector obtained by feature extraction, the extracted features are trained through a deep learning algorithm to build a data quality evaluation model, including the following steps: Combine the extracted timeliness, consistency, completeness and noise level into a feature vector X; Assign a quality score Q to each feature i ; Calculate the weight w of each feature i ; Use a deep learning algorithm to train the model, taking the feature vector X as input and the quality score of the annotations as output; Use the trained model to predict new data and get the value of each feature; Calculate the comprehensive score Q according to the weight; The weight w of each feature is calculated i It is expressed as: Among them, Q i is the quality score of the i-th feature, and n is the total number of features; The comprehensive score Q is expressed as: Q=w1T+w2C+w3I+w4SNR.

5. The multi-source data quality evaluation method according to claim 4, characterized in that: The method of using the constructed data quality evaluation model to monitor the newly input data in real time, identify deviations and anomalies in the data, and obtain real-time monitoring results includes the following steps: Use the constructed data quality evaluation model to conduct real-time evaluation of newly input data; Set up real-time monitoring indicators Q t ; Real-time monitoring of Q t When Q t When the drop exceeds the preset threshold, the system triggers an alarm and records the timestamp and feature value of the abnormal data; Through the adaptive algorithm, the weight w of each feature is adjusted according to the actual situation i (t); Use the new weight w i (t) Recalculate the quality score at the current moment.

6. The multi-source data quality assessment method according to claim 5, characterized in that: The setting of real-time monitoring indicator Q t It is expressed as: Q t =Q t-1 +ΔQ Among them, ΔQ represents the difference in quality score between the current moment and the previous moment; Adjust the weight w of each feature according to the actual situation i (t) is expressed as: w i (t)=w i (t-1)+α·e(t) Among them, w i (t) is the weight of the feature at time t, w i (t-1) is the weight of the feature at the previous time (t-1), α is the learning rate, which controls the amplitude of weight adjustment, and e(t) is the current error.

7. The multi-source data quality assessment method according to claim 6, characterized in that: The method of adjusting the data collection and transmission system according to the feedback information to obtain a comprehensive evaluation result of the data quality includes the following steps: Set up feedback adjustment model; By combining the current data quality score with the quality of the processed data, a feedback loop is formed to optimize the data collection and processing process; The setting feedback adjustment model is expressed as: F m =f(Q t ,C t ) Among them, C t Represents the data quality score after feedback adjustment.

8. A system based on the multi-source data quality evaluation method according to claim 1, characterized in that: The data acquisition and preprocessing module is used to obtain the operating status and related environmental information of the power transmission and transformation equipment from multiple data sources and preprocess the acquired data; A feature extraction module is used to extract features from the preprocessed data to obtain feature vectors of key data characteristics; The data quality evaluation model building module is used to train the extracted features through a deep learning algorithm based on the feature vectors obtained by feature extraction to build a data quality evaluation model; The real-time monitoring module is used to use the constructed data quality evaluation model to perform real-time monitoring on the newly input data, identify deviations and anomalies in the data, and obtain real-time monitoring results; The quality feedback and adjustment module is used to promptly feed back the real-time monitoring results to the data acquisition and transmission system. According to the feedback information, the data acquisition and transmission system is adjusted to obtain a comprehensive evaluation result of the data quality.

9. A computing device, characterized in that include: Memory, used to store programs; A processor, configured to load the program to execute the steps of the multi-source data quality assessment method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the steps of the multi-source data quality assessment method according to any one of claims 1 to 7 are implemented.

Images