A multi-source heterogeneous load data cleaning method, device, equipment and medium

Abstract

The application discloses a multi-source heterogeneous load data cleaning method, device, equipment and medium, relates to the technical field of urban lifeline, and carries out multi-level verification on multi-source heterogeneous load data; if the verification is passed, determines the daily load data of non-IoT users, the full-caliber total load of civil communities, compensates the heating gap load data, obtains the heating load data, strips the non-heating load data, and generates full-caliber statistical data; based on the flow change rate between current flow data and adjacent flow data, the heterogeneous load data is processed; a virtual instrument sequence is generated based on target flow data; the daily load value is determined by using the virtual instrument sequence, if the daily load value is less than a threshold value, all daily load values of the same user are superimposed and floating point elimination is performed, a total daily load value is obtained, the full-caliber statistical data and the total daily load value are stored, full-quantity customer coverage and closed-loop management are realized, and the physical law self-consistency and logical accuracy of data cleaning are improved.

Description

Technical Field

[0001] This invention relates to the field of urban lifeline technology, and in particular to a method, apparatus, equipment and medium for cleaning multi-source heterogeneous load data. Background Technology

[0002] Currently, the data sources for energy load forecasting systems mainly include three categories: real-time reported data from IoT (Internet of Things) meters, non-IoT user data based on monthly bills, and aggregated data from residential communities. Due to differences in the hardware types of data collection terminals, communication protocols, and deviations in business flow rules, the raw energy load data exhibits multi-source heterogeneity and physical-logical discontinuity. Load data acquisition and preprocessing still mainly rely on local monitoring and basic statistics: For industrial and commercial users with IoT terminals installed, the operation and maintenance system uses the SCADA (Supervisory Control and Data Acquisition) system to pull instantaneous traffic and cumulative volume in real time, and sets simple upper and lower limit thresholds for alarms; for non-IoT existing users without installed sensors, the system mainly relies on manual monthly meter reading and entry into the business system to form offline static billing data; the existing accounting method is based on the estimation method of the community-level master table, which does not take into account the proportion of online and offline users and the actual situation that the proportion may fluctuate. When the collected data has missing values, the various algorithms used will automatically judge it as a null value or a zero value, which will cause the statistical accuracy to deviate and affect the reliability of the data source.

[0003] As can be seen from the above, how to achieve seamless coverage and closed-loop management of all customers, eliminate statistical blind spots in the data base, effectively solve the problem of insufficient accuracy of basic data in energy dispatch scenarios, and improve the physical consistency and logical accuracy of data cleaning are problems that need to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, equipment, and medium for cleaning multi-source heterogeneous load data, which can achieve seamless coverage and closed-loop management of all customers, eliminate statistical blind spots in the data base, effectively solve the problem of insufficient accuracy of basic data in gas dispatching scenarios, and improve the physical consistency and logical accuracy of data cleaning. The specific solution is as follows:

[0005] In a first aspect, this application discloses a method for cleaning multi-source heterogeneous load data, including:

[0006] Obtain heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification, and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city;

[0007] If the verification passes, the daily load data of non-IoT users and the total load of the residential community are determined based on the heterogeneous load data. The heating load gap data is then compensated using the total load data to obtain the heating load data. Non-heating load data is then extracted from the total load data. Based on the non-heating load data, the daily load data, and the heating load data, a comprehensive statistical data set is generated. The heating load gap data is the load gap caused by IoT devices being offline or not deployed.

[0008] Based on the rate of change of current flow data and adjacent flow data, the heterogeneous load data is processed to obtain processed data; the current flow data and adjacent flow data are the data corresponding to the current time node and its adjacent time nodes read from the flow meter of the heterogeneous load data.

[0009] An improved time linear interpolation algorithm is used to generate a virtual meter sequence based on target flow data; the target flow data is data read from the flow metering instrument based on the processed data at a standard time.

[0010] The daily load value is determined using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters for the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The full-caliber statistical data and the total daily load value are then stored.

[0011] Optionally, the step of acquiring heterogeneous load data from different sources and performing multi-level verification on the heterogeneous load data includes:

[0012] Establish connections with heterogeneous databases; the heterogeneous databases include IoT data from the IoT online operating system and non-IoT data from the business billing system;

[0013] Heterogeneous load data from different sources are obtained from the heterogeneous database, and the heterogeneous load data is verified at multiple levels.

[0014] If multi-level validation fails, determine the current validation count;

[0015] If the current number of checks is less than the check count threshold, then the multi-level check process is repeated.

[0016] If the current number of verifications is not less than the verification number threshold, then the heterogeneous load data is populated, and then the process of determining the daily load data of non-IoT users and the total load of the entire residential community is executed.

[0017] Optionally, determining the daily load data of non-IoT users and the total load of the residential community based on the heterogeneous load data includes:

[0018] Filter out the billing data and corresponding number of days for non-IoT users from heterogeneous load data;

[0019] The daily load data of non-IoT users is determined based on the billing data and the number of days.

[0020] Using existing IoT devices in residential communities as statistical samples, corresponding online gas consumption data and the total number of IoT users are selected from the heterogeneous load data. Based on the online gas consumption data and the total number of IoT users, the total load of the residential community is determined.

[0021] Optionally, processing the heterogeneous load data based on the rate of change of traffic between the current traffic data and adjacent traffic data includes:

[0022] Read current flow data and adjacent flow data from flow metering instruments that handle heterogeneous load data;

[0023] Calculate the rate of change of traffic between the current traffic data and the adjacent traffic data;

[0024] Determine whether the flow rate change meets the verification criteria and data replacement conditions;

[0025] If the flow rate change rate meets the verification and data replacement conditions, the heterogeneous load data is processed based on the current flow data and adjacent flow data to obtain the processed data.

[0026] Optionally, the process of processing the heterogeneous load data based on current traffic data and adjacent traffic data to obtain processed data includes:

[0027] New traffic data is calculated based on adjacent traffic data;

[0028] Replace the current traffic data with the new traffic data to obtain the replaced traffic data;

[0029] The replaced traffic data is subjected to bidirectional abnormal data verification and cleaning to obtain processed data.

[0030] Optionally, the step of determining the daily load value using the virtual meter sequence, and if the daily load value is less than a threshold value, then the daily load values ​​of all flow meters for the same user are summed and floating-point numbers are eliminated, including:

[0031] First-order difference is performed on the virtual instrument sequence to calculate the daily load value;

[0032] The threshold value shall be determined according to the heating season rules;

[0033] If the daily load value is less than the threshold value, the daily load values ​​of all flow meters for the same user are summed and floating-point numbers are eliminated.

[0034] If the daily load value is greater than the threshold value, forced circuit breaker protection is executed, and the daily load value is set to null. Then, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated.

[0035] Optionally, storing the full-caliber statistical data and the total daily load value includes:

[0036] Add confidence labels and timestamps to the total daily load value;

[0037] The full-caliber statistical data, total daily consumption load value, and corresponding confidence level labels and timestamps are stored in a database; the database includes a structured database or a time series database.

[0038] The threshold value is dynamically updated based on the data in the database;

[0039] Establish a standard data service interface so that downstream systems can access data from the database through the standard data service interface.

[0040] Secondly, this application discloses a cleaning apparatus for multi-source heterogeneous load data, comprising:

[0041] A multi-level verification module is used to acquire heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification, and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city;

[0042] The full-caliber statistical data generation module is used to determine the daily load data of non-IoT users and the total load of the residential community based on the heterogeneous load data if the verification passes. It then uses the total load to compensate for the heating load gap to obtain heating load data, extracts non-heating load data from the total load, and generates full-caliber statistical data based on the non-heating load data, the daily load data, and the heating load data. The heating load gap is caused by IoT devices being offline or not deployed.

[0043] The data processing module is used to process the heterogeneous load data based on the rate of change of the current flow data and the adjacent flow data to obtain processed data; the current flow data and the adjacent flow data are the data corresponding to the current time node and its adjacent time nodes read from the flow meter of the heterogeneous load data.

[0044] A virtual meter sequence generation module is used to generate a virtual meter sequence based on target flow data using an improved time linear interpolation algorithm; the target flow data is data read from the flow metering instrument based on the processed data at a standard time.

[0045] The storage module is used to determine the daily load value using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The module also stores the full-caliber statistical data and the total daily load value.

[0046] Thirdly, this application discloses an electronic device, including:

[0047] Memory, used to store computer programs;

[0048] A processor is used to execute the computer program to implement the aforementioned method for cleaning multi-source heterogeneous load data.

[0049] Fourthly, this application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned method for cleaning multi-source heterogeneous load data.

[0050] As can be seen, this application provides a method for cleaning multi-source heterogeneous load data, including acquiring heterogeneous load data from different sources and performing multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification, and legality verification; the heterogeneous load data is load data related to lifeline facilities in the target city; if the verification passes, the daily load data of non-IoT users and the total load of residential communities are determined based on the heterogeneous load data, and the heating gap load data is compensated using the total load to obtain heating load data; non-heating load data is extracted from the total load, and full-scope statistical data is generated based on the non-heating load data, the daily load data, and the heating load data; the heating gap load data is due to IoT devices being offline or not deployed. The resulting load gap data; based on the rate of change of flow between the current flow data and adjacent flow data, the heterogeneous load data is processed to obtain processed data; the current flow data and adjacent flow data are the data corresponding to the current time node and its adjacent time node read from the flow meter of the heterogeneous load data; a virtual meter sequence is generated based on the target flow data using an improved time linear interpolation algorithm; the target flow data is the data read from the flow meter of the processed data based on standard time; the daily load value is determined using the virtual meter sequence; if the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value, and the full-caliber statistical data and the total daily load value are stored.This application performs multi-level verification on heterogeneous load data from different sources, effectively intercepting data caused by network jitter, weak base station signals, or packet distortion, ensuring the continuity and reliability of data collection in complex network environments. If the verification passes, the daily load data of non-IoT users and the total load of residential communities are determined based on the heterogeneous load data. The total load is used to compensate for the heating load gap, resulting in heating load data. Non-heating load data is then extracted from the total load. Based on the non-heating load data, daily load data, and heating load data, comprehensive statistical data is generated. This solves the problem of non-IoT users being "invisible" in real-time systems and the statistical gap in residential load caused by equipment offline, achieving comprehensive closed-loop coverage of gas load and eliminating statistical blind spots in the data base. It also addresses the relationship between current flow data and adjacent flow data. The flow rate change rate is used to process the heterogeneous load data to obtain processed data. An improved time linear interpolation algorithm is used to generate a virtual meter sequence based on the target flow data, eliminating spatial superposition deviation and interpolation pseudo-noise caused by time misalignment, ensuring accurate alignment and aggregation of multi-source data in the spatiotemporal dimension. The daily load value is determined using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The full-caliber statistical data and the total daily load value are stored to ensure normal data flow through the system. At the same time, it effectively resists dirty data crossing caused by network jitter or abnormal message format and successfully reaches the main process, ensuring the continuous operation of the load prediction main process under extreme communication conditions and preventing system downtime in the event of partial failure. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a flowchart of a method for cleaning multi-source heterogeneous load data disclosed in this application;

[0053] Figure 2 This application discloses a flowchart of multi-level verification, exponential backoff retry, and circuit breaker protection for heterogeneous load data.

[0054] Figure 3 This application discloses a flowchart for implementing full-caliber closed-loop statistics;

[0055] Figure 4This application discloses an overall processing flowchart for cleaning multi-source heterogeneous load data;

[0056] Figure 5 This is a schematic diagram of the structure of a cleaning device for multi-source heterogeneous load data disclosed in this application;

[0057] Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Currently, the data sources for energy load forecasting systems mainly include three categories: real-time reported data from IoT (Internet of Things) meters, non-IoT user data based on monthly bills, and aggregated data from residential communities. Due to differences in the hardware types of data collection terminals, communication protocols, and deviations in business flow rules, the raw energy load data exhibits multi-source heterogeneity and physical-logical discontinuity. Load data acquisition and preprocessing still primarily rely on local monitoring and basic statistics: For industrial and commercial users with installed IoT terminals, the operation and maintenance system uses the SCADA (Supervisory Control and Data Acquisition) system to retrieve instantaneous and cumulative traffic in real time, and sets simple upper and lower thresholds for alarms; for non-IoT existing users without installed sensors, the system mainly relies on manual monthly meter reading and entry into the business system, forming offline static billing data; the existing accounting method is based on the estimation method of the community-level master meter, which does not take into account the proportion of online and offline users and the actual situation that the proportion may fluctuate. When the collected data has missing values, the various algorithms used will automatically judge it as a null value or a zero value, which will cause the statistical accuracy to deviate and affect the reliability of the data source. As can be seen from the above, how to achieve seamless coverage and closed-loop management of all customers, eliminate statistical blind spots in the data base, effectively solve the problem of insufficient accuracy of basic data in energy dispatching scenarios, and improve the physical consistency and logical accuracy of data cleaning are problems that need to be solved in this field.

[0060] See Figure 1 As shown in the figure, this invention discloses a method for cleaning multi-source heterogeneous load data, which may specifically include:

[0061] Step S11: Obtain heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city.

[0062] In this embodiment, a connection is established with a heterogeneous database. The heterogeneous database includes IoT data from the IoT online operating system and non-IoT data from the business billing system. Heterogeneous load data from different sources is obtained from the heterogeneous database, and multi-level verification is performed on the heterogeneous load data. If the multi-level verification fails, the current number of verifications is determined. If the current number of verifications is less than the verification threshold, the multi-level verification process is repeated. If the current number of verifications is not less than the verification threshold, the heterogeneous load data is populated, and then the process of determining the daily load data of non-IoT users and the total load of the entire residential community is executed.

[0063] In this embodiment, the system first accesses a heterogeneous database, which includes, but is not limited to, IoT data from the IoT online operation system and non-IoT data from the business billing system. Considering the uncertainty of the network communication environment and the potential distortion of data packets during the online access process, a backoff and retry mechanism with logical circuit breaking functionality is established at the data access layer. The process of multi-level verification of heterogeneous load data, exponential backoff and retry, and circuit breaking protection is as follows: Figure 2 As shown, the specific steps are as follows:

[0064] 1. Multi-level verification: After the system initiates a data request, it first performs HTTP (Hypertext Transfer Protocol) layer status code verification, then performs business content logic verification and legality verification. If the heterogeneous data message contains the previously configured geographical or business identifier keyword "partner", the data is determined to be illegal data, completely blocking the situation where the server returns abnormal and malformed messages.

[0065] 2. Loop retry strategy with backoff mechanism: Set a threshold for the number of verification attempts. N The system performs three checks. If a single check fails, it checks if the current check count is less than the check count threshold. If it is, the system executes a step-by-step exponential sleep (e.g., ...). After a request is triggered again, the multi-level verification process is repeated to avoid data interruptions caused by momentary network jitter. The judgment logic formula is as follows:

[0066] ;

[0067] in, Indicates whether the request was successful. This indicates that the attempt was unsuccessful.

[0068] 3. Safety Circuit Breaker Mechanism: If the current number of verifications is not less than the verification threshold after multiple retries, the system will enter a logical circuit breaker. At this time, the system will use a pre-set default strategy method to fill in the data, such as filling in the average of the previous actual data or directly using 0, to avoid the overall scheduling "avalanche" caused by local data problems. Then, the system will execute the process of determining the daily load data of non-IoT users and the total load of the entire residential community.

[0069] This application employs multi-level verification logic to identify and intercept data packet distortion caused by weak base station signals and low battery voltage, and uses exponential backoff retries to ensure the continuity of high-frequency sampling under complex operating conditions.

[0070] It is worth noting that the heterogeneous load data proposed in this application is load data related to the lifeline facilities of the target city. In other words, the technical solution of this application can be applied not only to the field of urban gas technology, but also to the field of urban lifeline technology, including but not limited to water, heat and gas data and gas pipeline network data.

[0071] Step S12: If the verification passes, determine the daily load data of non-IoT users and the total load of the residential community based on the heterogeneous load data. Use the total load to compensate for the heating gap load data to obtain the heating load data. Remove the non-heating load data from the total load. Generate full-caliber statistical data based on the non-heating load data, the daily load data, and the heating load data. The heating gap load data is the gap load data caused by IoT devices being offline or not deployed.

[0072] In this embodiment, billing data and corresponding days for non-IoT users are filtered from heterogeneous load data; daily load data for non-IoT users are determined based on the billing data and the number of days; existing IoT devices in the residential community are used as statistical samples to filter corresponding online gas consumption data and the total number of IoT users from the heterogeneous load data; and the total load of the residential community is determined based on the online gas consumption data and the total number of IoT users.

[0073] Constrained by the low visibility of non-IoT users and the lack of residential community data in the natural gas industry, this application constructs a comprehensive gas heterogeneous mapping method to eliminate the aforementioned statistical blind spots. Taking urban gas as a specific example, the process is as follows: First, a blacklist removal method is used to exclusively determine the number of existing non-IoT users, and the monthly billing is calculated on a daily basis using the nearest-validity principle; second, considering the residential load at the community level, an online status correction logic is introduced, and a compensation algorithm is used to correct the accurate heating load gap. The process of achieving comprehensive closed-loop statistics is as follows: Figure 3 As shown, the specific steps are as follows:

[0074] For non-IoT users: This application utilizes a "blacklist removal" mechanism to isolate real-time terminals marked with IoT attributes from the full usage table, accurately identifying non-IoT users who only have monthly bills. The system automatically traces the valid historical records within the last 3 months to obtain the billing data of IoT users, and extracts the monthly total from the billing data according to the "most recent valid principle". And combine it with calendar functions to calculate the number of natural days in that month. Daily load data for non-IoT users was obtained. : .

[0075] For the total physical load of residential communities: For residential communities with partial IoT sensing capabilities, the total physical load of the community is calculated using the existing IoT devices in the community as a statistical sample. The calculation formula is as follows:

[0076] ;

[0077] in, and These represent the total gas consumption of both online heating and non-heating IoT users within the community. This represents the total number of IoT users currently online in the community. This represents the total number of registered households in the community.

[0078] For heating load data: To eliminate the problem of data statistical bias caused by communication interruptions or lack of IoT, a heating gap compensation method is introduced to calculate the heating load data. The calculation formula is as follows:

[0079] ;

[0080] in, For heating load data, To measure the basic heating load, The total number of users in residential communities using heating. The total number of online users in residential communities that are connected to the Internet of Things and have heating systems. This represents the average gas consumption of users in residential communities that are connected to the Internet of Things and use heating systems. ;

[0081] For the total load, this application proposes to separate heating and non-heating load data using the residual method: the non-heating load data of real residential kitchen heating is extracted from the total load. Through the logic of double correction of sample mean extrapolation and physical gap compensation, the gas consumption of residential use and heating can be accurately separated. Finally, based on the non-heating load data, daily load data and heating load data, the total statistical data is generated to achieve closed-loop statistics.

[0082] Step S13: Based on the rate of change of the current flow data and the adjacent flow data, process the heterogeneous load data to obtain processed data; the current flow data and the adjacent flow data are the data corresponding to the current time node and its adjacent time nodes read from the flow meter of the heterogeneous load data.

[0083] In this embodiment, current flow data and adjacent flow data are read from the flow metering instrument of heterogeneous load data; the flow change rate between the current flow data and the adjacent flow data is calculated; it is determined whether the flow change rate meets the verification judgment and data replacement conditions; if the flow change rate meets the verification judgment and data replacement conditions, the heterogeneous load data is processed based on the current flow data and adjacent flow data to obtain processed data.

[0084] The heterogeneous load data is processed based on the current traffic data and adjacent traffic data to obtain processed data, including: calculating new traffic data based on adjacent traffic data; replacing the current traffic data with the new traffic data to obtain replaced traffic data; and performing bidirectional abnormal data verification and cleaning on the replaced traffic data to obtain processed data.

[0085] For gas flow accumulation data, the physical law that "cumulative flow can only increase monotonically" is transformed into a hard constraint on the algorithm. This application proposes an improved verification judgment and data replacement condition: for instantaneous drops in bottoming out caused by signal interference (such as a change process of going up and down from 526K→155K→529K), a new physical judgment logic is adopted, taking the drop difference between the two sides of the current position as the judgment. If a drop occurs (assuming the current observation point is t), it is considered that the point is caused by signal interference. According to the above condition, this point is marked as caused by signal interference, that is, the value of this point is set to None (empty value), and a new value is obtained by smoothing the replacement using the average value of its neighbors before and after.

[0086] Understandably, taking natural gas as an example, this step reads the current flow rate data corresponding to the current time point from the gas meter, as well as the flow rate data of the previous and next time points adjacent to the current time point. Using the flow rate data of the previous and next time points as adjacent flow rate data, it determines whether the rate of change of flow rate between the current flow rate data and the flow rate data of the next time point is greater than a flow rate change rate threshold. Furthermore, it determines whether the rate of change of flow rate between the current flow rate data and the flow rate data of the previous time point is greater than a flow rate change rate threshold. If both are greater, the verification and data replacement conditions are satisfied. Then, new flow rate data is calculated based on the flow rate data of the previous and next time points, and the current flow rate data is smoothly replaced with the new flow rate data. The improved formula for the verification and data replacement conditions is as follows:

[0087] ;

[0088] in, This refers to the current traffic data corresponding to the current time point. This refers to the traffic data from the previous time point. For traffic data at the next time point, This is the threshold for the rate of change of flow.

[0089] For bidirectional outlier verification and cleaning: This application proposes a bidirectional monotonicity forced constraint method. This method is applied to all samples after the maximum value index in the Back Min (reverse monotonicity) reverse retrieval sequence to remove "high-order dead values" greater than the maximum value. Cummax (forward monotonicity) is used to sequentially determine the cumulative maximum value of the sequence. Under non-table-changing conditions, if the value of the current point is found to be significantly less than the historical maximum value, the point is identified as a "bottoming out" where the physical property rule has been broken and is removed. In addition, a sliding window outlier check is required: a 9-day sliding window dynamic median is used. If the difference between the current value and the median is greater than 20%, and the absolute value difference is huge, the data is considered outlier and cleaned to obtain the processed data.

[0090] To address the issue of false "V-shaped pit" drops in natural gas meters within industrial and factory settings caused by signal jitter, base station interference, or battery malfunctions, this application, building upon existing structured anomaly assessments, incorporates fluid mechanics experience into improved verification and data replacement conditions. Specifically, it utilizes a dual-sided relative drop judgment logic to determine whether a drop at a specific point within the cumulative flow sequence is an instantaneous decrease. Simultaneously, it combines the mandatory requirements of Back Min and Cummax to ensure that the base data consistently adheres to the fundamental physical law of monotonicity.

[0091] Step S14: Generate a virtual meter sequence based on the target flow data using an improved time linear interpolation algorithm; the target flow data is data read from the flow metering instrument based on the processed data at standard time.

[0092] Because the time reported by IoT devices is uncertain, this application proposes a standard time resampling alignment method based on time weight. Taking 08:00 as the standard time as an example, the resampling alignment logic of the 08:00 slice is used to make the full-caliber data spatially superimposed.

[0093] For example, starting from 08:00, the time distance between the two original observation points closest to the target time retrieved by the system is calculated, and the corresponding target flow data is obtained. Then, the virtual instrument data corresponding to this time is calculated using a weighting factor, as shown in the following formula:

[0094] ;

[0095] in, For virtual instrument data, The data is from the instrument data of the previous raw observation point closest to 08:00. The data is from the instrument data of the next raw observation point closest to 08:00. To and The corresponding timestamp, To and The corresponding timestamp.

[0096] If the distance between two observation points is greater than a set threshold (e.g., offline for 7 consecutive days), the interpolation operation is stopped to prevent spurious interpolation noise caused by over-extrapolation.

[0097] To address the challenges of random and unavoidable data collection times in natural gas IoT meters (such as large industrial and commercial meters and residential narrowband IoT meters) due to network congestion or power-saving strategies, this invention performs aligned resampling at a standard time (08:00). Functionally, it systematically finds the nearest neighbor observation points surrounding the target time, uses an improved time linear interpolation algorithm to calculate the virtual instrument data for that precise time, and supplements it with a threshold to prevent over-calculation and the introduction of spurious data.

[0098] Step S15: Determine the daily load value using the virtual instrument sequence. If the daily load value is less than the threshold value, then superimpose and eliminate floating-point values ​​of all flow metering instruments for the same user to obtain the total daily load value, and store the full-caliber statistical data and the total daily load value.

[0099] In this embodiment, a first-order difference is performed on the virtual meter sequence to calculate the daily load value; a threshold value is determined according to the heating season rules; if the daily load value is less than the threshold value, the daily load values ​​of all flow meters for the same user are superimposed and floating-point elimination is performed; if the daily load value is greater than the threshold value, forced circuit breaker protection is executed, and the daily load value is set to null. Then, the daily load values ​​of all flow meters for the same user are superimposed and floating-point elimination is performed to obtain the total daily load value, and a confidence label and timestamp are added to the total daily load value; the full-caliber statistical data, the total daily load value, and the corresponding confidence label and timestamp are stored in a database; the database includes a structured database or a time series database; the threshold value is dynamically updated based on the data in the database; a standard data service interface is established so that downstream systems can call the data in the database through the standard data service interface.

[0100] This application realizes the conversion from cumulative virtual meter data to daily load value, and introduces a safety circuit breaker mechanism to deal with unpurified sudden data: the virtual meter sequence is subjected to first-order difference to calculate the daily load value, the formula is: daily load value = flow meter data at 08:00 the next day - flow meter data at 08:00 on the current day;

[0101] Depending on the heating season, the system automatically selects the daily usage threshold based on the heating season rules. If the daily load exceeds the threshold (e.g., 50,000 cubic meters), it indicates that the water cannot be completely washed, and the system will execute forced circuit breaker protection and set it to None (empty value). The threshold setting formula is:

[0102] ;

[0103] in, For threshold value, This is the threshold value for the non-heating season. This is the threshold value for the heating season.

[0104] Then, the daily load values ​​of all flow meters for the same user are summed up.

[0105] If the daily load value is less than the threshold value, the daily load values ​​of all flow meters for the same user are directly summed and floating-point numbers are eliminated. Floating-point elimination means setting the result of the minimum value to zero, eliminating some minor interference caused by floating-point calculation, and finally obtaining the total daily load value.

[0106] The process of data storage and application integration is as follows: Figure 3As shown, for structured persistent storage: full-caliber load time-series data (including but not limited to timestamps, user identifiers, and gas volume values) are stored in a structured database or time-series database, and confidence labels are added to this data for later fault tracing and model iteration.

[0107] Furthermore, a unified standard data service interface is established. Downstream call scheduling (such as the "intelligent peak shaving system") needs to send a call request to the external cleaning engine. The request is then read from the table result set of the time slice of the "intelligent peak shaving system". All departments use the same logic to call and land on the load base.

[0108] This application can extract the distribution of anomalies from historical cleaning records at certain time intervals, and update the threshold value accordingly to the threshold size of the next scanning window. This continuous feedback optimization allows the cleaning engine to adjust the accuracy of the cleaning algorithm at any time according to the different load characteristics of each city.

[0109] In addition, this application can also perform traditional manual investigation and offline statistics. The operation and maintenance personnel obtain the monthly total of industrial and commercial households and residential communities through the meter reading cycle and attempt to manually align the daily readings of IoT terminals with it. Based on a single statistical general method, the system determines the warning threshold after the fixed flow rate upper and lower limit warning points and the smooth deletion of data after it exceeds the range (such as simple moving average method, single threshold filtering, etc.).

[0110] This step enables various dispatching business departments to make collaborative decisions based on a unified logical foundation and a unified load base. It also enables real-time perception of changes in the urban and industrial load characteristics of different cities through self-evolutionary feedback, dynamically adjusting the cleaning accuracy level, and providing reference for the allocation of long-term gas supply contracts during winter supply, peak winter gas consumption, and winter gas storage in summer. This improves the accuracy and safety of refined natural gas operation decisions.

[0111] In this embodiment, the overall processing flow for cleaning multi-source heterogeneous load data is as follows: Figure 4 As shown, the steps are as follows:

[0112] (1) Acquisition and initial screening of heterogeneous load data: Acquire heterogeneous load data from different sources, perform multi-level verification on the heterogeneous load data, and trigger exponential backoff retry and circuit breaker protection when the verification fails.

[0113] (2) Full-caliber closed-loop statistics: The full-caliber total load estimation of residential communities is carried out by applying blacklist elimination strategy, mean extrapolation, heating gap compensation and residual method to separate heating and non-heating load data, and complete the daily allocation of non-IoT households, restoration of residential community load and real-time mapping of IoT households.

[0114] (3) Processing heterogeneous load data: using improved verification judgment and data replacement conditions to intercept V-shaped signal pits, and simultaneously performing Back Min (reverse monotonicity) and Cummax (forward monotonicity) filtering to remove bottom anomalies; using the time-weighted standard time resampling alignment method to find the original observation point closest to the standard time, and using time-weighted linear interpolation technology to reduce the random sampling points to the daily standard time slice.

[0115] (4) Differential decomposition and multi-dimensional dynamic fuse protection: Calculate the daily load value and perform full-caliber spatial superposition. At the same time, determine the threshold value based on the heating season rules. If the daily load value is less than the threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated. If the daily load value is greater than the threshold value, the forced fuse protection is performed and the daily load value is set to null. Then the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated.

[0116] (5) Data storage and application integration: Store the full-caliber statistical data, total daily load value and corresponding confidence labels and timestamps in the database, and send them to the application system through the API (Application Programming Interface) interface and automatically update the threshold value according to the feedback.

[0117] The technical advantages of this application are as follows: It achieves full-caliber closed-loop coverage and "visibility" reconstruction of gas load: This invention effectively solves the problem of existing non-IoT users being "hidden" in the real-time system and the statistical gap caused by equipment offline in residential areas through a "blacklist removal" mechanism and a "heating gap compensation algorithm" for residential communities. It eliminates statistical blind spots in the data base, providing a complete logical benchmark for full-caliber peak-shaving prediction; and significantly improves the physical consistency and logical accuracy of data cleaning: It transforms the physical characteristic of "monotonically increasing cumulative flow" into a hard constraint of the algorithm, using a two-sided drop test and a back-end check. The Min&Cummax logic eliminates non-physical signals, accurately stripping away signal interference (such as V-shaped deep pit anomalies) that is submerged in the instantaneous signal interference of normal load fluctuations, identifying the actual flow rate drop or the illusory signal bottoming out, which can greatly reduce the "false kill rate" of abnormal operating conditions; it ensures high alignment and accuracy reshaping of heterogeneous data in the spatiotemporal dimensions: it introduces a time-weighted linear interpolation method to solve the problem of load superposition offset caused by randomized IoT terminal sampling time, and avoids interpolation pseudo noise caused by time misalignment by unifying heterogeneous data to a standard time slice (08:00 every day); it ensures that the spatiotemporal multi-dimensional load aggregation at the same moment has good consistency in both time and space dimensions; it enhances the system robustness and fault tolerance circuit breaking capability in industrial application scenarios: this invention integrates multi-level verification and retry logic based on backoff at the access layer, and combines A data security firewall is implemented by setting different threshold values ​​for different seasons. While ensuring the normal data flow of the system, it effectively resists the passage of dirty data caused by network jitter or abnormal packet format, and ensures that the data successfully reaches the main process. This guarantees the continuous operation of the load forecasting main process under extreme communication conditions and prevents the system from crashing during partial failures. It optimizes the decision-making accuracy of intelligent scheduling and improves the operational efficiency of enterprises. By using a physically self-consistent and comprehensive data foundation, it directly optimizes the source data and inputs it into the downstream load forecasting model. It also has functions such as accurate decomposition of residential heating and full-caliber load aggregation, which helps to reduce the risk of deviation in gas source procurement and balance scheduling for enterprises. Secondly, it has the effect of helping enterprises save energy and reduce consumption, and smooth peak and valley. At the same time, it is also of great significance for ensuring the security of urban energy supply and effectively realizing economic and social benefits.

[0118] In this embodiment, heterogeneous load data from different sources is acquired, and multi-level verification is performed on the heterogeneous load data. The multi-level verification includes status code verification, business content logic verification, and legality verification. The heterogeneous load data is load data related to the lifeline facilities of the target city. If the verification passes, the daily load data of non-IoT users and the total load of residential communities are determined based on the heterogeneous load data. The total load is used to compensate for the heating gap load data to obtain heating load data. Non-heating load data is extracted from the total load. Based on the non-heating load data, the daily load data, and the heating load data, a comprehensive statistical data set is generated. The heating gap load data is the gap load data caused by IoT devices being offline or not deployed. The heterogeneous load data is processed based on the rate of change of current flow data and adjacent flow data to obtain processed data. The current flow data and adjacent flow data are the data corresponding to the current time node and its adjacent time node read from the flow meter of the heterogeneous load data. A virtual meter sequence is generated based on the target flow data using an improved time linear interpolation algorithm. The target flow data is the data read from the flow meter of the processed data based on standard time. The daily load value is determined using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The full-caliber statistical data and the total daily load value are stored.This application performs multi-level verification on heterogeneous load data from different sources, effectively intercepting data caused by network jitter, weak base station signals, or packet distortion, ensuring the continuity and reliability of data collection in complex network environments. If the verification passes, the daily load data of non-IoT users and the total load of residential communities are determined based on the heterogeneous load data. The total load is used to compensate for the heating load gap, resulting in heating load data. Non-heating load data is then extracted from the total load. Based on the non-heating load data, daily load data, and heating load data, comprehensive statistical data is generated. This solves the problem of non-IoT users being "invisible" in real-time systems and the statistical gap in residential load caused by equipment offline, achieving comprehensive closed-loop coverage of gas load and eliminating statistical blind spots in the data base. It also addresses the relationship between current flow data and adjacent flow data. The flow rate change rate is used to process the heterogeneous load data to obtain processed data. An improved time linear interpolation algorithm is used to generate a virtual meter sequence based on the target flow data, eliminating spatial superposition deviation and interpolation pseudo-noise caused by time misalignment, ensuring accurate alignment and aggregation of multi-source data in the spatiotemporal dimension. The daily load value is determined using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The full-caliber statistical data and the total daily load value are stored to ensure normal data flow through the system. At the same time, it effectively resists dirty data crossing caused by network jitter or abnormal message format and successfully reaches the main process, ensuring the continuous operation of the load prediction main process under extreme communication conditions and preventing system downtime in the event of partial failure.

[0119] See Figure 5 As shown in the figure, an embodiment of the present invention discloses a cleaning device for multi-source heterogeneous load data, which may specifically include:

[0120] The multi-level verification module 11 is used to acquire heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city.

[0121] The full-caliber statistical data generation module 12 is used to determine the daily load data of non-IoT users and the total load of the residential community based on the heterogeneous load data if the verification passes; to compensate for the heating gap load data using the total load data to obtain the heating load data; to separate the non-heating load data from the total load data; and to generate full-caliber statistical data based on the non-heating load data, the daily load data, and the heating load data; the heating gap load data is the gap load data caused by the offline or non-deployed IoT devices.

[0122] Data processing module 13 is used to process the heterogeneous load data based on the rate of change of the current flow data and the adjacent flow data to obtain processed data; the current flow data and the adjacent flow data are the data corresponding to the current time node and the adjacent time node read from the flow meter of the heterogeneous load data.

[0123] The virtual meter sequence generation module 14 is used to generate a virtual meter sequence based on target flow data using an improved time linear interpolation algorithm; the target flow data is data read from the flow metering instrument based on the processed data at a standard time.

[0124] The storage module 15 is used to determine the daily load value using the virtual meter sequence. If the daily load value is less than the threshold value, the daily load values ​​of all flow metering instruments of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The storage module 15 stores the full-caliber statistical data and the total daily load value.

[0125] In some specific embodiments, the multi-level verification module 11 may specifically include:

[0126] The connection establishment module is used to establish connection relationships with heterogeneous databases; the heterogeneous databases include IoT data in the IoT online operation system and non-IoT data in the business billing system;

[0127] The heterogeneous load data acquisition module is used to acquire heterogeneous load data from different sources from the heterogeneous database and perform multi-level verification on the heterogeneous load data.

[0128] The current verification determination module is used to determine the current number of verification attempts if multiple levels of verification fail.

[0129] The repeat execution module is used to repeat the multi-level verification process if the current number of verifications is less than the verification count threshold.

[0130] The data filling module is used to fill the heterogeneous load data if the current number of verifications is not less than the verification number threshold, and then execute the process of determining the daily load data of non-IoT users and the total load of the entire residential community.

[0131] In some specific embodiments, the full-caliber statistical data generation module 12 may specifically include:

[0132] The billing data and corresponding number of days filtering module is used to filter out the billing data and corresponding number of days for non-IoT users from heterogeneous load data;

[0133] The daily load data determination module is used to determine the daily load data of non-IoT users based on the billing data and the number of days.

[0134] The total load determination module is used to select the corresponding online gas consumption data and the total number of IoT users from the heterogeneous load data using the existing IoT devices in the residential community as statistical samples, and determine the total load of the residential community based on the online gas consumption data and the total number of IoT users.

[0135] In some specific embodiments, the data processing module 13 may specifically include:

[0136] The data reading module is used to read current flow data and adjacent flow data from flow metering instruments that handle heterogeneous load data.

[0137] The flow rate change calculation module is used to calculate the flow rate change between the current flow data and the adjacent flow data;

[0138] The judgment module is used to determine whether the flow rate change meets the verification judgment and data replacement conditions;

[0139] The processed data determination module is used to process the heterogeneous load data based on the current flow data and adjacent flow data if the flow rate change rate meets the verification judgment and data replacement conditions, so as to obtain the processed data.

[0140] In some specific embodiments, the data processing module 13 may specifically include:

[0141] The new traffic data calculation module is used to calculate new traffic data based on adjacent traffic data;

[0142] The data replacement module is used to replace the current traffic data with the new traffic data to obtain the replaced traffic data;

[0143] The bidirectional abnormal data verification and cleaning module is used to perform bidirectional abnormal data verification and cleaning on the replaced traffic data to obtain processed data.

[0144] In some specific embodiments, the storage module 15 may specifically include:

[0145] The first-order difference module is used to perform first-order difference on the virtual instrument sequence to calculate the daily load value;

[0146] The threshold value determination module is used to determine the threshold value according to the heating season rules;

[0147] The load value superposition and floating-point elimination module is used to superimpose and eliminate the daily load values ​​of all flow metering instruments of the same user if the daily load value is less than the threshold value.

[0148] The forced circuit breaker protection module is used to execute forced circuit breaker protection if the daily load value is greater than the threshold value, set the daily load value to null, and then superimpose and eliminate the floating point values ​​of all flow meters of the same user.

[0149] In some specific embodiments, the storage module 15 may specifically include:

[0150] The tag and timestamp addition module is used to add confidence tags and timestamps to the total daily load value;

[0151] The data, tag, and timestamp storage module is used to store full-caliber statistical data, total daily load values, and corresponding confidence level tags and timestamps into a database; the database includes a structured database or a time series database.

[0152] The dynamic update module is used to dynamically update the threshold value based on the data in the database;

[0153] The interface establishment module is used to establish a standard data service interface so that downstream systems can access data in the database through the standard data service interface.

[0154] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multi-source heterogeneous load data cleaning method disclosed in any of the foregoing embodiments.

[0155] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0156] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0157] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the multi-source heterogeneous load data cleaning method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the multi-source heterogeneous load data cleaning device from external devices, as well as data collected by its own input / output interface 25.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the steps of the method for cleaning multi-source heterogeneous load data disclosed in any of the foregoing embodiments.

[0160] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The above provides a detailed description of the cleaning method, apparatus, equipment, and storage medium for multi-source heterogeneous load data provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for cleaning multi-source heterogeneous load data, characterized in that, include: Obtain heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; the multi-level verification includes status code verification, business content logic verification, and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city; If the verification passes, the daily load data of non-IoT users and the total load of the residential community are determined based on the heterogeneous load data. The heating load gap data is then compensated using the total load data to obtain the heating load data. Non-heating load data is then extracted from the total load data. Based on the non-heating load data, the daily load data, and the heating load data, a comprehensive statistical data set is generated. The heating load gap data is the load gap caused by IoT devices being offline or not deployed. Based on the rate of change of traffic between the current traffic data and adjacent traffic data, the heterogeneous load data is processed to obtain processed data; The current flow data and adjacent flow data are the data corresponding to the current time node and its adjacent time nodes read from the flow metering instrument of the heterogeneous load data; A virtual meter sequence is generated using a time linear interpolation algorithm based on target flow data; The target flow data is data read from the flow metering instrument of the processed data based on standard time. The daily load value is determined using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters for the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The full-caliber statistical data and the total daily load value are then stored.

2. The method for cleaning multi-source heterogeneous load data according to claim 1, characterized in that, The process of acquiring heterogeneous load data from different sources and performing multi-level verification on the heterogeneous load data includes: Establish connections with heterogeneous databases; the heterogeneous databases include IoT data from the IoT online operating system and non-IoT data from the business billing system; Heterogeneous load data from different sources are obtained from the heterogeneous database, and the heterogeneous load data is verified at multiple levels. If multi-level validation fails, determine the current validation count; If the current number of checks is less than the check count threshold, then the multi-level check process is repeated. If the current number of verifications is not less than the verification number threshold, then the heterogeneous load data is populated, and then the process of determining the daily load data of non-IoT users and the total load of the entire residential community is executed.

3. The method for cleaning multi-source heterogeneous load data according to claim 1, characterized in that, The determination of daily load data for non-IoT users and total load across all residential communities based on the heterogeneous load data includes: Filter out the billing data and corresponding number of days for non-IoT users from heterogeneous load data; The daily load data of non-IoT users is determined based on the billing data and the number of days. Using existing IoT devices in residential communities as statistical samples, corresponding online gas consumption data and the total number of IoT users are selected from the heterogeneous load data. Based on the online gas consumption data and the total number of IoT users, the total load of the residential community is determined.

4. The method for cleaning multi-source heterogeneous load data according to claim 1, characterized in that, The process of processing the heterogeneous load data based on the rate of change of current traffic data and adjacent traffic data includes: Read current flow data and adjacent flow data from flow metering instruments that handle heterogeneous load data; Calculate the rate of change of traffic between the current traffic data and the adjacent traffic data; Determine whether the flow rate change meets the verification criteria and data replacement conditions; If the flow rate change rate meets the verification and data replacement conditions, the heterogeneous load data is processed based on the current flow data and adjacent flow data to obtain the processed data.

5. The method for cleaning multi-source heterogeneous load data according to claim 4, characterized in that, The heterogeneous load data is processed based on current traffic data and adjacent traffic data to obtain processed data, including: New traffic data is calculated based on adjacent traffic data; Replace the current traffic data with the new traffic data to obtain the replaced traffic data; The replaced traffic data is subjected to bidirectional abnormal data verification and cleaning to obtain processed data.

6. The method for cleaning multi-source heterogeneous load data according to any one of claims 1 to 5, characterized in that, The step of determining the daily load value using the virtual meter sequence, and if the daily load value is less than a threshold value, then summing and eliminating floating-point values ​​for all flow meter values ​​of the same user, includes: First-order difference is performed on the virtual instrument sequence to calculate the daily load value; The threshold value shall be determined according to the heating season rules; If the daily load value is less than the threshold value, the daily load values ​​of all flow meters for the same user are summed and floating-point numbers are eliminated. If the daily load value is greater than the threshold value, forced circuit breaker protection is executed, and the daily load value is set to null. Then, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated.

7. The method for cleaning multi-source heterogeneous load data according to claim 6, characterized in that, The storage of the full-caliber statistical data and the total daily load value includes: Add confidence labels and timestamps to the total daily load value; The full-caliber statistical data, total daily consumption load value, and corresponding confidence level labels and timestamps are stored in a database; the database includes a structured database or a time series database. The threshold value is dynamically updated based on the data in the database; Establish a standard data service interface so that downstream systems can access data from the database through the standard data service interface.

8. A cleaning device for multi-source heterogeneous load data, characterized in that, include: A multi-level verification module is used to acquire heterogeneous load data from different sources and perform multi-level verification on the heterogeneous load data; The multi-level verification includes status code verification, business content logic verification, and legality verification; the heterogeneous load data is load data related to the lifeline facilities of the target city; The full-caliber statistical data generation module is used to determine the daily load data of non-IoT users and the total load of the residential community based on the heterogeneous load data if the verification passes. It then uses the total load to compensate for the heating load gap to obtain heating load data, extracts non-heating load data from the total load, and generates full-caliber statistical data based on the non-heating load data, the daily load data, and the heating load data. The heating load gap is caused by IoT devices being offline or not deployed. The data processing module is used to process the heterogeneous load data based on the rate of change of current traffic data and adjacent traffic data to obtain processed data; The current flow data and adjacent flow data are the data corresponding to the current time node and its adjacent time nodes read from the flow metering instrument of the heterogeneous load data; The virtual meter sequence generation module is used to generate virtual meter sequences based on target flow data using a time linear interpolation algorithm; The target flow data is data read from the flow metering instrument of the processed data based on standard time. The storage module is used to determine the daily load value using the virtual meter sequence. If the daily load value is less than a threshold value, the daily load values ​​of all flow meters of the same user are superimposed and floating-point numbers are eliminated to obtain the total daily load value. The module also stores the full-caliber statistical data and the total daily load value.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method for cleaning multi-source heterogeneous load data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the method for cleaning multi-source heterogeneous load data as described in any one of claims 1 to 7.

Images