A power marketing big data cleaning and feature extraction method and system

Abstract

The application relates to the technical field of electric power marketing data processing, and discloses an electric power marketing big data cleaning and feature extraction method and system. The method obtains high-frequency data flow of a to-be-analyzed transformer area, divides effective data sets and missing candidate sets according to physical sampling time labels, data arrival time labels and data collection cutoff time; generates a dynamic judgment threshold by fusing a global setting threshold, a clustering threshold, a local threshold and a historical sample confidence weight; generates a compensation type physical residual based on a transformer area total table, an auxiliary collection device and effective node power, and combines a joint attribution target function to distinguish physical power-off nodes and communication fault nodes; then maps node states into binary state labels, generates a double-end joint Boolean mask, and extracts effective time sequence segments; forms a constraint bandwidth based on a coarse-grained alignment path, and performs a dynamic time warping calculation in the constraint bandwidth to output a deviation degree value. The application can improve the accuracy of data cleaning and feature extraction.

Description

Technical Field

[0001] This invention relates to the field of power marketing data processing technology, specifically to a method and system for cleaning and feature extraction of big data in power marketing. Background Technology

[0002] With the application of smart meters, distribution transformer master meters, concentrators, and auxiliary data acquisition equipment in distribution transformer areas, the power marketing system can continuously acquire high-frequency data from both the user side and the distribution transformer side. This type of data can be used for scenarios such as distribution transformer operation analysis, load characteristic identification, topology verification, and abnormal power consumption judgment. Due to the large number of data acquisition nodes and the long data transmission links, data delays, data loss, communication interruptions, and partial power outages are prone to occur in actual operation. Therefore, it is necessary to clean and determine the status of the raw data before feature analysis.

[0003] Existing methods for cleaning electricity marketing data typically use successful data reception as the criterion for validity, directly marking unreceived data as missing data, or using historical averages or data from adjacent time points to fill in the gaps. These methods fail to adequately distinguish between physical sampling time and data arrival time, easily misclassifying late data as missing data. Furthermore, when nodes fail to report data, it is difficult to determine whether the failure is due to a physical power outage or a communication failure. In a physical power outage, a node's actual power consumption is zero, while in a communication failure, a node may still have actual load. Treating both situations uniformly as missing data would affect the results of power balance analysis and load feature extraction for distribution areas.

[0004] Furthermore, existing methods, when attributing missing states, do not adequately utilize the physical relationships between total power, auxiliary acquisition power, effective node power, and line loss within the same transformer area. When multiple nodes are simultaneously missing within the same calculation section, relying solely on historical node load or fixed thresholds for judgment is insufficient to adapt to operational differences across different transformer areas. In high-frequency time-series feature extraction, directly performing dynamic time warping calculations on the complete original sequence may result in communication fault segments participating in distance accumulation, and the large calculation range affects the efficiency and stability of deviation calculation results. Therefore, it is necessary to provide a method and system that can combine data arrival status, transformer area power relationships, and effective time-series segments for data cleaning and feature extraction. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for cleaning and extracting features from big data in power marketing. This method solves the problems of difficulty in distinguishing late data, power outage data, and communication anomaly data in the processing of high-frequency data collected in power marketing, inaccurate attribution of missing node status, and the high computational cost and susceptibility of high-frequency time-series feature extraction to interference from missing segments.

[0006] To address the above problems, the present invention provides the following technical solution: The first aspect of this invention provides a method for cleaning and feature extraction of big data in power marketing, the method comprising the following steps.

[0007] A data stream containing raw high-frequency data is acquired from the area to be analyzed. This data stream includes electrical quantity values ​​reported by nodes, as well as physical sampling time stamps and data arrival time stamps bound to these electrical quantity values. A late arrival waiting window is set for the calculation section to be computed. The time value of the calculation section is added to the time value of the late arrival waiting window to obtain the data collection deadline. When the system runtime reaches the data collection deadline, matching data records are extracted from the time series database using the physical sampling time stamp being equal to the calculation section and the data arrival time stamp being less than or equal to the data collection deadline as filtering conditions. Based on the full set of nodes in the area to be analyzed, nodes with matching data records are added to the valid dataset, and nodes without matching data records are added to the missing candidate set.

[0008] The above steps use the sampling occurrence time and data arrival time as the criteria for judgment, ensuring that the data collection of the calculation section corresponds to the actual sampling time. Data arriving within the late arrival waiting window is included in the valid range of the corresponding calculation section; data that has not arrived by the data collection deadline is included in the missing candidate set. This method reduces misjudgments of data validity caused by transmission delays.

[0009] Further, a global threshold, a clustering threshold based on transformer area feature parameters, and a local threshold based on historical healthy samples from the transformer area to be analyzed are obtained. The number of historical healthy samples from the transformer area to be analyzed within a preset historical time period is extracted. A local confidence weight coefficient is calculated based on this number of historical healthy samples, and the value 1 is subtracted from the local confidence weight coefficient to obtain a supplementary confidence weight coefficient. An empirical allocation parameter is then obtained, which determines the reference weights of the clustering threshold and the global threshold in the supplementary confidence weight coefficient. Finally, the local threshold, clustering threshold, and global threshold are weighted and fused using the local confidence weight coefficient, the supplementary confidence weight coefficient, and the reference weights to generate a dynamic judgment threshold.

[0010] In this threshold generation process, the local threshold reflects the historical operating status of the analyzed transformer area, the clustering threshold reflects the statistical characteristics of transformer areas of similar types, and the globally set threshold provides a unified judgment benchmark. When the number of historical healthy samples is large, the proportion of the local threshold in the fusion result increases; when the number of historical healthy samples is insufficient, the clustering threshold and the globally set threshold participate in supplementary judgment. Therefore, the dynamic judgment threshold can be adjusted according to the accumulation of historical samples of the transformer area, avoiding the problem of insufficient adaptation of a single fixed threshold to different transformer areas.

[0011] Furthermore, a compensatory physical residual is generated based on the power data from the transformer area master table, auxiliary acquisition equipment, and valid dataset. Specifically, this includes: extracting the input active power recorded in the transformer area master table and the compensatory input power reported by the auxiliary acquisition equipment; extracting the active power values ​​reported by each node in the valid dataset and summing them to obtain the total power consumption of nodes in normal acquisition and communication states; extracting the average line loss rate under historical normal operating conditions and multiplying the average line loss rate by the input active power to obtain the baseline line loss power; and subtracting the total power consumption of nodes in normal acquisition and communication states and the baseline line loss power from the sum of the input active power and the compensatory input power to obtain the compensatory physical residual.

[0012] Compensated physical residuals are used to characterize the power difference in the current calculation section that has not yet been explained by effective node power and line loss power. If nodes in the missing candidate set are in a physical power outage state, their objective power consumption is close to zero, and these nodes should not contribute to the historical load baseline power. If nodes in the missing candidate set are in a communication failure state, their actual power consumption may still exist, but it has not been reported normally. Therefore, compensated physical residuals can be used as a power basis to distinguish between physical power outages and communication failures.

[0013] Furthermore, the missing candidate set is divided into attribution categories based on a dynamic determination threshold. First, it is determined whether the absolute value of the compensated physical residual is less than or equal to the dynamic determination threshold. If this condition is met, it indicates that the effective dataset and line loss compensation can explain the input power of the transformer area, and the nodes in the missing candidate set are treated as being in a physical power outage state and assigned to the first node set. If the above condition is not met, the sum of the historical load baseline power of all nodes in the missing candidate set is calculated, and it is determined whether the absolute value of the difference between this historical load baseline power sum and the compensated physical residual is less than or equal to the dynamic determination threshold. If this condition is met, it indicates that the missing candidate set as a whole better reflects the situation where actual power consumption still exists under communication failure conditions, and the nodes in the missing candidate set are assigned to the second node set.

[0014] When neither of the above two boundary conditions is met, the absolute value of the difference between the sum of the historical load baseline power of the nodes to be assigned to the second node set and the compensated physical residual is used as the power deviation term, and a joint attribution objective function is constructed by combining it with the communication fault penalty weight. A dynamic programming algorithm is used to minimize the state combinations of nodes in the missing candidate set, and the nodes are assigned to either the first node set or the second node set based on the optimization result. The first node set corresponds to the physical power outage state, and the second node set corresponds to the communication fault state. This attribution process incorporates the power conservation relationship, historical load baseline, and communication fault penalty weight into the node state division, avoiding the direct determination of node anomalies based solely on missing data states.

[0015] Furthermore, the objective power consumption of nodes within the first node set at the corresponding computational cross-section is determined to be zero, and these nodes are included in the effective state for status label assignment. The binary status labels corresponding to nodes belonging to the effective dataset and nodes belonging to the first node set are assigned a value of 1; the binary status labels corresponding to nodes belonging to the second node set are assigned a value of 0. For the selected reference node and the child node to be analyzed within the analysis area, the binary status labels corresponding to both at each computational cross-section are extracted, and a bitwise logical AND operation is performed. When the status labels of both the reference node and the child node to be analyzed are 1, a two-ended joint Boolean mask with a value of 1 is generated; otherwise, a two-ended joint Boolean mask with a value of 0 is generated.

[0016] The aforementioned dual-ended joint Boolean mask is used to constrain the data source for subsequent feature extraction. Data from the same computational section is only included in subsequent time-series alignment calculations when both the reference node and the child node to be analyzed are in a valid state. For nodes experiencing physical power outages, their power consumption is treated as zero and retained as an objectively valid state; for nodes with communication failures, their missing data is not included in dynamic time warping calculations. This state mapping method allows for the differentiation between real power outage segments and communication anomaly segments before feature extraction.

[0017] Furthermore, based on multiple dual-ended joint Boolean masks within the analysis time window, the original sequences corresponding to the reference node and the original sequences corresponding to the child nodes to be analyzed are synchronously truncated, time axis gaps are removed, and the beginning and end are spliced ​​together to construct an effective reference time series sequence and an effective child node time series sequence. The effective reference time series sequence and the effective child node time series sequence constitute an effective time series segment.

[0018] After obtaining the valid time series segments, mean aggregation calculation is performed simultaneously on the valid reference time series sequence and the valid child node time series sequence according to the preset segmentation window length to generate a coarse-grained reference sequence and a coarse-grained child node sequence. Subsequently, a cumulative distance cost matrix is ​​constructed for the coarse-grained reference sequence and the coarse-grained child node sequence, and the minimum cumulative cost path corresponding to the cumulative distance cost matrix is ​​solved to obtain the coarse-grained alignment path.

[0019] Furthermore, the coarse-grained alignment path is mapped to a constraint bandwidth within the original high-frequency data computation space. Specifically, coarse-grained coordinate nodes are extracted from the coarse-grained alignment path, and each coarse-grained coordinate node is projected back into the original high-frequency data computation space of the effective time series segment according to its corresponding segmented window length. These nodes are then expanded into square matrix blocks and stitched together to form the projection path region. Next, according to a preset search radius, the boundary of the projection path region is extended outward along the horizontal and vertical coordinates of the computation space, and the extended coordinate set is truncated to obtain the constraint bandwidth.

[0020] Furthermore, dynamic time warping is performed within the original high-frequency data and constrained bandwidth of the effective time series segments. A local distance matrix is ​​constructed for the original high-frequency data of the effective time series segments, and the cumulative distance cost is recursively calculated only within the effective coordinate set covered by the constrained bandwidth. For coordinate points outside the constrained bandwidth, a distance cost with a preset maximum value is assigned to them so that they do not participate in effective path optimization. Finally, the global minimum cumulative distance cost to the endpoint of the matrix coordinates is calculated, and this global minimum cumulative distance cost is divided by the original sequence length corresponding to the effective time series segment. The normalized topological feature deviation is output as the deviation value.

[0021] The calculation process first uses a combined Boolean mask to eliminate invalid segments caused by communication failures, and then limits the search range of the original high-frequency dynamic time warping by using a coarse-grained alignment path. The dynamic time warping calculation is completed within the constrained bandwidth, which can reduce the computational cost of high-frequency sequence alignment and reduce the impact of abnormal communication segments on the deviation calculation.

[0022] A second aspect of this invention provides a power marketing big data cleaning and feature extraction system for performing the aforementioned power marketing big data cleaning and feature extraction method. The system includes terminal equipment, a communication network, and a main station cloud platform or edge computing gateway.

[0023] The main cloud platform or edge computing gateway is equipped with a data alignment module, a threshold generation module, a data attribution module, a mask generation module, and a deviation calculation module.

[0024] The data alignment module acquires a data stream containing raw high-frequency data from the area to be analyzed. Based on the physical sampling time stamp, data arrival time stamp, and the data collection deadline of the calculation section in the data stream, it divides the valid dataset into a missing candidate set. After the system runtime reaches the data collection deadline, the data alignment module jointly filters data records according to the physical sampling time stamp and the data arrival time stamp, and determines the nodes that have not successfully acquired data records based on the full set of nodes in the area to be analyzed.

[0025] The threshold generation module is used to obtain the global set threshold, clustering threshold, and local threshold, and calculate the local confidence weight coefficient based on the number of historical healthy samples in the area to be analyzed within a preset historical time period. The threshold generation module is also used to determine the reference weight of the clustering threshold and the global set threshold in the supplementary weights based on the supplementary confidence weight coefficient and empirical allocation parameters, and to perform weighted fusion of the local threshold, clustering threshold, and global set threshold to generate a dynamic judgment threshold.

[0026] The data attribution module is used to generate compensated physical residuals based on the input active power recorded in the transformer area master table, the compensated input power reported by the auxiliary acquisition equipment, the sum of active power of nodes in the effective dataset, and the baseline loss power. The data attribution module is also used to determine the boundary attribution conditions of the missing candidate set by combining a dynamic judgment threshold. When the boundary attribution conditions are not met, a joint attribution objective function including a power deviation term and a communication fault penalty weight is constructed. A dynamic programming algorithm is then used to optimize the state combinations of nodes in the missing candidate set, dividing the missing candidate set into a first node set and a second node set.

[0027] The mask generation module is used to determine the power consumption of nodes in the first node set at the corresponding calculation section as zero, and to assign the binary state label corresponding to the effective dataset and the first node set to 1, and to assign the binary state label corresponding to the second node set to 0. The mask generation module is also used to extract the binary state labels corresponding to the reference node and the child node to be analyzed, and to perform a bitwise logical AND operation on the two to generate a two-ended joint Boolean mask.

[0028] The deviation calculation module is used to synchronously extract effective time-series segments from the original high-frequency sequences of the reference node and the child node to be analyzed based on a dual-ended joint Boolean mask. The module also performs mean aggregation on the effective time-series segments to generate a coarse-grained sequence, solves for a coarse-grained alignment path based on the coarse-grained sequence, and maps the coarse-grained alignment path to a constraint bandwidth within the computational space of the original high-frequency data. Subsequently, the deviation calculation module performs dynamic time warping within the constraint bandwidth to obtain the global minimum cumulative distance cost, and normalizes it according to the original sequence length of the effective time-series segments to output the deviation value.

[0029] This invention provides a method and system for cleaning and feature extraction of big data in electricity marketing. It has the following beneficial effects: 1. This invention simultaneously introduces physical sampling time stamps, data arrival time stamps, and data collection deadlines into the data stream processing, dividing the node data within the computational cross-section into valid datasets and missing candidate sets. It can distinguish between normally arriving data, late data, and non-arriving data, avoiding reliance solely on whether data was successfully received for validity judgment, and improving the accuracy of high-frequency data cleaning results.

[0030] 2. This invention generates compensated physical residuals based on the total power of nodes in the transformer area table, auxiliary acquisition equipment, and valid dataset. It also combines dynamic judgment thresholds and joint attribution objective functions to classify the nodes in the missing candidate set into states. This invention can distinguish between physical power outage states and communication failure states, allowing power outage nodes to be included in the valid state with zero power consumption, and preventing communication failure nodes from participating in subsequent valid feature calculations, thereby reducing the impact of missing data on load analysis results.

[0031] 3. This invention extracts effective time segments of the reference node and the child node to be analyzed based on a dual-end joint Boolean mask, and generates a constraint bandwidth through coarse-grained alignment paths, performing dynamic time warping calculations within the constraint bandwidth. This reduces the search range of high-frequency sequence alignment calculations and minimizes the impact of communication fault segments on the time alignment results, thereby improving the stability and computational efficiency of deviation numerical calculations. Attached Figure Description

[0032] Figure 1 This is an architecture diagram of the power marketing big data cleaning and feature extraction system according to an embodiment of the present invention; Figure 2 This is a flowchart of the power marketing big data cleaning and feature extraction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the data extraction and window waiting mechanism according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the construction of the three-level judgment threshold in an embodiment of the present invention; Figure 5 This is a schematic diagram of the compensation-type physical residual calculation logic in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the construction of the joint attribution objective function according to an embodiment of the present invention; Figure 7 This is a comparison chart of ROC curves for the topology determination method according to an embodiment of the present invention; Figure 8 This is a space diagram for adaptive constraint bandwidth and fine-grained alignment path optimization in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] See Figure 1This invention provides a power marketing big data cleaning and feature extraction system, comprising: terminal equipment, a communication network, an edge computing gateway, and a main station cloud platform. The terminal equipment is deployed in the physical field of low-voltage distribution areas. The terminal equipment includes a main meter and a miniature synchronous phasor measurement device deployed on the transformer side of the distribution area, a smart IoT meter deployed on the user side, and auxiliary acquisition equipment deployed on the photovoltaic inverter and energy storage controller sides. The terminal equipment establishes a data connection with the edge computing gateway or the main station cloud platform through the communication network. The communication network includes one or more of a high-speed carrier network, a low-power wireless network, or a mobile cellular network. Each terminal equipment is internally equipped with a real-time clock unit. The terminal equipment synchronizes its clock through a main station broadcast time synchronization protocol or a network time synchronization protocol. The terminal equipment collects electrical quantity data at a set period. The electrical quantity data is uploaded to the edge computing gateway or the main station cloud platform via the communication network for aggregation and storage. The edge computing gateway or the main station cloud platform is internally configured with a data alignment module, a threshold generation module, a data attribution module, a mask generation module, and a deviation calculation module.

[0035] See Figure 2 This invention provides a method for cleaning and feature extraction of big data in electricity marketing, comprising the following steps: S100, the data alignment module receives the data stream reported by the terminal device. For each data record, the physical sampling time label and data arrival time label are used. At the set calculation section and the corresponding late waiting window deadline, the reported records of each node in the station area are traversed. Nodes with complete section data are included in the valid dataset, and nodes that have not obtained section data are included in the missing candidate set.

[0036] S200, the threshold generation module obtains the global set threshold, the clustering threshold based on the characteristic parameters of the transformer area, and the local threshold based on the historical healthy samples of the transformer area. It extracts the cumulative time length of the effective healthy samples in the transformer area to calculate the confidence weight. It uses the confidence weight to perform weighted fusion calculation on the global set threshold, the clustering threshold and the local threshold, and outputs the dynamic judgment threshold of the current section.

[0037] S300, the data attribution module extracts the power values ​​from the total table of the transformer area, auxiliary acquisition equipment and effective datasets, calculates the compensated physical residual of the section, constructs a joint attribution objective function with the compensated physical residual as a constraint, performs boundary extreme value judgment and outputs pruning results or solves the joint attribution objective function through dynamic programming algorithm, and determines the first set of nodes in the physical power outage state and the second set of nodes in the communication failure state in the missing candidate set based on the solution results.

[0038] S400, the mask generation module maps physical power outage state and communication failure state to corresponding binary state labels, extracts the state label of the sub-node to be analyzed and the state label of the set reference node, performs logical AND operation, and outputs the corresponding double-ended joint Boolean mask.

[0039] Through the above mask generation and discrete state mapping, this system achieves deep data cleaning of the underlying big data of power marketing, stripping away the dirty data intertwined with communication anomalies and physical power outages, and providing a high-quality clean data source for subsequent analysis.

[0040] The S500 deviation calculation module extracts continuous valid time-series segments based on a dual-end joint Boolean mask. It then calls a segmented aggregation approximation algorithm to perform order reduction compression on the time-series data segments to obtain a coarse-grained alignment path. This coarse-grained alignment path is mapped to the high-frequency data space to generate a constraint bandwidth. Dynamic time warping calculation is then performed within the original high-frequency resolution data and the constraint bandwidth to output the corresponding node topology deviation distance value. Thus, the system completes the feature extraction process from massive time-series power consumption sequences. The output topology feature deviation can directly empower advanced application scenarios such as transformer substation relationship identification and lean line loss analysis in the power marketing system.

[0041] In low-voltage distribution areas, due to differences in communication network media and link interference, data reported by terminal devices may experience network delays. To address the timing discrepancies caused by network delays, this embodiment introduces a dual-time-axis mechanism to achieve asynchronous multi-source data analysis. Step S100 specifically includes the following sub-steps.

[0042] S101, the data alignment module sends clock synchronization commands to each terminal device via the communication network. The real-time clock unit inside the terminal device performs time base calibration according to the received clock synchronization commands. For the process of terminal devices performing time base calibration using network time protocols or precision time protocols, those skilled in the art can implement it according to standard communication protocols; the underlying protocol parsing and time compensation mechanisms are well-known technologies in the field and will not be elaborated here. The terminal device acquires the electrical quantity values ​​of its node according to a preset acquisition period. As a preferred method, this acquisition period can be set to a fixed time interval value between 1 minute and 15 minutes. When acquiring electrical quantity values, the terminal device reads the current time value of its internal real-time clock unit and sets this time value as the physical sampling time stamp. Physical sampling time stamp It is bound to the corresponding electrical quantity value to characterize the moment when the electrical quantity value is acquired.

[0043] S102, the terminal device will be bound to a physical sampling time stamp. The electrical quantity values ​​are encapsulated into low-level data packets and sent via the communication network to the edge computing gateway or main cloud platform deployed with the data alignment module. The low-level data packets generated by each node may experience delays of varying durations due to factors such as network bandwidth, routing hop count, and channel interference during transmission. When the data alignment module receives and parses the low-level data packet, it calls the system time interface of its system to obtain the current absolute time value and sets this absolute time value as the data arrival time stamp. Data arrival time stamp This is used to characterize the moment when the data alignment module actually receives the underlying data packet.

[0044] S103, the data alignment module is based on the parsed electrical quantity values ​​and physical sampling time stamps. With data arrival time label This involves constructing data records with dual time dimensions for each node. Specifically, the data alignment module matches the active power values ​​collected by any node with the physical sampling time labels generated above. and data arrival time labels The data is then correlated to form a complete data record. Based on this, the data alignment module writes the data record into the underlying time-series database for storage.

[0045] Through the above processing, the discrete, low-level collected data is given two independent time dimensions: data generation and data reception. Even if the terminal device experiences a communication failure during operation and historical data is centrally reported after the network is restored, the data alignment module can still rely on the fixed physical sampling time stamp. This helps identify the true acquisition sequence of electrical quantities, thereby reducing the interference of communication network delays on time series data analysis.

[0046] See Figure 3 After acquiring and storing data records with dual timestamps, the system aligns and truncates the asynchronously arriving data. Step S100 may further include the following sub-steps.

[0047] S104, the data alignment module sets multiple consecutive calculation sections during the system's runtime. In this embodiment, the cross-section is calculated. As the standard time reference point for data synchronization assessment of the distribution area, its time interval is consistent with the preset acquisition cycle of the terminal equipment. As a preferred method, when the preset acquisition cycle of the terminal equipment is 15 minutes, the data alignment module generates corresponding calculation sections on the time axis with a fixed step size of 15 minutes. .

[0048] S105, Due to the time consumption of communication network transmission, the arrival time of data packets collected by the terminal device at the data alignment module is often later than the corresponding physical sampling time stamp in the data record. To tolerate network transmission latency, the data alignment module is configured for the computational cross-section. Configure a late wait window Late arrival waiting window This is a fixed-length time value, preferably ranging from 1 minute to 5 minutes. This time value can be set by technical personnel based on the historical average latency level of the communication network and the timeliness requirements of the business system for analysis. A longer delay waiting window can be configured. It can capture more delayed data, but it may also reduce the overall timeliness of data analysis.

[0049] S106, After determining the above parameters, the data alignment module uses the calculated cross-section. With late waiting window This determines the data collection deadline for the cross-section. Specifically, the data alignment module will calculate the cross-section... Time value and late waiting window The time values ​​are added together to generate the corresponding data collection deadline. The data alignment module is internally configured with timed triggering logic. When the current running time of the main cloud platform or edge computing gateway reaches the data collection deadline, the data alignment module triggers a timed event for the computing section. Data extraction operations.

[0050] S107, after triggering the data extraction operation, the data alignment module traverses the underlying time-series database and performs condition matching on historical data records based on time tags. In this embodiment, the extraction condition set by the data alignment module is: the physical sampling time tag in the data record. Equal to the current calculated cross section And the data in the data record arrives at the time stamp. The timeframe is less than or equal to the aforementioned data collection deadline. For data records that simultaneously meet both of these conditions, the data alignment module extracts and integrates them into the current calculation profile. The system collects cross-sectional data. During this process, data records arriving later than the data collection deadline will not be included in the current cross-section's extraction scope but will be stored as existing data in the time-series database. Through this waiting and extraction mechanism, the system can capture historical data delayed due to network latency within a limited time frame.

[0051] After acquiring the aforementioned cross-sectional data set, the system classifies and determines the data reporting status of each node within the transformer area, distinguishing between nodes reporting data normally and nodes with missing data. Step S100 may further include the following sub-steps.

[0052] S108, the data alignment module reads the topology files of the transformer area equipment and extracts the complete set of all nodes that need to be monitored within the transformer area. The process of reading the topology files to obtain equipment file information can be implemented by those skilled in the art based on the conventional file management rules of electricity information collection systems, and is well-known technology in the field, so it will not be described in detail here. In this embodiment, the data alignment module traverses the currently calculated cross-section obtained above. The cross-sectional data set is then compared one by one with the data arrival status of each node. If a node has a corresponding data record in the cross-sectional data set, it indicates that the data reported by that node meets the physical sampling time label. Equal to calculating cross-section And the data arrives at the time stamp. The condition is that the node's time is less than or equal to the data collection deadline. For nodes that meet this condition, the data alignment module includes them in the valid dataset. It was included in the valid dataset. The node in the current calculation section The current state is usually considered to be in normal data acquisition and communication mode, and the active power value recorded is used to support subsequent logical calculations.

[0053] S109, Determining the valid dataset Then, the data alignment module matches the above full set of nodes with the valid dataset. By comparing the data, nodes that had not yet obtained current cross-sectional data by the data collection deadline were identified. The data alignment module then grouped these nodes that had not reported data in a timely manner into the missing data candidate set. As a preferred method, the data alignment module calls the set difference operation to remove valid datasets from the full set of nodes. The remaining nodes are then assigned to the missing candidate set. In specific circumstances, if a power outage occurs in the distribution area or the communication master node goes offline, resulting in a loss of valid data sets... Even if the set is empty, the difference operation still holds true; in this case, all nodes in the full set of nodes are included in the missing node candidate set. .

[0054] For those classified into the missing candidate set The reasons why nodes in a dataset fail to provide data on time typically fall into two categories. One is that the node experiences an actual power outage, causing the corresponding terminal equipment to lose power and thus be unable to collect and report data. The other is that the node has a normal power supply, but is affected by a communication network outage or channel congestion, causing the underlying data packets to fail to reach the receiving end in time, thereby triggering a communication failure. Because there are multiple possibilities for missing data, the actual operational status of a node cannot be confirmed solely based on the missing data status. Therefore, after completing the classification and partitioning of the dataset, the data alignment module will... With missing candidate set The data is synchronously output to the data attribution module, serving as the data basis for subsequent joint attribution of execution status.

[0055] See Figure 4 When assessing the status of missing node data, it is usually necessary to set a dynamic judgment threshold as a criterion. Due to differences in network topology, equipment type, and operating load across different distribution areas, a single fixed parameter is difficult to adapt to complex field environments. To provide a more accurate judgment benchmark, in this embodiment, the threshold generation module constructs a multi-dimensional, three-level judgment threshold. Step S200 above may specifically include the following sub-steps.

[0056] S201, During the process of constructing the three-level judgment threshold, the threshold generation module obtains the globally set threshold. Globally set threshold This reflects a general error tolerance benchmark across the region, typically applicable to cold start scenarios where the analysis system is deployed for the first time or lacks historical data accumulation. As a preferred approach, a global threshold is set. The specific values ​​can be pre-configured in the system parameter library by maintenance personnel based on power grid operation procedures and historical statistical experience of the entire network. When historical data for a distribution area is missing, this parameter can provide a basic basis for the system to make judgments.

[0057] S202, considering that a single global parameter has weak specificity, the threshold generation module further calculates and obtains the clustering threshold based on the transformer area feature parameters. This leads to the introduction of a threshold benchmark based on the similarity of distribution transformer areas. Specifically, the threshold generation module extracts static feature parameters of the distribution transformer area to be analyzed. These parameters may include the rated capacity of the transformer, the total number of connected users, the proportion of photovoltaic equipment installed capacity, and the total length of the main line. Subsequently, the threshold generation module combines these parameters into a distribution transformer area feature vector and inputs it into the system's preset K-Means clustering algorithm. By calculating the Euclidean distance between feature vectors, the system matches similar distribution transformer areas in the same cluster as the feature vector in the sample database of distribution transformer areas under its jurisdiction. After determining the similar distribution transformer area group, the threshold generation module extracts the residual data of each distribution transformer area in the group under the historical normal operation section, calculates the average level of the residual data of the group, and sets its feature mean as the clustering threshold of the distribution transformer area to be analyzed. .

[0058] S203, In order to reflect the inherent electrical line loss and acquisition error characteristics of the transformer area itself, the threshold generation module calculates and obtains a local threshold based on the historical health samples of the transformer area to be analyzed. In practical implementation, the threshold generation module retrieves operational data of the analyzed transformer area within a preset historical time period (e.g., the past 30 days) from the underlying time-series database. By filtering out cross-sections where all nodes within the transformer area have normally collected and reported data, the threshold generation module defines these sets as historical health samples for the transformer area and extracts the historical residual data recorded in these historical health samples. This process is then used to calculate the local threshold. At that time, the threshold generation module adds the mathematical mean of the historical residual data in the historical healthy samples to the product of the fault tolerance adjustment coefficient and the standard deviation of the historical residual data, obtains the corresponding value, and sets it as the local threshold. The tolerance adjustment coefficient can be set according to the tolerance for residual fluctuations. As a preferred approach, its value can be set between 2 and 3. In certain specific scenarios, if the area to be analyzed is a newly commissioned area, resulting in insufficient historical healthy samples to support the calculation of statistical parameters (e.g., fewer than 3 samples), then this local threshold... This can be temporarily recorded as a null value. In this case, the lack of a judgment criterion can be addressed by a subsequent weight smoothing mechanism using the aforementioned globally set threshold. Clustering threshold Perform parameter compensation.

[0059] Through the above processing steps, the threshold generation module obtains the judgment benchmark parameters from three dimensions: global setting, group similarity matching, and the historical statistics of the transformer area itself, providing a data foundation for the subsequent generation of dynamic judgment thresholds.

[0060] After obtaining the aforementioned three-level judgment thresholds, the system evaluates the applicability of different levels of thresholds in determining the current data missing status. Since the completeness of the historical data of the transformer substation determines the reference value of local features, in this embodiment, the threshold generation module calculates dynamic confidence weights based on the time accumulation characteristics. Step S200 may further include the following sub-steps.

[0061] S204, Before performing weight calculation, the threshold generation module assesses the completeness of the historical data for the area to be analyzed. Local threshold. The reliability depends on the amount of data accumulated in the area to be analyzed. Therefore, the threshold generation module counts the number of historical health samples actually accumulated in the area to be analyzed within a preset historical time period. This number of historical health samples reflects the effective cumulative length of the area's normal operation status in the time dimension, and thus determines the degree of trust the system has in the characteristics of the area itself.

[0062] S205, to quantify the reference value of the transformer area's own historical data for the current judgment benchmark, the threshold generation module calculates the local confidence weight coefficient based on the aforementioned number of historical healthy samples. In this embodiment, the threshold generation module introduces an exponential decay model to construct the weight coefficient. This model reflects the diminishing marginal effect of information gain resulting from the continuous accumulation of sample data, causing the local confidence weight coefficient to... It can gradually increase with the increase of the number of historical healthy samples, and eventually smoothly approach the value of 1. The specific calculation logic is shown in the following formula: ; In the formula, This represents the local confidence weighting coefficient to be calculated; This represents the number of historical healthy samples obtained by the threshold generation module. This represents the preset decay adjustment parameter. This decay adjustment parameter is used to control the convergence rate of the weight coefficients as the sample size increases. As a preferred method, the decay adjustment parameter... The value range is set to 0.01 to 0.05. Relevant technical personnel can fine-tune this parameter according to the expected historical data dependence of the business system.

[0063] S206, after calculating the local confidence weight coefficient Subsequently, the threshold generation module synchronously determines the supplementary confidence weight coefficients for external experience compensation. Based on the normalization characteristics of probability weights, the threshold generation module combines the value 1 with the aforementioned local confidence weight coefficients. Subtracting the two gives the corresponding supplementary confidence weight coefficient (i.e. This supplementary confidence weight coefficient is used to guide the clustering threshold in scenarios where historical data for the transformer substation itself is insufficient. With global threshold settings The degree of intervention in the judgment. For example, when the area to be analyzed is a newly commissioned device and the number of historical health samples is zero, the local confidence weight coefficient... If the initial value is zero, the confidence weight coefficient is set to 1, indicating that the system will rely on the operational experience of similar groups or a globally applicable benchmark when generating dynamic judgment thresholds to avoid algorithmic dead zones. Through the above steps, the discrete number of samples is converted into dynamic weight parameters that can be used for subsequent smooth fusion calculations.

[0064] After obtaining the aforementioned three-level judgment thresholds and their corresponding confidence weight coefficients, the system performs a weighted fusion operation on the thresholds to generate a dynamic benchmark adapted to the current cross-sectional state. Step S200 may further include the following sub-steps.

[0065] S207, Before performing the fusion calculation, the threshold generation module determines the internal allocation ratio of the supplementary confidence weight coefficients. Group similarity features, compared to global settings, are usually closer to the actual power consumption environment of the area to be analyzed; therefore, the threshold generation module uses clustering thresholds in the supplementary confidence weight coefficients. A higher reference weight is allocated. As a preferred approach, the threshold generation module incorporates empirical allocation parameters. This is to adjust the ratio between the two. Specifically, Represents the clustering threshold The proportion of supplementary experience, Represents a global threshold setting The proportion of supplementary experience. In this embodiment, the experience allocation parameter. The value can be set from 0.7 to 0.8. Technical personnel can adjust this parameter based on the number of similar station area samples available within the system. Generally, the more abundant the sample size, the more efficient the parameter allocation becomes. The value can be set higher accordingly.

[0066] S208, Based on the above weight allocation logic, the threshold generation module comprehensively considers local thresholds. Clustering threshold and global threshold settings Calculate the dynamic judgment threshold at the current time section. The specific calculation logic is shown in the following formula: ; In the formula, This represents the dynamic threshold to be calculated. This represents the local confidence weight coefficient obtained from the preceding steps; This represents a local threshold generated based on historical data of the area to be analyzed. This represents the clustering threshold generated based on group similarity; This indicates the system's preset global threshold value; This represents the empirical allocation parameter used to assign supplementary weights.

[0067] To avoid algorithmic logic dead zones, when the area to be analyzed is a newly commissioned device, resulting in local threshold values ​​being acquired... When the value is null, the system calculates the local confidence weight coefficient based on the preceding logic. The value is zero. In this scenario, the formula is related to the local threshold. If the result of the multiplication is zero, the system will call the operating experience of similar groups and the network-wide benchmark when generating the judgment benchmark to maintain the continuity of the calculation process.

[0068] This weighted fusion calculation mechanism is based on the Bayesian smoothing principle. The system uses a clustering threshold... With global threshold settings As prior knowledge, the local threshold of the transformer area itself As posterior observation data. In the initial stage after a transformer substation is first connected to the system or after a long power outage, the local confidence weighting coefficient is relatively low due to the limited amount of observation data. The local confidence weighting coefficient is relatively small, and the judgment criterion is dominated by prior knowledge. As the equipment continues to operate and valid data accumulates, the local confidence weighting coefficient... As the weights gradually increase, the system smoothly transfers the weights to the inherent error characteristics of the transformer area. Based on this, the threshold generation module calculates the generated dynamic judgment threshold. It is passed to the subsequent data attribution module as a tolerance limit for evaluating the operating status of the node.

[0069] See Figure 5 To obtain the dynamic judgment threshold applicable to the current cross section Subsequently, the system analyzes the specific reasons for missing node data using the principle of energy conservation. In this embodiment, the data attribution module calculates the compensating physical residual. The total power actually lost within the current cross-section is quantified. Step S300 above may specifically include the following sub-steps.

[0070] S301, the data attribution module extracts the transformer substation master table data and auxiliary acquisition equipment data for the current section from the system database. The transformer substation master table records the total three-phase active power input for the entire substation. If the transformer substation master table data for the current section is missing, the system will directly stop the attribution analysis for the current section and trigger an alarm; if the transformer substation master table data is normal, the subsequent analysis will continue.

[0071] Auxiliary data acquisition equipment typically includes distributed power sources or energy storage devices connected within the distribution area. When no auxiliary data acquisition equipment is detected within the distribution area, the data attribution module sets the corresponding auxiliary input power parameter to zero to maintain the continuity of the algorithm logic. For the collection and parsing of data from the distribution area's main meter and auxiliary data acquisition equipment, those skilled in the art can implement it according to conventional electricity information collection protocols; its message parsing mechanism is a well-known technology in the field and will not be elaborated upon here.

[0072] S302, The data attribution module traverses the valid datasets obtained from the previous steps. The system extracts the active power values ​​reported by each node in the dataset at the current cross-section. By summing these active power values, the data attribution module obtains the total power consumption of nodes currently in normal data acquisition and communication status within the distribution area. If a large-scale network outage occurs in the distribution area, resulting in a loss of valid dataset... Since it is an empty set, the data attribution module assigns the total power consumption a value of zero.

[0073] S303. To eliminate the interference of inherent line losses on subsequent power balance calculations, as a preferred method, the data attribution module estimates the baseline line loss power for the current section based on the historical operation log of the transformer area. Specifically, the data attribution module extracts the average line loss rate of the transformer area under historical normal operation conditions, and multiplies this average line loss rate by the input active power recorded in the current transformer area's master table. The product is then used as the baseline line loss power for that transformer area in the current section.

[0074] S304, based on the principle of power balance, the data attribution module integrates the above parameters to calculate the compensated physical residual at the current cross-section. After deducting the power consumption of known normal nodes and the estimated baseline loss power from the sum of the total input power and auxiliary input power in the distribution area, the remaining power represents the power of nodes in unknown states (i.e., the missing candidate set). The energy consumption of the nodes in the diagram is reflected in the current cross-section. The specific calculation logic is shown in the following formula: ; In the formula, This represents the compensated physical residual to be calculated; This indicates the input active power recorded in the current cross-section lower station area table; This represents the total compensated input power reported by the auxiliary acquisition device; This indicates a valid dataset obtained from the preceding steps; Indicates a valid dataset The Middle The active power reported by each node; This represents the estimated baseline power loss.

[0075] Through the above calculation steps, the data attribution module integrates the individual power consumption parameters into a compensating physical residual. The residual value quantifies the number of candidates that were included in the missing set due to unclear state. This data represents the actual power consumed by the node at the current cross-section. Obtaining this value provides the system with clear data support for distinguishing whether a node is experiencing a power outage or a communication failure.

[0076] See Figure 6 In obtaining compensated physical residuals Subsequently, the system transforms the node state inference process into an optimization problem. In this embodiment, the data attribution module constructs an objective function that integrates historical load baseline deviation and communication failure penalty weights to infer the missing candidate set. The specific operating status of each node. Step S300 above may further include the following sub-steps.

[0077] S305, To avoid abnormal program interruptions during the calculation process, the data attribution module checks for missing candidate sets. Is it an empty set? If it is an empty set, it means that there are no nodes with unknown states in the current section, and the system skips the subsequent attribution calculation. When the candidate set is missing... When not empty, the data attribution module extracts the power consumption data of each node in the set under similar historical cross-sections and calculates the corresponding historical load baseline power. This parameter characterizes the expected power consumption level of a node under normal power supply conditions. Based on this, the data attribution module obtains the communication failure penalty weight for each node. As a preferred approach, communication failure penalty weights The system assigns a value based on the reciprocal of the node's historical communication success rate to reflect the prior probability differences in communication failures among different nodes. To prevent computational overflow due to a zero historical communication success rate, a very small constant (e.g., 0.01) is pre-added to the historical communication success rate before calculating the reciprocal. The extraction and statistical analysis of the node's historical communication success rate can be achieved by consulting the routine logs of the network management system, which is a well-known technique in the field and will not be elaborated upon here.

[0078] S306. To evaluate the rationality of different node state combination schemes, the data attribution module introduces binary state decision variables for each node. And construct a joint attribution objective function. When the state decision variable A value of 1 indicates that the corresponding node is considered to be experiencing a communication failure and is still consuming power; a value of 0 indicates that the corresponding node is considered to be in a power outage state and is consuming zero power. The data attribution module will compensate for the physical residuals. Historical load baseline power of all nodes identified as communication failures The absolute value of the difference between the sums is used as the power deviation term, while a penalty term is introduced to restrict unreasonable judgment combinations. The specific mathematical model is shown in the following equation: ; In the formula, This represents the value of the joint attribution objective function to be calculated; Represents the decision variables based on the state of each node. The resulting decision vector; This represents the compensated physical residual obtained from the pre-calculation. Indicates a missing candidate set; Indicates missing candidate set The first in One node; Indicates the first The state decision variables of each node; Indicates the first Historical load baseline power of each node; This represents the preset penalty balance coefficient; Indicates the first The communication failure penalty weight for each node.

[0079] S307, in the joint attribution objective function Internally, the power deviation term ensures that the total expected power consumed by nodes identified as communication failures is close to the actual power lost by the distribution area, thus satisfying the energy conservation constraint. The penalty term, on the other hand, penalizes the balance coefficient. The system's fault tolerance boundary is adjusted to prevent the algorithm from misclassifying nodes with low power consumption and stable historical communication performance as communication failures. In this embodiment, the penalty balance coefficient... The value range is set from 0.1 to 0.5. Relevant personnel can configure this coefficient based on the stability of the actual communication environment in the distribution area. For distribution areas with poorer network environments, the penalty balance coefficient should be increased. The value of can be adjusted accordingly to improve the system's tolerance to communication anomalies. This is achieved by finding a joint attribution objective function. Minimize the decision vector The data attribution module can output the optimal state combination scheme that satisfies the dual constraints of power balance and communication probability, thereby accurately distinguishing the fault type of the node.

[0080] After constructing the joint attribution objective function, to reduce the computational resource consumption of the optimal solution search process, the data attribution module introduces a heuristic boundary pruning mechanism based on extreme value conditions before executing the conventional optimization algorithm. This mechanism utilizes the system's boundary prior knowledge to pre-filter deterministic scenarios with consistent states. Step S300 may further include the following sub-steps.

[0081] S308, In this embodiment, the data attribution module utilizes the dynamic judgment threshold generated in the preceding steps. For compensating physical residuals Perform a lower bound extreme value condition check. The specific check logic is based on a comparison of the compensated physical residual. Is the absolute value less than or equal to the dynamic determination threshold? If this condition is met (i.e., the residual fluctuation is within the positive and negative tolerance range), it indicates that the electrical energy lost by the transformer substation at the current cross-section is within the normal tolerance range, meaning there is a missing candidate set. The nodes in the dataset typically do not consume energy. In this case, the data attribution module triggers the lower bound pruning rule, directly determining the missing candidate set. All nodes experience an actual power outage, and all corresponding state decision variables are... The value is uniformly assigned to 0, thus ending the attribution calculation for the current section.

[0082] S309, when the aforementioned lower bound extreme value condition is not met, the data attribution module continues to execute the upper bound extreme value condition determination. Specifically, the data attribution module calculates the missing candidate set. All nodes (node ​​number marked as) Historical load baseline power The sum is calculated and compared with the expected total power consumption and the compensated physical residual. The difference between them. The judgment logic is to check whether the absolute value of the difference between the two is less than or equal to the dynamic judgment threshold. If this condition is met, it indicates that the lost power at the current cross-section basically matches the total expected power consumption of the candidate nodes. The candidate nodes are typically in normal power consumption mode, and the data was only not successfully delivered to the main station due to obstruction of the underlying communication network. Therefore, the data attribution module triggers the upper bound pruning rule to determine the missing candidate set. All nodes experienced communication failures, and all corresponding state decision variables were... The value is uniformly assigned to 1.

[0083] S310, To ensure the completeness of the algorithm logic, if the parameter states of the current section do not satisfy the above-mentioned lower and upper bound extreme value conditions, it indicates that the candidate set is missing. The system handles nodes in a hybrid state, encompassing both power outages and communication failures. In this situation, the data attribution module halts heuristic pruning, retains the currently constructed joint attribution objective function, and inputs the relevant parameters into the subsequent solution process for precise optimization of the combined states. By setting the aforementioned heuristic boundary pruning rules, the system helps avoid the computational resource consumption caused by calling complex optimization algorithms in extreme single-state scenarios, improving the overall execution efficiency of data attribution while maintaining judgment accuracy.

[0084] If the heuristic boundary pruning conditions are not met, the data attribution module transforms the optimization process of the aforementioned joint attribution objective function into an integer knapsack problem with a penalty term. By introducing a dynamic programming mechanism, the system can derive a combination of node states that meets the constraints under limited computational resources. Step S300 may further include the following sub-steps.

[0085] S311, in actual electrical data acquisition, historical load baseline power Compensation-type physical residuals Typically, this is represented as a floating-point number with decimals. Directly using it as the state index in dynamic programming can easily lead to an excessively large state space. To ensure the feasibility of the algorithm, in this embodiment, the data attribution module sets a discrete scaling factor. As a preferred approach, this discrete scaling factor is set to 100 or 1000. The system calculates the historical load baseline power corresponding to each node. Compensating physical residuals and dynamic judgment threshold Multiply the result by the discrete scaling factor and round down to convert it to an integer parameter. The converted parameter is defined here as the discrete baseline power. Discrete physical residuals and discrete decision threshold .

[0086] S312, based on the transformed integer parameters, the data attribution module uses discrete physical residuals. With discrete decision threshold The sum determines the maximum search capacity of dynamic programming. (Right now ), and construct a one-dimensional state array. And a corresponding path record array. This state array The elements in the table represent the discrete baseline power of the node determined to be a communication failure. The sum is exactly (in ≤ Under the condition of ), the minimum cumulative communication penalty weight value that the system can withstand. During the initialization phase, the data attribution module will The value is assigned to 0, and the remaining elements in the array are assigned the set upper limit value (such as the maximum constant supported by the system) to ensure the normal operation of the subsequent minimization comparison logic.

[0087] S313, After initialization is complete, the data attribution module traverses the missing candidate set. For each node in the process, the state array is updated step by step through the state transition equation. For the basic two-dimensional to one-dimensional state compression mechanism for solving the 0-1 knapsack problem using a one-dimensional array, those skilled in the art can consult conventional algorithm manuals. Its underlying computational logic is well-known in the field and will not be elaborated upon here. In this embodiment, regarding the first... The system uses a reverse traversal of the capacity of the nodes. The state is updated in this way. To avoid negative array indices, the program only updates the state when the condition is met. ≥ Under the constraints, the specific computational logic is executed, and the corresponding state transition equation is shown below: ; In the formula, Indicates capacity is The minimum cumulative communication penalty weight value at that time; This represents a function that takes the minimum value. This represents the capacity index of the current dynamic programming state. Indicates the first Discrete baseline power after node transformation; This represents the preset penalty balance coefficient; Indicates the first The communication failure penalty weight corresponding to each node.

[0088] S314, State Array After the full update is complete, the data attribution module searches for the optimal solution within the set tolerance range. Specifically, the system searches for the optimal solution within the discrete physical residuals. The surrounding area is determined by discrete thresholds. The defined upper and lower limit intervals (i.e.) ,in Within the maximum value function, find the capacity index value that minimizes the total generation value, which is determined by the absolute value of the current power deviation and the corresponding penalty term. The values ​​are added together. After locating the optimal capacity index, the data attribution module performs reverse backtracking by consulting the path record array to determine which nodes' state decision variables are under this optimal combination. It is adopted and assigned a value of 1. For the remaining nodes not covered by the backtracking path, their state decision variables... All were assigned the value 0.

[0089] At this point, the data attribution module has completed the analysis of the missing candidate set. Classification and identification of mixed states. The system classifies state decision variables. Nodes with a value of 0 are classified as devices experiencing power outages, while nodes with a value of 1 are classified as devices experiencing communication failures. This determination is then output to the terminal maintenance system, providing a data-supported classification basis for subsequent emergency repair dispatching.

[0090] After completing the attribution solution for nodes with unknown states, the system needs to convert the diagnostic results into structured identifiers that can be recognized by subsequent data reconstruction processes. In this embodiment, the mask generation module formulates discrete state mapping rules to uniformly convert different types of node states within the transformer area into binary label formats. The above step S400 may specifically include the following sub-steps.

[0091] S401, To ensure the integrity of the mapping logic, the mask generation module pre-confirms that the complete set of nodes within the transformer area consists of a valid dataset. With missing candidate set It consists of two parts to prevent label assignment omissions. This is for valid datasets. The nodes in the data processing module, whose electricity consumption information collection and underlying communication processes are both in normal working order, objectively reflect the actual energy consumption. Based on this, the mask generation module maps the discrete states corresponding to these nodes with labels. The value is directly assigned to 1 to indicate that the underlying data of the node under the current section is in a valid state.

[0092] S402, for missing candidate sets In the data structure, the mask generation module performs numerical transformation based on the state decision variables output from the previous steps. When the state decision variable corresponding to a node is 0, it indicates that the device has experienced an actual power outage, and its objective power consumption is determined to be zero. This value represents known real information, and such nodes are treated as valid data (i.e., the corresponding label is assigned a value of 1), which helps prevent subsequent data reconstruction algorithms from incorrectly filling in the values. When the state decision variable is 1, it indicates that the device has experienced a communication failure; it is in a power-consuming state but data transmission is blocked. This data loss represents an unknown state, and the mask generation module maps the discrete states of such nodes to labels. The value is assigned to 0 to indicate that it is invalid data that needs to be interpolated and reconstructed.

[0093] S403, by integrating the above transformation conditions, the mask generation module establishes a discrete state mapping logic covering all nodes. Specifically, let the global traversal index of all nodes within the station area be... For any number A node, if the node belongs to a valid dataset. Or the node belongs to the missing candidate set. Furthermore, the corresponding state decision variable takes a value of 0, and the mask generation module uniformly maps its discrete state to labels. Assign a value of 1; if the node belongs to the missing candidate set. Furthermore, the corresponding state decision variable takes a value of 1, and the mask generation module then maps its discrete state to labels. The value is assigned to 0.

[0094] After executing the above mapping rules, the mask generation module clearly defines the conversion logic of classifying actual power outage states and normal data acquisition states as valid tags, while classifying communication failure states as invalid tags. This mechanism isolates the complex states intertwined with electrical and communication anomalies, laying the data foundation for the subsequent construction of a dual-end joint mask matrix. For the storage and database retrieval matching process of the aforementioned binary tags, those skilled in the art can use conventional relational database field mapping techniques; the access operations are well-known in the field and will not be elaborated upon here.

[0095] In scenarios involving electrical data analysis of distribution areas (such as line loss calculation or topology verification), the data validity of a single node is often insufficient to meet the requirements of joint calculation. From a business perspective, hierarchical correlation calculations require that the data of both the reference node (such as the distribution area master table) and the child node to be analyzed (such as the branch box or user table) be valid simultaneously; only then does the comparison difference between the two have actual physical meaning. Based on this, in this embodiment, the mask generation module initiates the construction process of a dual-end joint Boolean mask to filter out calculation links that meet the condition of complete validity at both ends. The above step S400 may further include the following sub-steps.

[0096] S404, the mask generation module extracts the label data corresponding to the reference nodes under the current section based on the discrete state mapping results determined in the previous steps. For the sake of consistency in subsequent matrix operations, this is defined here as the discrete label of the reference node. Simultaneously, the system extracts each sub-node to be analyzed (assuming the total number of sub-nodes to be analyzed is...). The global traversal index is denoted as The corresponding label data is defined as the discrete label of the child node. Both types of labels are binary variables with values ​​of 0 or 1, used to indicate whether the underlying data of the corresponding node under this section is in a reliable and valid state.

[0097] S405, to identify topology links where both ends of the data are valid, the mask generation module uses bitwise AND operations to fuse the aforementioned tags. Specifically, the system discretizes the tags of common reference nodes. Discrete labels for each child node Perform a step-by-step comparison to generate the corresponding dual-ended joint Boolean mask. Based on fundamental logic and operational rules, the dual-ended joint Boolean mask is calculated and output only when the data of both the reference node and the child node to be analyzed are simultaneously determined to be valid (i.e., both corresponding discrete labels are set to 1). The value is assigned to 1. If any end of the calculation link contains default data that is judged as a communication failure (i.e., there is a case where the tag value is 0), the calculation result of that link will be forcibly set to 0. This judgment mechanism covers the extreme case where the reference node itself is missing, which helps to prevent the error propagation caused by the lack of data at one end.

[0098] S406, after completing the bit-by-bit logical calculation for all target child nodes, the mask generation module combines the two-end joint Boolean masks generated for each link. The nodes are stitched together dimensionally according to a pre-defined sequence of transformer area topology nodes. This is achieved by... By integrating discrete mask variables, the system constructs a sequence of length [missing information]. One-dimensional or multi-dimensional bi-ended joint Boolean mask matrix As a preferred approach, this dual-ended joint Boolean mask matrix This will be directly used as a filter window function input into subsequent data reconstruction algorithms. At the matrix dot product level, this mask matrix can directly shield computation pairs containing default invalid data, thereby ensuring the input quality of the reconstructed model. Regarding the storage structure of the aforementioned Boolean matrix in the computer's underlying memory and the specific implementation of bitwise logical operations, those skilled in the art can use conventional tensor operation libraries (such as NumPy or TensorFlow) for configuration. The underlying hardware scheduling mechanism is a well-known technology in this field and will not be elaborated upon here.

[0099] After obtaining the dual-end status identifiers of the target device, to prevent abnormal default data from interfering with subsequent time series alignment calculations, in this embodiment, the deviation calculation module uses a pre-generated mask sequence to filter and reorganize the historical operating data of the distribution area, extracting continuous and valid data segments for calculating the deviation. Step S500 may further include the following sub-steps.

[0100] S501, the deviation calculation module pre-sets an analysis time window. As a preferred approach, the span of this analysis time window can be set to the duration of a complete electricity consumption data collection cycle (e.g., 24 hours or 7 days) to cover the fluctuation characteristics of daily or weekly load. Within this time range, the system extracts the time-series electrical data of the reference node and the specific sub-node to be analyzed according to the set sampling step size, respectively constructing a one-dimensional original sequence of the reference node. With the original sequence of child nodes Simultaneously, the system iterates through all historical computation sections within the analysis time window, retrieving the dual-ended joint Boolean mask generated for the corresponding computation link under each sampling section. Boolean masks at each time point Arrange the data in chronological order to generate a masked time series with the same dimensions as the original sequence data. .

[0101] S502, by introducing a masking mechanism, the deviation calculation module will use the mask time series. As a filtering condition, the original sequence of the reference node With the original sequence of child nodes The system performs synchronized data capture. Specifically, it iterates through all sampling points within the analysis time window. When the mask value corresponding to a given time point is 1, it retains the reference node and child node data for that section; when the mask value is 0, the system directly discards the data for that time point. Through this iterative comparison operation, the system can eliminate sections with communication failures or missing data, and obtain the corresponding valid reference node and child node data.

[0102] S503. After the above truncation operation, the remaining valid data typically exhibits a discrete state with intervals on the timeline. Directly inputting these discrete fragments with gaps into the time series alignment algorithm can easily cause mapping deviations in the flexible alignment process. Therefore, the deviation calculation module removes the original absolute timestamps from the selected valid data points and concatenates them only according to the order in which the data was generated. By eliminating the gaps on the timeline, the system constructs a valid reference time series with coherent internal data changes. With effective child node time sequence .

[0103] S504, after completing the data splicing, considering that the truncated sequence may be too short due to long-term communication anomalies, the deviation calculation module calculates the effective child node time sequence generated by the splicing. The length is verified. If the sequence length is lower than the preset minimum computational length threshold (e.g., three consecutive sampling points), it indicates that the node lacks alignment samples within the current time window. In this case, the system terminates subsequent deviation calculations for that node and assigns it a preset default maximum deviation value to prevent abnormal computational interruption. For sequence pairs whose length meets the requirements, the system uses them as basic inputs to pass to the subsequent multi-scale deviation calculation model. For the above-mentioned time series slicing and array recombination operations in memory, those skilled in the art can call conventional tensor operation libraries for configuration. The underlying addressing mechanism is a well-known technology in this field and will not be elaborated here.

[0104] After acquiring continuous valid data segments, the data points generated by high-frequency acquisition are relatively dense. If the basic dynamic time warping algorithm is executed directly at the original resolution, the computational complexity will increase quadratically. To optimize computational efficiency, in this embodiment, the deviation calculation module uses a segmented aggregation approximation algorithm to reduce the order of the long sequence and then obtain the alignment path at the reduced scale. The above step S500 may further include the following sub-steps.

[0105] S505, the deviation calculation module extracts the valid reference time series generated in the previous step. With effective child node time sequence Thanks to the preceding masking operation, the total data length of the two sets of sequences remains the same, which is denoted here as the original sequence length. When performing dimensionality compression, the deviation calculation module pre-configures a segmented window length. The length of this segmented window. It is usually set to be greater than 1 and less than the length of the original sequence. A positive integer, the actual value of which can be adaptively adjusted according to the processing power of the computing hardware. This is achieved by adjusting the length of the original sequence. Divide by the segmented window length The deviation calculation module then rounds down to the nearest integer to determine the length of the coarse-grained sequence after the order reduction operation. In actual calculations, if the length of the original sequence... Unable to be segmented by window length The integer division and deviation calculation module truncates fragmented data at the end of the sequence that is less than the length of a complete window to prevent out-of-bounds errors in subsequent matrix slicing operations.

[0106] S506, For sequence data with confirmed length, the deviation calculation module calculates the effective reference time series sequence. With effective child node time sequence Synchronous mean aggregation is performed. Specifically, the deviation calculation module divides the original one-dimensional sequence into coarse-grained sequence lengths at equal intervals according to time sequence. The corresponding continuous and non-overlapping sub-intervals, the number of elements contained in each interval corresponding to the segmented window length. The data points are calculated. The deviation calculation module calculates the arithmetic mean of the data points contained in each sub-interval, and uses this mean to replace the time series features within the corresponding interval. Through this calculation, the deviation calculation module generates data with dimensions reduced to a coarse-grained sequence length. The coarse-grained reference sequence and the coarse-grained sub-node sequence.

[0107] S507, after data dimensionality reduction, the deviation calculation module inputs the generated coarse-grained reference sequence and coarse-grained child node sequence into a pre-set dynamic time warping model for similarity comparison. During this process, the deviation calculation module constructs a model where both row and column dimensions are the length of the coarse-grained sequence. The cumulative distance cost matrix is ​​used to find the minimum cumulative cost path that aligns the two sets of sequences using dynamic programming. This path, composed of connected matrix coordinate nodes, represents the mapping relationship between the two ends of the data in the dimensionality reduction state. Here, the output is defined as the coarse-grained alignment path.

[0108] S508, the deviation calculation module records the coordinate information contained in the coarse-grained alignment path to define the time mapping channels of upper and lower level nodes under the compression scale. Based on this coarse-grained alignment path, the deviation calculation module adjusts the original sequence length... The global optimization process of the dimension is constrained to a limited search range, providing boundary guidance for subsequent fine-grained deviation calculations at the original resolution.

[0109] After obtaining the macroscopic alignment route at the dimensionality reduction scale, the system needs to restore the route to the original high-frequency data space to guide fine-grained sequence alignment. To balance computational overhead and alignment accuracy, in this embodiment, the deviation calculation module performs coordinate magnification and boundary expansion operations to generate a constraint bandwidth (adaptive search constraint bandwidth) for the original data dimension. The above step S500 may further include the following sub-steps.

[0110] S509, The deviation calculation module extracts the coarse-grained alignment path and the corresponding segmented window length generated in the previous step. This coarse-grained alignment path consists of discrete coordinate nodes from a series of dimensionality reduction cost matrices. To achieve dimensionality restoration, the deviation calculation module projects each coarse-grained coordinate node in this path back to a scale equal to the original sequence length. In the high-frequency computational space. Specifically, the mapping method is that for any point on the dimensionality reduction path, the system expands it into a corresponding dimension of size [missing information - likely a specific dimension]. × The system constructs a basic projected path region by synchronously expanding all nodes covered on the path and stitching them together in the original high-frequency cost matrix.

[0111] S510, simply using the aforementioned projection path region may be insufficient to cover the fluctuation characteristics of local data when small time drifts occur. To give the comparison model a certain degree of fault tolerance, the deviation calculation module is pre-configured with a search radius. As a preferred method, this search radius Set to an integer greater than or equal to 0. If the value is 0, it means the system strictly follows the coarse-grained path mapping; if the value is greater than 0, the actual value can be set based on the empirical range of communication delay between devices in the distribution area. Based on this parameter, the deviation calculation module initiates a bidirectional expansion operation on the periphery of the projection path area.

[0112] S511, during the extension operation, the deviation calculation module extends the boundary of the projection path region outward along the horizontal and vertical coordinates of the original high-frequency cost matrix by a number corresponding to the search radius. The data grid is defined. Considering the potential risk of underlying memory addressing out-of-bounds errors, the deviation calculation module performs boundary truncation on the extended coordinate set. Specifically, the system sets coordinate values ​​less than 1 to 1 after expansion, and sets coordinates greater than the original sequence length... Set the coordinate values ​​to After the outward expansion and boundary regularization processes described above, the system defines a search channel within a set range in the original high-frequency matrix space. Here, the overall coordinate set containing the basic mapping block and the outward expansion width is defined as the constraint bandwidth.

[0113] S512, after establishing the aforementioned spatial boundary, the deviation calculation module uses this constraint bandwidth to limit the dynamic time warping calculation at the original resolution. When the system is within the valid reference time sequence... With effective child node time sequence When performing path optimization within the high-frequency comparison space, the deviation calculation module only allows the calculation of cumulative distance cost within the valid coordinate set covered by the constraint bandwidth. For matrix coordinate points outside this bandwidth range, the system defaults to setting their distance cost to infinity. Through this mechanism, the deviation calculation module can prune redundant calculation branches, reduce the computational overhead of global traversal in the original space, and limit excessive misalignment of time-series data that lacks practical business significance. The matrix mask configuration and boundary condition determination of the spatial coordinate set can be implemented using conventional programming control logic by those skilled in the art; its underlying scheduling is a well-known technology in the field and will not be elaborated upon here.

[0114] After establishing the spatial search boundary, the system completes the final sequence alignment and distance measurement at the original high-frequency data scale. In this embodiment, the deviation calculation module performs dynamic programming optimization based on the pre-constructed constraint bandwidth and outputs the deviation value used to determine the electrical affiliation. The above step S500 may further include the following sub-steps.

[0115] S513, the deviation calculation module extracts the effective reference time series at the original resolution. With effective child node time sequence To perform local optimization, the deviation calculation module constructs a module whose row and column dimensions are both the original sequence length. The local distance matrix. In specific calculations, the deviation calculation module only considers the matrix coordinates points within the constraint bandwidth. Calculate the absolute difference between the elements at corresponding indices of the two time series to form the local distance. For coordinate points outside the bandwidth range, the system directly assigns their local distance and corresponding initial cumulative distance cost to a preset maximum value (or infinity) to forcibly block any optimization path that crosses the bandwidth boundary and strictly limit the subsequent optimization calculation range.

[0116] S514, after generating the local distance matrix, the deviation calculation module calculates the cumulative distance cost under the constraint of bandwidth. Due to communication clock drift between the acquisition devices, data nodes may exhibit leading, lagging, or synchronous characteristics in terms of timing. Therefore, when extrapolating the cumulative cost backward, the system needs to comprehensively compare the cumulative cost status of the preceding nodes. The specific formula for calculating the cumulative cost is as follows: ; In the formula, This indicates alignment from the start endpoint of the time series to the current coordinate. Minimum cumulative distance cost of the location; Representing coordinates Local distance at; This indicates taking the minimum of the three pre-accumulated costs within the parentheses; and These represent the valid reference time series. With effective child node time sequence For discrete data indices, the value range of both is restricted to the coordinate set contained within the constraint bandwidth. For matrix boundaries (such as...) =1 or The initial state setting of (=1) can be configured by those skilled in the art using conventional boundary extreme value filling logic, which is a well-known technology in the field and will not be described in detail here.

[0117] S515, by executing the above recursive operation layer by layer, the deviation calculation module calculates the endpoint of the matrix coordinates. Global minimum cumulative distance cost at the location Since the cumulative algebraic value is derived from the sum of local distances along the entire mapping path, its magnitude is affected by the total sequence length. To eliminate the imbalance in evaluation criteria caused by differences in the amount of effective data, as a preferred approach, the deviation calculation module uses the globally minimum cumulative distance cost. Divide by the original sequence length Then, perform data normalization. Since the sequence length was verified in the previous step, the denominator here is the original sequence length. It is always greater than zero. After calculation, the deviation calculation module outputs the normalized topology feature deviation. This value is used to characterize the degree of deviation between the reference node and the child node to be analyzed under the condition of reasonable time misalignment, and serves as the direct calculation basis for determining the topology affiliation of nodes.

[0118] The system compares the deviation of the topology feature with a preset topology attribution threshold. If the deviation is less than or equal to the threshold, the sub-node to be analyzed is determined to belong to the reference node in terms of physical topology; otherwise, it is determined not to belong to the node. Thus, the system completes the topology verification and accurate determination of the low-voltage distribution area equipment.

[0119] Specific application examples Scenario Background and Basic Data Extraction: A low-voltage power supply area (area A) in a certain city is defined as containing 1 distribution transformer (configured with an area master meter), 1 photovoltaic inverter (configured with auxiliary data acquisition equipment), and 50 user-side smart IoT meters. The preset data acquisition cycle is 15 minutes.

[0120] At 10:00 AM on a certain day, the data alignment module triggered the late waiting window (set to 3 minutes) deadline. After traversing the time-series database, 45 nodes had complete data (included in the valid dataset). Five nodes failed to obtain current cross-sectional data (and were added to the missing candidate set). (Node numbers are counted from 1 to 5).

[0121] Dynamic threshold calculation: The system extracts operation records from area A within a preset historical time period and calculates the actual number of historical healthy samples. =30. Preset attenuation adjustment parameter. =0.05.

[0122] Based on the confidence weight calculation logic, obtain the local confidence weight coefficients. Therefore, the supplementary confidence weight coefficient is 0.223. Empirical allocation parameters are then set. =0.8.

[0123] Extracting threshold parameters at various levels: Local thresholds derived from the historical characteristics of the transformer substation itself. =1.2kW; Clustering threshold derived from K-Means clustering =2.0kW; System global setting threshold =3.0kW.

[0124] Perform weighted fusion calculation and output the dynamic judgment threshold for the current section. kW.

[0125] Physical residual calculation and joint state attribution: Extracting the measured power at the 10:00 section: The total active power input for the transformer substation is 85.0 kW, and the photovoltaic equipment compensation input is 5.0 kW. Based on historical normal operating day data, the baseline power loss for this section is estimated to be 3.0 kW. Valid dataset. The total power consumption of the 45 nodes is 75.0 kW.

[0126] Calculate compensated physical residuals kW.

[0127] For nodes 1 to 5 in the missing candidate set, the system extracts their historical load baseline power as follows: 5.0kW, 4.5kW, 2.0kW, 0.5kW, and 3.0kW, respectively. Nodes 1, 2, and 5 have historically low communication success rates (communication failure penalty weight). (Set to 1.5), nodes 3 and 4 have a higher historical communication success rate (communication failure penalty weight) (Set to 5.0). Set the penalty balance coefficient. =0.2.

[0128] It was determined that the residual of 12.0kW did not trigger the upper and lower bound extreme value pruning conditions. The system invoked the dynamic programming algorithm to solve the joint attribution objective function, and the output optimal solution state vector was: the state decision variables corresponding to nodes 1, 2, and 5. =1 (determined as a communication failure), the state decision variables for nodes 3 and 4. =0 (indicated as an actual power outage).

[0129] Calculation and verification: The total expected power consumption of the nodes identified as communication failure nodes is 5.0 + 4.5 + 3.0 = 12.5 kW. The absolute difference between this value and the physical residual |12.0 - 12.5| = 0.5 kW satisfies the constraint that it is less than the dynamic judgment threshold of 1.422 kW, and the judgment result conforms to the power balance logic.

[0130] Mask generation and deviation calculation: Based on the attribution results, the mask generation module maps nodes 3 and 4 to valid label 1, and nodes 1, 2, and 5 to invalid default label 0.

[0131] When performing time-series topology verification with a 7-day execution window, taking node 6 (the child node to be analyzed) and the distribution transformer (the reference node) as an example: a two-ended joint Boolean mask is generated by performing a logical AND operation on the discrete state labels of both nodes. The system uses this mask to extract valid historical time-series segments and performs head-to-tail concatenation to obtain a valid reference time-series sequence and a valid child node time-series sequence with a length of 600 sampling points.

[0132] Set the segment window length to 10, call the segmented aggregation approximation algorithm to reduce the above sequence to a coarse-grained sequence of length 60, and calculate to obtain the coarse-grained alignment path.

[0133] Map the path to the original high-frequency space and configure the search radius. =3, establish a constraint bandwidth. Within this constraint bandwidth, perform fine-grained dynamic time warping calculations, ultimately outputting a normalized topology feature deviation of 0.15. This value is less than the system's preset topology attribution threshold of 0.20, therefore, node 6 is determined to be physically topologically subordinate to the distribution transformer.

[0134] The operation logs and measurement data of 10 actual low-voltage distribution areas (containing a total of about 1,500 data acquisition nodes) were selected as test samples over a continuous period of 3 months.

[0135] Test method settings: Traditional comparison group: Node state determination adopts fixed residual tolerance limit, no mask filtering is performed on the default data in the topology verification stage, and the time series similarity is calculated by full matrix Euclidean distance algorithm.

[0136] The test group of this invention deployed a three-level dynamic judgment threshold, a joint attribution objective function, and a reduced-order dynamic time warping algorithm based on adaptive constraint bandwidth according to the above steps.

[0137] Test Result Analysis: Node status attribution accuracy: Real system fault logs were used as the comparison benchmark. The traditional comparison group, lacking prior probability weights for communication faults and dynamic threshold adjustment, achieved a classification accuracy of 72.4% in mixed states. The test group of this invention improved its classification accuracy to 95.8% by incorporating physical power deviation and communication penalty weight constraints into the objective function.

[0138] Algorithm computational overhead: For a time-series alignment task within a 7-day time window (corresponding to approximately 672 sampling points), the traditional comparison group's average time per run for full-size matrix optimization was 1.85 seconds. The present invention's test group, by invoking a segmented aggregation approximation order reduction and adaptive bandwidth constraint mechanism, avoided matrix operations in a large number of invalid regions, reducing the average time per run to 0.12 seconds. Test results show that this computational efficiency meets the batch concurrent processing requirements of the main cloud platform.

[0139] Figure 7 The receiver operating characteristic (ROC) curves of the method of this invention and traditional comparison methods are shown in the identification of node topological affiliation. The horizontal axis represents the false positive rate (i.e., the proportion of non-corresponding nodes that are mistakenly identified as belonging to the same node), and the vertical axis represents the true positive rate (i.e., the proportion of belonging nodes that are correctly identified). The test group (the method of this invention, marked with solid circles) introduces a joint Boolean mask to eliminate invalid data breakpoints and tolerates reasonable clock drift. Its curve is significantly closer to the upper left corner (AUC reaches 0.96), showing better discrimination ability and a lower false positive rate compared to the control group (marked with dashed squares, AUC is 0.81).

[0140] Figure 8This diagram illustrates the optimization constraint range established by the deviation calculation module in the original high-frequency data space. The horizontal and vertical axes represent the time-series data indices of the reference node and the child node to be analyzed, respectively. The gray-filled area in the figure represents the system's expansion through dimensionality reduction path mapping and search radius extension. =3) The generated constraint bandwidth, with the thick black solid line representing the globally optimal alignment path found by the system within this finite space. By limiting the computational range to the light gray constraint bandwidth, the system blocks invalid matching branches that exceed this boundary (the white blank area does not participate in the distance accumulation calculation), thereby reducing computational resource consumption while maintaining time misalignment tolerance.

Claims

1. A method for cleaning and feature extraction of big data in electricity marketing, characterized in that, include: Obtain the data stream containing the original high-frequency data of the station area to be analyzed, and divide the effective dataset and the missing candidate set based on the physical sampling time label, data arrival time label and data collection cutoff time of the calculation section of the data stream; By integrating global threshold settings, clustering thresholds, local thresholds, and confidence weights based on historical samples, a dynamic judgment threshold is generated. Based on the power generation compensation physical residual of the transformer area master table, auxiliary acquisition equipment and the effective dataset, and combined with the dynamic judgment threshold to solve the joint attribution objective function, the missing candidate set is divided into a first set of nodes in a physical power outage state and a second set of nodes in a communication failure state. The first node set is processed as having zero power consumption and included in the effective state. The effective dataset, the first node set, and the second node set are mapped to binary status labels. A logical AND operation is performed on the binary status labels of the reference node selected from the analysis area and the child node to be analyzed to generate a two-end joint Boolean mask. Based on the dual-end joint Boolean mask, the effective time sequence segment is extracted, and a coarse-grained alignment path is generated by reducing the order and mapping it to a constraint bandwidth. Dynamic time warping calculation is performed on the original high-frequency data of the effective time sequence segment and the constraint bandwidth, and the deviation value is output.

2. The method for cleaning and feature extraction of big data in power marketing according to claim 1, characterized in that, The data collection cutoff time based on the physical sampling time stamp of the data stream, the data arrival time stamp, and the calculation section is used to divide the valid dataset and the missing candidate set, including: Extract the electrical quantity values ​​from the data stream, as well as the physical sampling time stamp and the data arrival time stamp bound to the electrical quantity values; A late waiting window is set for the calculation section, and the time value of the calculation section is added to the time value of the late waiting window to generate the data collection deadline. When the system runtime reaches the data collection deadline, the time series database is traversed to extract matching data records, with the physical sampling time label equal to the calculation section and the data arrival time label less than or equal to the data collection deadline as extraction conditions. Obtain the full set of nodes for the area to be analyzed, assign nodes in the full set that contain matching data records to the valid dataset, and assign the remaining nodes in the full set after removing the valid dataset to the missing candidate set.

3. The method for cleaning and feature extraction of big data in electricity marketing according to claim 1, characterized in that, The process of fusing a globally set threshold, a clustering threshold, a local threshold, and a confidence weight based on historical samples to generate a dynamic judgment threshold includes: Obtain the global threshold, the clustering threshold based on the characteristic parameters of the transformer area, and the local threshold based on the historical healthy samples of the transformer area to be analyzed; Extract the number of historical health samples in the area to be analyzed within a preset historical time period, calculate the local confidence weight coefficient based on the number of historical health samples, and subtract the value 1 from the local confidence weight coefficient to obtain the supplementary confidence weight coefficient. Obtain empirical allocation parameters for allocating supplementary weights, and determine the reference weights of the clustering threshold and the globally set threshold in the supplementary confidence weight coefficients based on the empirical allocation parameters; The local confidence weight coefficient, the supplementary confidence weight coefficient, and the reference weight are used to perform a weighted fusion calculation on the local threshold, the clustering threshold, and the global set threshold to generate the dynamic judgment threshold.

4. The method for cleaning and feature extraction of big data in electricity marketing according to claim 1, characterized in that, The generation of compensated physical residuals based on the power distribution area master table, auxiliary acquisition equipment, and the effective dataset includes: Extract the input active power recorded in the transformer area master table and the compensated input power reported by the auxiliary acquisition device; Extract the active power values ​​reported by each node in the effective dataset, and perform an accumulation and summation operation on the active power values ​​to generate the total power consumption of nodes in normal acquisition and communication status. Extract the average line loss rate under historical normal operating conditions, and multiply the average line loss rate by the input active power to generate the baseline line loss power; The total power consumption and the baseline loss power are subtracted from the sum of the input active power and the compensated input power to generate the compensated physical residual.

5. The method for cleaning and feature extraction of big data in electricity marketing according to claim 1, characterized in that, By combining the dynamic determination threshold to solve the joint attribution objective function, the missing candidate set is divided into a first set of nodes in a physical power outage state and a second set of nodes in a communication failure state, including: Determine whether the absolute value of the compensated physical residual is less than or equal to the dynamic determination threshold. If so, trigger the lower bound pruning rule and assign all nodes in the missing candidate set to the first node set. If the lower bound pruning rule is not met, the sum of the historical load baseline power of all nodes in the missing candidate set is calculated, and it is determined whether the absolute value of the difference between the sum of the historical load baseline power and the compensated physical residual is less than or equal to the dynamic judgment threshold. If so, the upper bound pruning rule is triggered, and all nodes in the missing candidate set are assigned to the second node set. If neither the lower bound pruning rule nor the upper bound pruning rule is satisfied, the absolute value of the difference between the compensated physical residual and the sum of the historical load baseline power of the nodes to be assigned to the second node set is used as the power deviation term. The joint attribution objective function is constructed in combination with the communication fault penalty weight, and the state combination scheme is minimized and optimized using a dynamic programming algorithm. Based on the optimization result, the nodes in the missing candidate set are assigned to the first node set or the second node set.

6. The method for cleaning and feature extraction of big data in electricity marketing according to claim 1, characterized in that, The effective dataset, the first node set, and the second node set are mapped to binary state labels. A logical AND operation is performed on the binary state labels of the selected reference node and the child node to be analyzed to generate a two-ended joint Boolean mask, including: The objective power consumption of the nodes in the first node set at the corresponding calculation section is determined to be zero, and the nodes in the first node set are used as valid states to participate in the assignment of the binary state label. Assign the binary status label of the node belonging to the valid dataset and the node belonging to the first node set to 1; Assign the binary state label corresponding to the node belonging to the second node set to 0; Extract the binary status labels corresponding to the reference node and the child node to be analyzed, and perform a bitwise logical AND operation on the binary status labels corresponding to the reference node and the child node to be analyzed; When the binary status label corresponding to the reference node and the child node to be analyzed are both 1, the dual-ended joint Boolean mask with a value of 1 is output; otherwise, the dual-ended joint Boolean mask with a value of 0 is output.

7. The method for cleaning and feature extraction of big data in power marketing according to claim 1, characterized in that, Based on the aforementioned dual-end joint Boolean mask, valid time segments are extracted, and a coarse-grained alignment path is generated by reducing its order, including: Based on the multiple dual-ended joint Boolean masks within the analysis time window as filtering conditions, synchronized data truncation is performed on the original sequence corresponding to the reference node and the original sequence corresponding to the child node to be analyzed, and the time axis gaps are removed and the beginning and end are spliced ​​together to construct an effective reference time sequence and an effective child node time sequence as the effective time sequence segment. Based on the preset segmented window length, mean aggregation calculation is performed synchronously on the effective reference time series sequence and the effective child node time series sequence to generate a coarse-grained reference sequence and a coarse-grained child node sequence; Construct a cumulative distance cost matrix for the coarse-grained reference sequence and the coarse-grained child node sequence, and solve for the minimum cumulative cost path corresponding to the cumulative distance cost matrix to generate the coarse-grained alignment path.

8. The method for cleaning and feature extraction of big data in electricity marketing according to claim 7, characterized in that, Mapping the coarse-grained alignment path to a constrained bandwidth includes: Extract the coarse-grained coordinate nodes in the coarse-grained alignment path, project each coarse-grained coordinate node back into the computation space corresponding to the original high-frequency data of the effective time series segment according to the corresponding segmented window length, and expand it into a square matrix block, and splice it to construct the projection path region; Extract the preset search radius, extend the boundary of the projection path region outward along the horizontal and vertical coordinates of the computation space, and perform boundary truncation processing on the extended coordinate set to generate the constraint bandwidth.

9. The method for cleaning and feature extraction of big data in electricity marketing according to claim 1, characterized in that, Dynamic time warping is performed on the original high-frequency data within the effective time segment and the constrained bandwidth, and the deviation value is output, including: A local distance matrix is ​​constructed for the original high-frequency data of the effective time series segment, and the cumulative distance cost is recursively calculated within the effective coordinate set covered by the constraint bandwidth. Assign a preset maximum distance cost to coordinate points outside the constrained bandwidth range, and calculate the global minimum cumulative distance cost to reach the endpoint of the matrix coordinates; Divide the global minimum cumulative distance cost by the original sequence length corresponding to the effective time segment, and output the normalized topological feature deviation as the deviation value.

10. A system for cleaning and extracting features from big data in electricity marketing, characterized in that, For executing the power marketing big data cleaning and feature extraction method as described in any one of claims 1 to 9, the system includes terminal equipment, a communication network, and a main station cloud platform or edge computing gateway, wherein the main station cloud platform or edge computing gateway is configured with: The data alignment module is used to divide the valid dataset into the missing candidate set based on the physical sampling time label of the data stream, the data arrival time label, and the data collection deadline of the calculation section. The threshold generation module is used to integrate the globally set threshold, clustering threshold, local threshold, and confidence weight based on historical samples to generate a dynamic judgment threshold. The data attribution module is used to generate compensated physical residuals based on the power of the transformer area master table, auxiliary acquisition equipment and the effective dataset, and to construct and solve the joint attribution objective function in combination with the dynamic judgment threshold, and to divide the missing candidate set into a first node set and a second node set. The mask generation module is used to process the first node set as having zero power consumption and include it in the valid state, map the valid dataset, the first node set, and the second node set to the binary state label, and generate a two-end joint Boolean mask; The deviation calculation module is used to extract effective time segments based on the dual-end joint Boolean mask, reduce the order to generate coarse-grained alignment paths and map them to constraint bandwidths, perform dynamic time warping calculations within the original high-frequency data of the effective time segments and the constraint bandwidths, and output the deviation value.

Images