A power marketing data cleaning and deduplication method and system

Abstract

The application provides a power marketing data cleaning and deduplication method and system, wherein the neighborhood density and local attribute fluctuation degree of data points are calculated, if the neighborhood density is lower than a threshold value and the fluctuation degree is higher than a threshold value, the initial noise point is marked; otherwise, the density is divided into core points or boundary points; the asymmetric extended neighborhood is set for the boundary points, the sum of distance decay function values from the initial noise point to the boundary points in the neighborhood is calculated, if the total sum is higher than a first threshold value, the confirmed noise point is reclassified; if the initial noise point is located in the overlapping neighborhood and the geometric mean value of noise correlation indexes of the group of boundary points is lower than a second threshold value, the noise point is reclassified as a secondary boundary point and is classified into the nearest core point cluster; the confirmed noise point is isolated, the stability in the cluster is calculated; the distance from the data point to the cluster center is corrected according to the stability, the main data record with the minimum corrected distance is selected, and the redundant records are combined.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a method and system for cleaning and deduplicating electricity marketing data. Background Technology

[0002] Electricity marketing systems accumulate a large amount of user electricity consumption data during operation, which forms the basis for load forecasting, user profiling, and marketing strategy formulation. Currently, data cleaning methods for this type of data include statistical and clustering-based approaches. Density-based clustering algorithms can discover clusters of arbitrary shapes without pre-specifying the number of clusters. However, traditional density algorithms, when identifying noise points, typically rely on the spatial neighborhood density of data points, ignoring the unique business attributes of electricity marketing data. When processing electricity data with complex business logic and diverse data characteristics, they are prone to misclassifying some normal data points located in sparse areas as noise, or failing to identify data points with high density but abnormal user behavior characteristics.

[0003] In density clustering, boundary points, due to their low neighborhood density, are spatially very close to noise points, making it difficult to distinguish whether low-density points at the cluster edge are a reasonable extension of the cluster or noise that should be removed. In the data deduplication process, existing methods typically ignore the differences in data quality and stability between data points at different locations after noise removal. In particular, boundary points that were once adjacent to noise points should have lower reliability than core points far from the noise region. Therefore, the lack of a deduplication mechanism that combines the degree of correlation between data points and noise with their stability makes the deduplication results susceptible to the influence of unreliable boundary points, potentially resulting in selected master data records that are not optimally representative, thus affecting data quality. Summary of the Invention

[0004] This invention proposes a method for cleaning and deduplicating electricity marketing data to address the lack of a deduplication mechanism in existing technologies that combines the noise correlation degree and stability of data points. The method includes:

[0005] For each data point in the electricity marketing data, calculate the neighborhood density and the local attribute fluctuation based on the periodicity of electricity consumption. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point.

[0006] An asymmetric extended neighborhood is set for each boundary point, and the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point is calculated as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point.

[0007] If the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster containing the core point with the closest weighted distance to the noise point; all confirmed noise points are isolated, and within each cluster composed of core points, boundary points, and secondary boundary points, the data recording stability is calculated for each data point, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index;

[0008] Based on the stability, the distance from each data point to the cluster centroid is corrected, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged.

[0009] Furthermore, this invention also relates to a power marketing data cleaning and deduplication system, comprising the following modules:

[0010] The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point.

[0011] The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point.

[0012] The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster containing the core point whose weighted distance is closest to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold; isolate all confirmed noise points and calculate the data record stability for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index;

[0013] The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records.

[0014] This invention identifies initial noise points by combining the neighborhood density of data points with the local attribute fluctuation based on the periodicity of electricity consumption. By constructing an asymmetric extended neighborhood for boundary points and calculating the noise correlation index of these boundary points, the degree of correlation between boundary points and noise points is represented. This allows boundary points affected by noise to be reclassified as confirmed noise points, while simultaneously restoring initial noise points in neighboring stable clusters as secondary boundary points, solving the problem of distinguishing between boundary points and noise points in traditional methods. In the data deduplication stage, the stability of data records, inversely proportional to the noise correlation index, is used to correct the distance from data points to the cluster centroid, ensuring that the selected master data records within the cluster are spatially representative, thereby improving the cleaning and deduplication effect of electricity marketing data. Attached Figure Description

[0015] Figure 1 A flowchart of the first embodiment;

[0016] Figure 2 This is a schematic diagram of the initial classification of data points;

[0017] Figure 3 This is a diagram illustrating the stability of data recording. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] In the first embodiment, the present invention proposes a method for cleaning and deduplicating electricity marketing data, such as... Figure 1 ,include:

[0020] S1. For each data point in the electricity marketing data, calculate the neighborhood density and the local attribute fluctuation based on the periodicity of electricity consumption. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point.

[0021] Obtain the neighborhood radius and minimum number of neighborhood points. For any data point p, the neighborhood density of the data point is the number of data points within the neighborhood of the data point's neighborhood radius. For local attribute volatility, extract the data point p and all data points within the neighborhood of the data point's neighborhood radius in the same period dimension, such as daily electricity consumption data for 30 consecutive days, to form a local dataset. Calculate the standard deviation or coefficient of variation of all daily electricity consumption data in this local dataset as the local attribute volatility of point p.

[0022] The density threshold is preferably set to the minimum number of neighboring points. For a data point p, if the neighborhood density of the data point is less than the minimum number of neighboring points and the local attribute volatility of the data point is greater than a preset volatility threshold, then the point is marked as an initial noise point. If these two conditions are not met, the neighborhood density of the data point is checked. If the neighborhood density is greater than or equal to the minimum number of neighboring points, then the point is determined to be a core point; if the neighborhood density of the data point is less than the minimum number of neighboring points, but the point is within the neighborhood radius of a core point, then the point is determined to be a boundary point, such as... Figure 2 More specifically, the neighborhood radius ε and the minimum neighborhood number threshold MinPts for core point determination are obtained; for any data point that is not an initial noise point, the number of data points N in the ε neighborhood is calculated; if N is greater than or equal to MinPts, the data point is determined to be a core point; if N is less than MinPts, the data point is determined to be a boundary point.

[0023] S2, set an asymmetric extended neighborhood for each boundary point, and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, then the noise point is reclassified as a confirmed noise point;

[0024] For each boundary point b, define an extended neighborhood with a radius k times the radius of the original neighborhood, where k > 1. Search for all initial noise points within this extended neighborhood. For each found initial noise point n, calculate the Euclidean distance d between the noise point and the boundary point b, and apply a Gaussian attenuation function to calculate the attenuation value. Sum the distance attenuation function values ​​of all initial noise points within the extended neighborhood of the boundary point b; the result is the noise correlation index of the boundary point b.

[0025] A pre-set empirical value is used as the first preset threshold. Iterate through all boundary points and check the noise correlation index calculated for each boundary point. If the noise correlation index of a boundary point is greater than... This indicates that the boundary point is associated with a large number of or very close initial noise points, and the boundary point itself is highly likely to be noise. Therefore, the boundary point category is changed from boundary point to confirmed noise point.

[0026] S3, If the initial noise point is located in the overlapping neighborhood of a group of boundary points, and the geometric mean of the noise correlation index of the group of boundary points is lower than the second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster of the core point with the closest weighted distance to the noise point; all confirmed noise points are isolated, and the data recording stability is calculated for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index;

[0027] Specifically, for an initial noise point n, find all boundary points in its extended neighborhood that contain it, forming a boundary point set B. Calculate the geometric mean of the noise correlation index of all boundary points in set B. If this geometric mean is lower than a preset second threshold... If the boundary points around the initial noise point are relatively stable, then the category of the initial noise point n is changed to a quadratic boundary point. Calculate the weighted distance from the quadratic boundary point n to all core points within each cluster. The weighted distance can be the Euclidean distance divided by the neighborhood density of the corresponding core point. Select the core point with the smallest weighted distance and assign the quadratic boundary point to the cluster to which that core point belongs.

[0028] Remove all data points marked as confirmed noise points from the dataset. Within the remaining clusters, assign a stability S to each data point. Set the stability S of all core points to the maximum value, e.g., 1. Set the stability S of all quadratic boundary points to the minimum value, e.g., 0.1. For each boundary point b, the stability S of that boundary point b is 1 / (1 + the sum of noise correlation indices), ensuring that boundary points with higher noise correlation indices have lower stability. Figure 3 .

[0029] S4. Based on the stability, the distance from each data point to the cluster centroid is corrected, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged.

[0030] For each cluster, the cluster centroid C is calculated, which is the average of the coordinates of all data points within the cluster in each dimension. For each data point p within the cluster, the Euclidean distance D between the data point and the cluster centroid C is calculated, and this distance is corrected using the data recording stability S of the data point to obtain the corrected distance. Preferably, the calculation formula is as follows: Within a cluster, the corrected distances of all data points are compared, and the data point with the smallest corrected distance is selected as the master data record for that cluster. All other data points within the cluster are considered redundant records, and this redundant record information can be deleted or marked as items to be merged, thereby completing the deduplication process.

[0031] In an optional embodiment, calculating the neighborhood density of the data points and the local attribute fluctuation based on the periodicity of electricity consumption characteristics includes:

[0032] For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

[0033] Specifically, for a specified data point, such as user A, electricity consumption records for 36 months, from January 2020 to December 2022, are obtained. The data is grouped by month; for example, electricity consumption of 100 kWh in January 2020, 110 kWh in January 2021, and 105 kWh in January 2022 are grouped together, and the standard deviation of these three values ​​is calculated. The same calculation is performed on the data from February to December, resulting in 12 standard deviation values.

[0034] The arithmetic mean of the 12 standard deviations is calculated, and the calculated value is defined as the local attribute volatility of user A's data point.

[0035] In an optional embodiment, the step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes:

[0036] Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

[0037] Choose a boundary point B and determine its K nearest neighbors, assuming K is 5. Compare the densities of boundary point B with those of its five neighbors, and find the neighbors whose density values ​​are lower than B's. For example, [the text abruptly ends here, likely due to an incomplete sentence or missing information]. and Asymmetric expansion of the neighborhood only occurs towards... and Two low-density directions are defined, and the neighborhood range is a fan-shaped region centered at B with a preset value ε, such as 1.5. All points marked as initial noise points are searched within this asymmetric extended neighborhood. Assume two noise points are found. and Measure and record B. Euclidean distance It is 0.8, B to Euclidean distance The value is 1.2. The distance attenuation function value for each noise point is calculated according to the formula. The value is approximately 0.527. The value is approximately 0.237. Adding the distance attenuation function values ​​of all the noise points together yields 0.764, which is the noise correlation index of boundary point B.

[0038] In an optional embodiment, if the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then reclassifying the initial noise point as a secondary boundary point includes:

[0039] If the geometric mean of the noise correlation index of the group boundary points is less than the second preset threshold, the initial noise point will be reclassified as a secondary boundary point.

[0040] Specifically, an initial noise point N is identified, and the boundary points to which all asymmetric extended neighborhoods containing this noise point belong are determined. It is assumed that the noise point N also falls on a boundary point. , and If the three boundary points are within the neighborhood of a given point, then they form an association group.

[0041] Obtain the noise correlation index of each of the three boundary points, for example... The index is 0.08. The index is 0.05. The exponent is 0.12. The geometric mean of this set of exponents is calculated, which is approximately 0.078. The calculated geometric mean of 0.078 is compared with a preset second threshold of 0.1. 0.078 < 0.1, satisfying the condition, and the initial noise point N is reclassified as a quadratic boundary point.

[0042] In an optional embodiment, the step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes:

[0043] The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise point is assigned to the cluster containing the core point.

[0044] Specifically, the spatial coordinates and average electricity consumption of point N are normalized to a value between 0 and 1. Simultaneously, the core points already existing in the dataset... and It also underwent the same normalization process. (Based on the core point) For example, assuming k1=0.6 and k2=0.4, calculate N and Normalized Euclidean distance between The absolute value of the difference between the normalized mean electricity consumption is 0.3. The value is 0.5. According to the formula, the weighted distance is 0.38. Calculate the distance from N to... The distance, assuming It is 0.4. If the value is 0.2, then the weighted distance is 0.32. Comparing the two weighted distances, 0.32 is less than 0.38, indicating that point N is closer to the core point. Therefore, the quadratic boundary point N is assigned to the core point. In the cluster it belongs to.

[0045] To assign a stability value to different types of points in the dataset, in an optional embodiment, the boundary point stability is inversely proportional to the noise correlation index, including:

[0046] Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point.

[0047] Specifically, for all points identified as core points, the stability of these points is set to the highest value of 1.0. For all points reclassified as secondary boundary points, the stability of these points is set to a lower fixed value of 0.1. For the original boundary points, the stability of these points needs to be calculated using a formula. For example, for a boundary point B, the noise correlation index of that point is obtained. The value is 2.5. Substituting into the above formula, the stability of boundary point B is calculated. The value is approximately 0.357. If the noise correlation index of another boundary point C is low, for example, 0.5, then the stability of that point is calculated to be 0.7. The less noise a boundary point is associated with, the higher its stability.

[0048] To correct the distance from each data point to the centroid of the cluster to which the data point belongs, in an optional embodiment, correcting the distance from each data point to the cluster centroid based on the stability includes:

[0049] Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

[0050] Specifically, a data point P is selected, and the original Euclidean distance from it to the centroid C of the cluster containing the data point is calculated. Assume the distance is 10 units. Obtain the data record stability S for data point P. For example, if P is a core point, the stability S is 1.0; if P is a boundary point, the calculated stability S is 0.5; if P is a quadratic boundary point, the stability S is 0.1. Calculate the corrected distance using the formula. For a core point P, the corrected distance is 10. For a boundary point P, the corrected distance is 20. For a quadratic boundary point P, the corrected distance is 100. The lower the stability of a point, the farther away it is conceptually from the cluster center.

[0051] In the second embodiment, the present invention also proposes a power marketing data cleaning and deduplication system, comprising the following modules:

[0052] The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point.

[0053] The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point.

[0054] The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster containing the core point whose weighted distance is closest to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold; isolate all confirmed noise points and calculate the data record stability for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index;

[0055] The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records.

[0056] In an optional embodiment, calculating the neighborhood density of the data points and the local attribute fluctuation based on the periodicity of electricity consumption characteristics includes:

[0057] For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

[0058] In an optional embodiment, the step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes:

[0059] Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

[0060] In an optional embodiment, if the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then reclassifying the initial noise point as a secondary boundary point includes:

[0061] If the geometric mean of the noise correlation index of the group boundary points is less than the second preset threshold, the initial noise point will be reclassified as a secondary boundary point.

[0062] In an optional embodiment, the step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes:

[0063] The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise point is assigned to the cluster containing the core point.

[0064] In an optional embodiment, the boundary point stability is inversely proportional to the noise correlation index, including:

[0065] Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point.

[0066] In an optional embodiment, correcting the distance from each data point to the cluster centroid based on the stability includes:

[0067] Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for cleaning and deduplicating electricity marketing data, characterized in that, Includes the following steps: For each data point in the electricity marketing data, calculate the neighborhood density and the local attribute fluctuation based on the periodicity of electricity consumption. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. An asymmetric extended neighborhood is set for each boundary point, and the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point is calculated as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. If the initial noise point is located in the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than the second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster of the core point with the closest weighted distance to the noise point. Isolate all confirmed noise points, and calculate the data recording stability for each data point within each cluster consisting of core points, boundary points, and secondary boundary points. The stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index. The distance from each data point to the cluster centroid is corrected based on the stability, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged. The step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes: The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise points are assigned to the cluster containing the core point. The boundary point stability is inversely proportional to the noise correlation index, including: Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point; The step of correcting the distance from each data point to the cluster centroid based on the stability includes: Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

2. The method according to claim 1, characterized in that, The calculation of the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption of the data points includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

3. The method according to claim 1, characterized in that, The step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

4. The method according to claim 1, characterized in that, The step of determining whether a data point is a core point or a boundary point based on its neighborhood density includes: Obtain the neighborhood radius ε and the minimum neighborhood number threshold MinPts for core point determination; For any data point that is not an initial noise point, calculate the number of data points N in its ε-neighborhood; If N is greater than or equal to MinPts, then the data point is determined to be a core point; If N is less than MinPts, then the data point is determined to be a boundary point.

5. A power marketing data cleaning and deduplication system, characterized in that, Includes the following modules: The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster of the core point with the closest weighted distance to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold. Isolate all confirmed noise points, and calculate the data recording stability for each data point within each cluster consisting of core points, boundary points, and secondary boundary points. The stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index. The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records. The step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes: The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise points are assigned to the cluster containing the core point. The boundary point stability is inversely proportional to the noise correlation index, including: Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point; The step of correcting the distance from each data point to the cluster centroid based on the stability includes: Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

6. The system according to claim 5, characterized in that, The calculation of the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption of the data points includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

7. The system according to claim 5, characterized in that, The step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

Images