Power marketing data cleaning and duplicate removal method and system

Abstract

The invention provides an electric power marketing data cleaning and deduplication method and system, and the method comprises the steps: calculating the neighborhood density and local attribute fluctuation of a data point, and marking the data point as an initial noise point if the neighborhood density is lower than a threshold value and the fluctuation is higher than the threshold value; otherwise, dividing into core points or boundary points according to density; setting an asymmetric expansion neighborhood for the boundary point, calculating the sum of distance attenuation function values from an initial noise point to the boundary point in the neighborhood, and if the sum is higher than a first threshold value, re-classifying as a confirmation noise point; if the initial noise points are located in the overlapped neighborhood and the geometric mean value of the noise correlation indexes of the group of boundary points is lower than a second threshold value, the noise points are classified into secondary boundary points, and the secondary boundary points are classified into a nearest core point cluster; noise points are isolated and confirmed, and the stability is calculated in the cluster; and correcting the distance from the data point to the cluster centroid according to the stability, selecting the minimum corrected distance as a main data record, and combining redundant records.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data processing, and particularly relates to a power marketing data cleaning and deduplication method and system. BACKGROUND

[0002] A power marketing business system will accumulate a large amount of user power consumption data in the running process, and the data is the basis for load forecasting, user portrait analysis and marketing strategy formulation. At present, the cleaning method for such data includes a statistical method and a clustering method. The density clustering algorithm can discover clusters of arbitrary shape and does not need to specify the number of clusters in advance, but when identifying noise points, the traditional density algorithm is usually based on the spatial neighborhood density of data points, ignoring the business attributes unique to power marketing data. When processing power data with complex business logic and diverse data characteristics, it is easy to misjudge some normal but sparse area data points as noise, or cannot identify those data points with abnormal user behavior characteristics although the density is not low.

[0003] In density clustering, boundary points are very close to noise points in spatial distribution due to low neighborhood density, and it is difficult to distinguish whether the low-density points at the edge of the cluster are reasonable extensions of the cluster or noise that should be removed. In the data deduplication link, existing methods usually ignore the differences in data quality and stability of data points at different positions after removing noise. In particular, the reliability of the boundary point data record should be lower than that of the core point away from the noise area. Therefore, there is a lack of a deduplication mechanism that can combine the noise correlation degree and stability of data points, so that the deduplication result is easily affected by unreliable boundary points, and the selected main data record may not be the optimal representative, thereby affecting the quality of the data. SUMMARY

[0004] The application provides a power marketing data cleaning and deduplication method, which is used to solve the problem that the prior art lacks a deduplication mechanism that can combine the noise correlation degree and stability of data points, and includes the following steps: For each data point in the power marketing data, the neighborhood density of the data point and the local attribute fluctuation degree based on the periodicity of power consumption are calculated. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation degree is higher than the fluctuation degree threshold, the data point is determined as an initial noise point, otherwise the data point is determined as a core point or a boundary point according to the neighborhood density of the data point; An asymmetric extended neighborhood is set for each boundary point, and the sum of the distance decay function values of each initial noise point in the neighborhood to the boundary point is calculated as the noise correlation index of the boundary point. If the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point; If the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster containing the core point with the closest weighted distance to the noise point; all confirmed noise points are isolated, and within each cluster composed of core points, boundary points, and secondary boundary points, the data recording stability is calculated for each data point, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index; Based on the stability, the distance from each data point to the cluster centroid is corrected, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged.

[0005] Furthermore, this invention also relates to a power marketing data cleaning and deduplication system, comprising the following modules: The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster containing the core point whose weighted distance is closest to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold; isolate all confirmed noise points and calculate the data record stability for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index; The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records.

[0006] This invention identifies initial noise points by combining the neighborhood density of data points with the local attribute fluctuation based on the periodicity of electricity consumption. By constructing an asymmetric extended neighborhood for boundary points and calculating the noise correlation index of these boundary points, the degree of correlation between boundary points and noise points is represented. This allows boundary points affected by noise to be reclassified as confirmed noise points, while simultaneously restoring initial noise points in neighboring stable clusters as secondary boundary points, solving the problem of distinguishing between boundary points and noise points in traditional methods. In the data deduplication stage, the stability of data records, inversely proportional to the noise correlation index, is used to correct the distance from data points to the cluster centroid, ensuring that the selected master data records within the cluster are spatially representative, thereby improving the cleaning and deduplication effect of electricity marketing data. Attached Figure Description

[0007] Figure 1 A flowchart of the first embodiment; Figure 2 This is a schematic diagram of the initial classification of data points; Figure 3 This is a diagram illustrating the stability of data recording. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0009] In the first embodiment, the present invention proposes a method for cleaning and deduplicating electricity marketing data, such as... Figure 1 ,include: S1. For each data point in the electricity marketing data, calculate the neighborhood density and the local attribute fluctuation based on the periodicity of electricity consumption. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. Obtain the neighborhood radius and minimum number of neighborhood points. For any data point p, the neighborhood density of the data point is the number of data points within the neighborhood of the data point's neighborhood radius. For local attribute volatility, extract the data point p and all data points within the neighborhood of the data point's neighborhood radius in the same period dimension, such as daily electricity consumption data for 30 consecutive days, to form a local dataset. Calculate the standard deviation or coefficient of variation of all daily electricity consumption data in this local dataset as the local attribute volatility of point p.

[0010] The density threshold is preferably set to the minimum number of neighboring points. For a data point p, if the neighborhood density of the data point is less than the minimum number of neighboring points and the local attribute volatility of the data point is greater than a preset volatility threshold, then the point is marked as an initial noise point. If these two conditions are not met, the neighborhood density of the data point is checked. If the neighborhood density is greater than or equal to the minimum number of neighboring points, then the point is determined to be a core point; if the neighborhood density of the data point is less than the minimum number of neighboring points, but the point is within the neighborhood radius of a core point, then the point is determined to be a boundary point, such as... Figure 2 More specifically, the neighborhood radius ε and the minimum neighborhood number threshold MinPts for core point determination are obtained; for any data point that is not an initial noise point, the number of data points N in the ε neighborhood is calculated; if N is greater than or equal to MinPts, the data point is determined to be a core point; if N is less than MinPts, the data point is determined to be a boundary point.

[0011] S2, set an asymmetric extended neighborhood for each boundary point, and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, then the noise point is reclassified as a confirmed noise point; For each boundary point b, define an extended neighborhood with a radius k times the radius of the original neighborhood, where k > 1. Search for all initial noise points within this extended neighborhood. For each found initial noise point n, calculate the Euclidean distance d between the noise point and the boundary point b, and apply a Gaussian attenuation function to calculate the attenuation value. Sum the distance attenuation function values ​​of all initial noise points within the extended neighborhood of the boundary point b; the result is the noise correlation index of the boundary point b.

[0012] A pre-set empirical value is used as the first preset threshold. Iterate through all boundary points and check the noise correlation index calculated for each boundary point. If the noise correlation index of a boundary point is greater than... This indicates that the boundary point is associated with a large number of or very close initial noise points, and the boundary point itself is highly likely to be noise. Therefore, the boundary point category is changed from boundary point to confirmed noise point.

[0013] S3, If the initial noise point is located in the overlapping neighborhood of a group of boundary points, and the geometric mean of the noise correlation index of the group of boundary points is lower than the second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster of the core point with the closest weighted distance to the noise point; all confirmed noise points are isolated, and the data recording stability is calculated for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index; Specifically, for an initial noise point n, find all boundary points in its extended neighborhood that contain it, forming a boundary point set B. Calculate the geometric mean of the noise correlation index of all boundary points in set B. If this geometric mean is lower than a preset second threshold... If the boundary points around the initial noise point are relatively stable, then the category of the initial noise point n is changed to a quadratic boundary point. Calculate the weighted distance from the quadratic boundary point n to all core points within each cluster. The weighted distance can be the Euclidean distance divided by the neighborhood density of the corresponding core point. Select the core point with the smallest weighted distance and assign the quadratic boundary point to the cluster to which that core point belongs.

[0014] Remove all data points marked as confirmed noise points from the dataset. Within the remaining clusters, assign a stability S to each data point. Set the stability S of all core points to the maximum value, e.g., 1. Set the stability S of all quadratic boundary points to the minimum value, e.g., 0.1. For each boundary point b, the stability S of that boundary point b is 1 / (1 + the sum of noise correlation indices), ensuring that boundary points with higher noise correlation indices have lower stability. Figure 3 .

[0015] S4. Based on the stability, the distance from each data point to the cluster centroid is corrected, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged.

[0016] For each cluster, the cluster centroid C is calculated, which is the average of the coordinates of all data points within the cluster in each dimension. For each data point p within the cluster, the Euclidean distance D between the data point and the cluster centroid C is calculated, and this distance is corrected using the data recording stability S of the data point to obtain the corrected distance. Preferably, the calculation formula is as follows: Within a cluster, the corrected distances of all data points are compared, and the data point with the smallest corrected distance is selected as the master data record for that cluster. All other data points within the cluster are considered redundant records, and this redundant record information can be deleted or marked as items to be merged, thereby completing the deduplication process.

[0017] In an optional embodiment, calculating the neighborhood density of the data points and the local attribute fluctuation based on the periodicity of electricity consumption characteristics includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

[0018] Specifically, for a specified data point, such as user A, electricity consumption records for 36 months, from January 2020 to December 2022, are obtained. The data is grouped by month; for example, electricity consumption of 100 kWh in January 2020, 110 kWh in January 2021, and 105 kWh in January 2022 are grouped together, and the standard deviation of these three values ​​is calculated. The same calculation is performed on the data from February to December, resulting in 12 standard deviation values.

[0019] The arithmetic mean of the 12 standard deviations is calculated, and the calculated value is defined as the local attribute volatility of user A's data point.

[0020] In an optional embodiment, the step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

[0021] Choose a boundary point B and determine its K nearest neighbors, assuming K is 5. Compare the densities of boundary point B with those of its five neighbors, and find the neighbors whose density values ​​are lower than B's. For example, [the text abruptly ends here, likely due to an incomplete sentence or missing information]. and Asymmetric expansion of the neighborhood only occurs towards... and Two low-density directions are defined, and the neighborhood range is a fan-shaped region centered at B with a preset value ε, such as 1.5. All points marked as initial noise points are searched within this asymmetric extended neighborhood. Assume two noise points are found. and Measure and record B. Euclidean distance It is 0.8, B to Euclidean distance The value is 1.2. The distance attenuation function value for each noise point is calculated according to the formula. The value is approximately 0.527. The value is approximately 0.237. Adding the distance attenuation function values ​​of all the noise points together yields 0.764, which is the noise correlation index of boundary point B.

[0022] In an optional embodiment, if the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then reclassifying the initial noise point as a secondary boundary point includes: If the geometric mean of the noise correlation index of the group boundary points is less than the second preset threshold, the initial noise point will be reclassified as a secondary boundary point.

[0023] Specifically, an initial noise point N is identified, and the boundary points to which all asymmetric extended neighborhoods containing this noise point belong are determined. It is assumed that the noise point N also falls on a boundary point. , and If the three boundary points are within the neighborhood of a given point, then they form an association group.

[0024] Obtain the noise correlation index of each of the three boundary points, for example... The index is 0.08. The index is 0.05. The exponent is 0.12. The geometric mean of this set of exponents is calculated, which is approximately 0.078. The calculated geometric mean of 0.078 is compared with a preset second threshold of 0.1. 0.078 < 0.1, satisfying the condition, and the initial noise point N is reclassified as a quadratic boundary point.

[0025] In an optional embodiment, the step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes: The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise point is assigned to the cluster containing the core point.

[0026] Specifically, the spatial coordinates and average electricity consumption of point N are normalized to a value between 0 and 1. Simultaneously, the core points already existing in the dataset... and It also underwent the same normalization process. (Based on the core point) For example, assuming k1=0.6 and k2=0.4, calculate N and Normalized Euclidean distance between The absolute value of the difference between the normalized mean electricity consumption is 0.3. The value is 0.5. According to the formula, the weighted distance is 0.38. Calculate the distance from N to... The distance, assuming It is 0.4. If the value is 0.2, then the weighted distance is 0.32. Comparing the two weighted distances, 0.32 is less than 0.38, indicating that point N is closer to the core point. Therefore, the quadratic boundary point N is assigned to the core point. In the cluster it belongs to.

[0027] To assign a stability value to different types of points in the dataset, in an optional embodiment, the boundary point stability is inversely proportional to the noise correlation index, including: Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point.

[0028] Specifically, for all points identified as core points, the stability of these points is set to the highest value of 1.0. For all points reclassified as secondary boundary points, the stability of these points is set to a lower fixed value of 0.1. For the original boundary points, the stability of these points needs to be calculated using a formula. For example, for a boundary point B, the noise correlation index of that point is obtained. The value is 2.5. Substituting into the above formula, the stability of boundary point B is calculated. The value is approximately 0.357. If the noise correlation index of another boundary point C is low, for example, 0.5, then the stability of that point is calculated to be 0.7. The less noise a boundary point is associated with, the higher its stability.

[0029] To correct the distance from each data point to the centroid of the cluster to which the data point belongs, in an optional embodiment, correcting the distance from each data point to the cluster centroid based on the stability includes: Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

[0030] Specifically, a data point P is selected, and the original Euclidean distance from it to the centroid C of the cluster containing the data point is calculated. Assume the distance is 10 units. Obtain the data record stability S for data point P. For example, if P is a core point, the stability S is 1.0; if P is a boundary point, the calculated stability S is 0.5; if P is a quadratic boundary point, the stability S is 0.1. Calculate the corrected distance using the formula. For a core point P, the corrected distance is 10. For a boundary point P, the corrected distance is 20. For a quadratic boundary point P, the corrected distance is 100. The lower the stability of a point, the farther away it is conceptually from the cluster center.

[0031] In the second embodiment, the present invention also proposes a power marketing data cleaning and deduplication system, comprising the following modules: The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster containing the core point whose weighted distance is closest to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold; isolate all confirmed noise points and calculate the data record stability for each data point in each cluster composed of core points, boundary points and secondary boundary points, wherein the stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index; The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records.

[0032] In an optional embodiment, calculating the neighborhood density of the data points and the local attribute fluctuation based on the periodicity of electricity consumption characteristics includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

[0033] In an optional embodiment, the step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

[0034] In an optional embodiment, if the initial noise point is located within the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold, then reclassifying the initial noise point as a secondary boundary point includes: If the geometric mean of the noise correlation index of the group boundary points is less than the second preset threshold, the initial noise point will be reclassified as a secondary boundary point.

[0035] In an optional embodiment, the step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes: The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise point is assigned to the cluster containing the core point.

[0036] In an optional embodiment, the boundary point stability is inversely proportional to the noise correlation index, including: Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point.

[0037] In an optional embodiment, correcting the distance from each data point to the cluster centroid based on the stability includes: Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

[0038] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0039] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for cleaning and deduplicating electricity marketing data, characterized in that, Includes the following steps: For each data point in the electricity marketing data, calculate the neighborhood density and the local attribute fluctuation based on the periodicity of electricity consumption. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. An asymmetric extended neighborhood is set for each boundary point, and the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point is calculated as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. If the initial noise point is located in the overlapping neighborhood of a set of boundary points, and the geometric mean of the noise correlation index of the set of boundary points is lower than the second preset threshold, then the initial noise point is reclassified as a secondary boundary point and assigned to the cluster of the core point with the closest weighted distance to the noise point. Isolate all confirmed noise points, and calculate the data recording stability for each data point within each cluster consisting of core points, boundary points, and secondary boundary points. The stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index. Based on the stability, the distance from each data point to the cluster centroid is corrected, and then the data point with the smallest corrected distance is selected as the master data record within the cluster, and redundant data records are merged.

2. The method according to claim 1, characterized in that, The calculation of the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption of the data points includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

3. The method according to claim 1, characterized in that, The step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

4. The method according to claim 1, characterized in that, The step of determining whether a data point is a core point or a boundary point based on its neighborhood density includes: Obtain the neighborhood radius ε and the minimum neighborhood number threshold MinPts for core point determination; For any data point that is not an initial noise point, calculate the number of data points N in its ε-neighborhood; If N is greater than or equal to MinPts, then the data point is determined to be a core point; If N is less than MinPts, then the data point is determined to be a boundary point.

5. The method according to claim 1, characterized in that, The step of reclassifying the initial noise points into secondary boundary points and assigning them to the cluster containing the core point with the closest weighted distance to the noise points includes: The spatial coordinates and average electricity consumption of the data points are normalized. Based on the normalized data, the weighted distance from the initial noise point to each core point is calculated. The formula for calculating the weighted distance is as follows: ,in This is the normalized Euclidean distance. The absolute value of the difference between the normalized average electricity consumption values, where k1 and k2 are coefficients; (Select...) The smallest core point is identified, and the initial noise point is assigned to the cluster containing the core point.

6. The method according to claim 1, characterized in that, The boundary point stability is inversely proportional to the noise correlation index, including: Through formula Calculate the stability of the boundary points ,in The noise correlation index of the boundary point.

7. The method according to claim 1, characterized in that, The step of correcting the distance from each data point to the cluster centroid based on the stability includes: Using formula Calculate the corrected distance ,in S is the original Euclidean distance from the data point to the cluster centroid, and S is the data recording stability of the data point.

8. A system for cleaning and deduplicating electricity marketing data, characterized in that, Includes the following modules: The determination module is used to calculate the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption for each data point in the electricity marketing data. If the neighborhood density of the data point is lower than the density threshold and the local attribute fluctuation is higher than the fluctuation threshold, it is determined to be an initial noise point; otherwise, it is determined to be a core point or a boundary point based on the neighborhood density of the data point. The classification module is used to set an asymmetric extended neighborhood for each boundary point and calculate the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point; if the noise correlation index of the boundary point is higher than a first preset threshold, the noise point is reclassified as a confirmed noise point. The calculation module is used to reclassify the initial noise point as a secondary boundary point and assign it to the cluster of the core point with the closest weighted distance to the noise point if the initial noise point is located in the overlapping neighborhood of a set of boundary points and the geometric mean of the noise correlation index of the set of boundary points is lower than a second preset threshold. Isolate all confirmed noise points, and calculate the data recording stability for each data point within each cluster consisting of core points, boundary points, and secondary boundary points. The stability of the core point is a preset maximum value, the stability of the secondary boundary point is a preset minimum value, and the stability of the boundary point is inversely proportional to the noise correlation index. The merging module is used to correct the distance from each data point to the cluster centroid based on the stability, and then select the data point with the smallest corrected distance within the cluster as the master data record, and merge redundant data records.

9. The system according to claim 8, characterized in that, The calculation of the neighborhood density and local attribute fluctuation based on the periodicity of electricity consumption of the data points includes: For each data point, electricity consumption data for 36 consecutive months is extracted. Using a 12-month period, the standard deviation of electricity consumption for the same month over three consecutive years is calculated. The arithmetic mean of the standard deviations for the 12 months is then calculated to obtain the local attribute volatility of the data point.

10. The system according to claim 8, characterized in that, The step of setting an asymmetric extended neighborhood for each boundary point and calculating the sum of the distance attenuation function values ​​from each initial noise point in the neighborhood to the boundary point as the noise correlation index of the boundary point includes: Centered on the boundary point, a region with radius ε is defined as an asymmetric extended neighborhood in the direction where the density of the nearest neighbors of the boundary point K is lower than that of the boundary point; for any initial noise point within the neighborhood, the distance attenuation function value of the noise point is... , where d is the Euclidean distance from the initial noise point to the boundary point; the noise correlation index is obtained by summing the distance attenuation function values ​​of all initial noise points in the neighborhood.

Images