A data cleaning method, system and device based on big data analysis

Abstract

The present invention relates to the field of data analysis and processing, and in particular to a data cleaning method, system and device based on big data analysis. Environmental data is obtained, and the environmental data includes multiple data points, each data point includes multiple attributes, and each attribute corresponds to an attribute value. According to the abnormality of the attribute value, the environmental data is preprocessed, and the weights of all data points are determined by analyzing the data of each data point in the neighborhood of the preprocessed environmental data; the weights of the data points are used as the weights of data interpolation, and the preprocessed environmental data are interpolated and completed to obtain the environmental data after data cleaning. The present invention solves the problem that the accuracy of environmental data interpolation is affected due to inaccurate weight judgment of data points during data cleaning. The present invention is more accurate in interpolating environmental data, ensuring the integrity and continuity of environmental data.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and processing, and in particular to a data cleaning method, system and equipment based on big data analysis. Background Art

[0002] The purpose of big data analysis is to extract valuable information and insights from a large amount of disorganized data. In order to ensure the accuracy and reliability of the analysis results, data cleaning is an essential step. Since environmental data is closely related to public health, the monitoring and analysis of pollutants can help prevent and control diseases. Therefore, it is particularly important to crawl, analyze and predict environmental data. However, the crawled environmental data may contain errors, anomalies, inconsistent formats, etc., which may lead to missing data.

[0003] Since the data usually obtained contains a lot of errors, duplications, incomplete data, etc., cleaning the data can effectively improve the quality of the data and ensure the consistency of the data. The cleaned data is more suitable for building and optimizing analysis models, which can improve the prediction accuracy and reliability of the model. The lack of data will lead to NAN outliers in the data set, resulting in inaccurate analysis and modeling of these data, or program errors and inability to continue running. Therefore, interpolation and completion of data is an important part of cleaning.

[0004] The problem in the prior art is that the sensors may be affected by equipment interference and electronic noise, resulting in inaccurate monitoring data and missing attribute values, which leads to incomplete data that is usually crawled. This affects the continuity, modeling, and prediction of environmental data. However, in actual operation, the following problems may occur: KNN interpolation requires that each data point is independent, but the changes in environmental data are correlated in time, which leads to inaccurate weight judgment of data points. If there are a large number of missing attributes in the data, the accuracy of the data will be reduced, the difference in missing distance calculation is not obvious, and the representativeness of the data points is inaccurate, which affects the weight judgment of the data points. Summary of the invention

[0005] In order to solve the technical problem that the accuracy of environmental data interpolation is affected by inaccurate weight judgment of data points during environmental data cleaning, the purpose of the present invention is to provide a data cleaning method, system and device based on big data analysis. The technical solutions adopted are as follows:

[0006] In a first aspect, the present invention provides a data cleaning method based on big data analysis, the method comprising: first, acquiring environmental data, the environmental data comprising a plurality of data points, each data point comprising a plurality of attributes, each attribute corresponding to an attribute value, and preprocessing the environmental data according to abnormal conditions of the attribute values ​​to obtain preprocessed environmental data; then, determining weights of all data points by analyzing data of each data point in a neighborhood of the preprocessed environmental data; using the weights of the data points as weights for data interpolation, and interpolating and completing the preprocessed environmental data to obtain environmental data after data cleaning.

[0007] The present invention first pre-processes the environmental data, then performs multi-angle analysis on each attribute of each data point in the environmental data to determine the weight of each data point, and finally interpolates and completes the environmental data based on the newly obtained data point weight as the weight of data interpolation. The accuracy of the interpolated data is improved by adjusting the weight of the data point, making the interpolation of the environmental data more accurate.

[0008] Furthermore, according to the abnormality of the attribute value, the environmental data is preprocessed, specifically: the attribute value of each attribute of each data point is judged one by one, and if the attribute value is an abnormal value, the attribute value corresponding to the abnormal value is set to a null value.

[0009] Furthermore, the method includes: determining the weight of the data point based on the reference value of the data point and the importance of the data point; determining the initial reference value of the data point by analyzing the data of each data point in the preprocessed environmental data within the time series neighborhood; correcting the initial reference value and determining the reference value of the data point by analyzing the data of each data point in the preprocessed environmental data within the attribute neighborhood; determining the importance of the data point by analyzing the data of each data point in the preprocessed environmental data within the spatial neighborhood.

[0010] Furthermore, by analyzing the data of each data point in the preprocessed environmental data within the time series neighborhood, the initial reference value of the data point is determined, specifically including: in time series, constructing a time series neighborhood for the ith data point, analyzing the missing attribute values ​​of the jth attribute of each data point in the time series neighborhood, and obtaining the initial reference value of the jth attribute of the ith data point, where i and j are natural numbers.

[0011] Furthermore, by analyzing the data of each data point in the preprocessed environmental data within the attribute neighborhood, the initial reference value is corrected to obtain the reference value of all data points, specifically including: constructing an attribute neighborhood for all attributes of the ith data point, and correcting the initial reference value of the jth attribute of the ith data point according to the correlation between the jth attribute and other attributes in the attribute neighborhood to obtain the reference value of the jth attribute of the ith data point; obtaining the reference value of the ith data point according to the reference value of the jth attribute of the ith data point and the overall missing status of the attributes of the ith data point; traversing all data points to obtain the reference value of the data point.

[0012] Furthermore, the importance of the data point is determined by analyzing the data of each data point in the spatial neighborhood of the preprocessed environmental data, specifically including: spatially, constructing a spatial neighborhood for the yth data point, analyzing the spatial neighborhood where the data point is missing, analyzing the spatial position relationship between the complete data points and the missing data points in the spatial neighborhood, and then combining the reference value of the yth data point to obtain the importance of the yth data point, where y is a natural number; traversing all data points to obtain the importance of the data point.

[0013] Furthermore, the weight of the data point is determined based on the reference value of the data point and the importance of the data point, specifically including: for a complete data point, the reference value of the complete data point is used as the weight of the data point; for an incomplete data point, the importance of the incomplete data point is used as the weight of the data point.

[0014] Furthermore, the weight of the data point is used as the weight of data interpolation, and the preprocessed environmental data is interpolated and completed using the KNN algorithm to obtain the environmental data after data cleaning.

[0015] In a second aspect, the present invention provides a data cleaning system based on big data analysis, including: a data acquisition module: used to obtain environmental data, the environmental data includes multiple data points, each data point includes multiple attributes, each attribute corresponds to an attribute value, and the environmental data is preprocessed according to the abnormal situation of the attribute value to obtain the preprocessed environmental data; a feature analysis module: used to analyze the preprocessed environmental data to obtain the weight of the data point; wherein the weight of the data point is determined according to the reference value of the data point and the importance of the data point; by analyzing the data of each data point in the preprocessed environmental data in a time series neighborhood, the initial reference value of the data point is determined; by analyzing the data of each data point in the preprocessed environmental data in an attribute neighborhood, the initial reference value is corrected to determine the reference value of the data point; by analyzing the data of each data point in the preprocessed environmental data in a spatial neighborhood, the importance of the data point is determined; a data completion module: used to use the weight of the data point as the weight of data interpolation, and use the KNN algorithm to interpolate and complete the preprocessed environmental data to obtain the environmental data after data cleaning.

[0016] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the processor calls computer execution instructions in the memory to execute a data cleaning method based on big data analysis according to the first aspect.

[0017] The present invention has the following beneficial effects:

[0018] The present invention analyzes the correlation and importance between data points, specifically reflects the correlation and importance between data points through the attribute values ​​of the data points, obtains the reference value and importance of the data points, and then determines the weight of the data points based on this to interpolate the environmental data, making the interpolation of the environmental data more accurate.

[0019] The present invention determines the reference value of environmental data by considering its time series characteristics, corrects the reference value by the correlation between data point attributes, determines the importance of data points by the distribution of missing data points and complete data points in the neighborhood, and uses the importance of incomplete data points and the reference value of complete data points as weights to interpolate and complete environmental data using the KNN algorithm. The accuracy of interpolated data is improved by adjusting the weights of data points.

[0020] The present invention ensures the integrity and continuity of environmental data by cleaning and interpolating environmental data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flow chart of a data cleaning method based on big data analysis provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of various attributes within an attribute neighborhood in one embodiment of the present invention;

[0024] Figure 3 A structural diagram of a data cleaning system based on big data analysis provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the data cleaning method, system and device based on big data analysis proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0027] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0028] The following is a detailed description of a data cleaning method, system and device based on big data analysis provided by the present invention in conjunction with the accompanying drawings.

[0029] See also Figure 1 , which shows a flow chart of a data cleaning method based on big data analysis provided by an embodiment of the present invention, which determines the weight of the data points by introducing the correlation and importance between the data points, so that the interpolation of the environmental data is more accurate.

[0030] Specifically, the method embodiment of the present invention is a data cleaning method based on big data analysis, comprising:

[0031] First, environmental data is obtained. The environmental data in this embodiment refers to spatial quality data, including attributes such as temperature, humidity, and pollution. The environmental data includes a number of data points, which have a certain continuity in time and space. Each data point includes multiple attributes, such as temperature, humidity, pollution, etc. Each attribute has a corresponding attribute value, which is displayed in digital form. According to the abnormality of the attribute value, the environmental data is preprocessed to obtain the preprocessed environmental data.

[0032] The specific content of preprocessing environmental data is: the abnormal situation of attribute value refers to the situation where the attribute value is an abnormal value, which may be caused by errors, exceptions, inconsistent formats, etc. The attribute value of the abnormal value is set to a null value, that is, the attribute value is defined as missing. The attribute value of each attribute of each data point of the environmental data point is judged and preprocessed in turn to obtain the preprocessed environmental data.

[0033] Environmental data is crucial for environmental data monitoring because it can reflect air quality characteristics and affect people's travel and health. Crawl relevant data such as temperature, humidity, and pollutant concentration. Each crawled data point is obtained, and each data point has multiple attributes. Through anomaly detection, the abnormal value corresponding to each data point attribute is obtained, and the monitored abnormal value is set to a null value, that is, this attribute is missing, thereby completing the preprocessing.

[0034] Secondly, the weights of all data points are determined by analyzing the data of each data point in the neighborhood of the preprocessed environmental data, wherein the weights of the data points are determined according to the reference value and importance of the data points.

[0035] Due to the simple principle of the KNN algorithm, the small number of parameters to be set, and its flexible adjustment features and adaptability to the local characteristics of the data, the KNN algorithm is often used to interpolate and complete the monitored environmental data. When using the KNN algorithm to interpolate data, it is assumed that each data point is independent, without considering the continuity of the monitored environmental data in time series. Therefore, the reference value of the data point is determined by combining the characteristics that the monitored environmental data should meet the continuity in time series and the consistency of changes between adjacent data points; and then the weight of the data point is determined by combining the possibility that the range of the missing data point falls into the k neighborhood.

[0036] The initial reference value of the data point is determined by analyzing the data of each data point in the time series neighborhood in the preprocessed environmental data. In terms of time series, a time series neighborhood is constructed for the ith data point, and the missing attribute values ​​of the jth attribute of each data point in the time series neighborhood are analyzed to obtain the initial reference value of the jth attribute of the ith data point, where i and j are natural numbers.

[0037] Determine the reference value of the data.

[0038] Since the original KNN algorithm regards the environmental data sampled at each moment in the k-neighborhood as independent data points, there is no correlation between the data points. However, the collected environmental data is time series data, which shows strong time dependence. That is, the current air quality is affected by the past air quality, such as the diffusion of pollution sources and weather changes, which are continuous in time. Therefore, the adjacent data points in the time series of the monitored air quality data (i.e., environmental-related data or environmental data) should be stable.

[0039] Select the i-th data point of the air quality data in the time series , for the selected i-th data point The jth attribute of is analyzed. A total of 2N+1 data points in a neighborhood with a radius of N in the time series are selected for analysis, and the formula is constructed as follows:

[0040]

[0041]

[0042] Indicates the attribute value corresponding to the jth attribute of the ith data point. If the value is missing (i.e., a null value), it does not participate in the corresponding calculation. It represents the difference between two attribute values ​​of the same attribute of adjacent data points in the time series. If there is a missing value between two adjacent attribute values ​​in the time series, the missing value is skipped and the calculation with the next non-missing attribute value is performed. Indicates the mean value of the attribute value difference calculation within the time series neighborhood. Similarly, if there are missing values, they will not be included in the statistics (that is, in the mean calculation, they will not be included in the denominator statistics). It reflects the change trend information corresponding to the attribute value of the data point, and reflects the change trend information of the jth attribute of the i-th data point. The closer the value is to 0, the more stable the change of the environmental attribute is. On the contrary, the larger the value is, the more drastic the change of the temporal neighborhood is.

[0043] The variance represents the changing trend of the attribute value of the data point, reflecting the stability of the data point in the time series neighborhood. The smaller the variance, the more consistent the changing trend information of the attribute value in the time series neighborhood. Represents the average change trend within the time series neighborhood. It represents the loss rate of attribute j in the data point in the time series neighborhood, that is, the loss of the j-th attribute of the data point in the time series neighborhood constructed for the ith data point. The value is obtained by dividing the number of missing attributes in the time series neighborhood by the total number of attributes in the time series neighborhood, which reflects the missing status of the attribute. The larger the value, the more serious the data missing is, so the importance of the attribute is reduced, which in turn affects the reference value of the attribute.

[0044] Indicates the reference value of the jth attribute of the ith data point, reflecting the consistency characteristics of the continuous changes in the environmental data of the attribute in the time series neighborhood corresponding to the data point The more obvious it is, the greater the reference value; the weaker the missing situation, the greater the reference value.

[0045] This determines the initial reference values ​​corresponding to the different attributes of each data point in the environmental data.

[0046] However, the collected attributes are not independent, and there is correlation between attributes. For example, information such as humidity will also affect air quality information. Increased humidity causes pollution to aggregate into larger particles and sink, reducing the concentration of pollutants in the air, thereby improving air quality. Therefore, there is correlation between attributes. Therefore, if there is a missing attribute in environmental data, if the attribute is correlated with other attributes, the missing data of the attribute can be represented by the attribute values ​​of other attributes.

[0047] However, the local characteristics of the missing attributes in the k-neighborhood constructed by different data points are inconsistent, that is, the missing attributes of data points in different k-neighborhoods are different, resulting in large differences in the reference value obtained through the data characteristics in the time series in a certain local attribute neighborhood. Therefore, the reference value is corrected by using the performance of the attribute in the attribute neighborhood.

[0048] Therefore, although the data on the attribute of environmental data is seriously missing, resulting in a seriously low reference value, its reference value should be greater because it can reflect the changes in other attribute data; on the contrary, if a certain attribute is missing too little, but because the attribute cannot be used to judge other attribute data, the importance of the attribute of such data should be reduced. For example, humidity data and pollutant concentration data are negatively correlated. The drier the air, the higher the concentration of pollutants in the air. Therefore, although there may be more missing data in humidity or pollutant concentration data, the data of other attributes can be used to better predict the data, resulting in misjudgment of attribute importance.

[0049] By analyzing the data of each data point in the attribute neighborhood of the preprocessed environmental data, the initial reference value is corrected to determine the reference value of the data point; an attribute neighborhood is constructed for all attributes of the ith data point, and the initial reference value of the jth attribute of the ith data point is corrected according to the correlation between the jth attribute and other attributes in the attribute neighborhood to obtain the reference value of the jth attribute of the ith data point; the attribute neighborhood in this embodiment is a feature space neighborhood, such as Figure 2 Indicates.

[0050] The k-neighborhood of a data point is constructed as follows, removing attributes with missing data. Select any i-th data point, construct the k-neighborhood, and obtain the neighborhood data set. The construction formula is as follows:

[0051]

[0052] in, It represents the correlation between the original data attribute z and attribute j. The correlation is obtained by the Pearson correlation coefficient between attributes z and j in the attribute neighborhood corresponding to the data point. , represents the maximum and minimum correlations between the remaining attributes and the j-th attribute, Indicates the reference value of the attribute corresponding to the data point. Represents the reference value of the jth attribute of the ith data point after correction.

[0053] According to the reference value of the jth attribute of the ith data point and the overall missing state of the attributes of the ith data point, the reference value of the ith data point is obtained; all data points are traversed to obtain the reference values ​​of all data points.

[0054] Since the missing of all attributes cannot represent the missing of local attributes, the reference value of the data point is determined by using the reference value relationship of the attributes in the neighborhood. The formula is as follows:

[0055]

[0056]

[0057] Where i represents the i-th data point in the k-neighborhood; K represents the number of data points in the k-neighborhood constructed with the i-th data point as the center. represents the jth attribute value of the i-th data point in the neighborhood; represents the indicative function; Indicates missing values; if the i-th data point is not a missing value, the output result is 1; Indicates the modified attribute reference value, represents the number of complete attributes with no missing attributes in the k-neighborhood, It indicates that the missing attributes in the k-neighborhood reflect the reference value of the data point corresponding to the i-th data point in the neighborhood.

[0058] Determine the importance of the data point.

[0059] The importance of the data point is determined by analyzing the data of each data point in the spatial neighborhood of the preprocessed environmental data. In terms of space, a spatial neighborhood is constructed for the yth data point, and the spatial neighborhood where the data point is missing is analyzed. The spatial position relationship between the complete data point and the missing data point in the spatial neighborhood is analyzed, and then the importance of the yth data point is obtained by combining the reference value of the yth data point, where y is a natural number; all data points are traversed to obtain the importance of the data point.

[0060] However, in order to determine the specific location of the missing data points in the environmental data in space, it is necessary to remove the attributes with missing data, resulting in the KNN algorithm relying on fewer attributes for distance calculation, which leads to smaller distance differences between environmental data in different neighborhoods. If only a small number of attributes without missing attributes are considered for environmental data points with large differences, the environmental data will also have a high similarity. The difference between data points is not obvious when judged by only a small number of attributes, which in turn affects the precision and accuracy of environmental data interpolation. Therefore, in judging the importance of data points, it is necessary to consider the missing attributes of each data point in spatial position.

[0061] Since the missing attributes will make it impossible to determine the location of data points in space, the importance of data points is determined by using the range of missing attribute data points and the location information of the constructed spatial neighborhood.

[0062] Remove all data points with missing attribute values ​​to obtain a complete set of data points, determine the location information / coordinates of these complete data points in space, and also construct spatial neighborhoods for such data points (without missing attributes). For missing data, due to incomplete data, the location in this data space is not fixed, so the location of the data in space is a range information. The importance of the data point is determined by the proportion of the range of the data in the k-neighborhood.

[0063] If there is no missing data range in the spatial neighborhood at this time, the importance of the data point determined by this is meaningless. Therefore, only the spatial neighborhood containing the missing data range Select a k-neighborhood space with missing data (The central data point of the neighborhood is y), and the construction formula is as follows:

[0064]

[0065] in Indicates missing data points The area formed by the corresponding position range, represents the total number of missing data points, Represents the constructed k-neighborhood area; Represents the intersection of two regions; if there is no possible range of missing data points in the region, the value is 0; the larger the area ratio, the greater the importance of the data point y; i represents a complete data point in the intersection, represents the total number of complete data points, Represents the reference value of data point i. The greater the reference value in the neighborhood, the greater the importance of the corresponding data point. Represents the number of complete data points in the intersection. Indicates the relative importance of complete data points y and missing data points x; Indicates the reference value of data point y; Indicates the importance of data point y.

[0066] The weight of a data point is determined based on the reference value of the data point and the importance of the data point. Specifically, for a complete data point, the reference value of the complete data point is used as the weight of the data point; for an incomplete data point, the importance of the incomplete data point is used as the weight of the data point. At this point, the weight of the data point is determined.

[0067] If it is not a complete data point, its importance is equal to its reference value, and the importance is used as the weight in the KNN interpolation algorithm.

[0068] At this point, the weights of the data points are determined.

[0069] Finally, the weight of the data point is used as the weight of data interpolation, and the KNN algorithm is used to interpolate and complete the preprocessed environmental data to obtain the environmental data after data cleaning. The weight of KNN data interpolation is determined by the temporal characteristics of the data point passing through the environmental data and the missing situation of the neighborhood. After obtaining the weight, the weighted KNN interpolation algorithm is used to complete the interpolation of the data, and the calculated weighted average value is filled into the missing position.

[0070] Based on the same inventive concept as the method embodiment, the embodiment of the present application also provides a data cleaning system based on big data analysis, such as Figure 3 As shown, the system specifically includes:

[0071] Data acquisition module: used to obtain environmental data. Environmental data includes multiple data points, each data point includes multiple attributes, each attribute has an attribute value. According to the abnormality of the attribute value, the environmental data is preprocessed to obtain the preprocessed environmental data. The specific process of preprocessing is: the attribute value of each attribute of each data point is judged one by one. If the attribute value is an abnormal value, the attribute value corresponding to the abnormal value is set to a null value.

[0072] Feature analysis module: used to analyze the pre-processed environmental data to obtain the weight of the data point; wherein the weight of the data point is determined based on the reference value of the data point and the importance of the data point.

[0073] The initial reference value of the data point is determined by analyzing the data of each data point in the time series neighborhood in the preprocessed environmental data. In terms of time series, a time series neighborhood is constructed for the i-th data point, and the missing attribute values ​​of the j-th attribute of each data point in the time series neighborhood are analyzed to obtain the initial reference value of the j-th attribute of the i-th data point, where i and j are natural numbers.

[0074] By analyzing the data of each data point in the attribute neighborhood of the preprocessed environmental data, the initial reference value is corrected and the reference value of the data point is determined. The attribute neighborhood is constructed for all attributes of the ith data point. According to the correlation between the jth attribute and other attributes in the attribute neighborhood, the initial reference value of the jth attribute of the ith data point is corrected to obtain the reference value of the jth attribute of the ith data point; according to the reference value of the jth attribute of the ith data point and the overall missing state of the attributes of the ith data point, the reference value of the ith data point is obtained; all data points are traversed to obtain the reference value of the data point.

[0075] The importance of the data point is determined by analyzing the data of each data point in the spatial neighborhood of the preprocessed environmental data. In terms of space, a spatial neighborhood is constructed for the yth data point, and the spatial neighborhood where the data point is missing is analyzed. The spatial position relationship between the complete data point and the missing data point in the spatial neighborhood is analyzed, and then the importance of the yth data point is obtained by combining the reference value of the yth data point, where y is a natural number; all data points are traversed to obtain the importance of the data point.

[0076] The weight of a data point is determined based on the reference value of the data point and the importance of the data point, specifically including: for a complete data point, the reference value of the complete data point is used as the weight of the data point; for an incomplete data point, the importance of the incomplete data point is used as the weight of the data point.

[0077] Data completion module: It is used to use the weight of the data point as the weight of data interpolation, and use the KNN algorithm to interpolate and complete the preprocessed environmental data to obtain the environmental data after data cleaning.

[0078] Based on the same inventive concept as the method embodiment, the embodiment of the present application also provides an electronic device, such as Figure 4 As shown, the electronic device is used to perform the above Figure 1 The method embodiment shown is a data cleaning method based on big data analysis. The relevant features can be found in the above method embodiment and will not be repeated here.

[0079] It should be noted that the sequence of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A data cleaning method based on big data analysis, characterized in that: The method comprises: Acquire environmental data, the environmental data including multiple data points, each data point including multiple attributes, each attribute corresponding to an attribute value, and preprocess the environmental data according to an abnormality of the attribute value to obtain preprocessed environmental data; Determine the weights of all data points by analyzing the data of each data point in the neighborhood of the preprocessed environmental data; Using the weight of the data point as the weight of data interpolation, interpolating and completing the preprocessed environmental data to obtain the environmental data after data cleaning; The step of preprocessing the environmental data is as follows: judging the attribute value of each attribute of each data point one by one, and if the attribute value is an abnormal value, setting the attribute value corresponding to the abnormal value to a null value; The method for determining the weights of all data points is as follows: the weights of the data points are determined according to the reference value of the data points and the importance of the data points; the initial reference value of the data points is determined by analyzing the data of each data point in the preprocessed environmental data in the time series neighborhood; the reference value of the data point is determined by correcting the initial reference value by analyzing the data of each data point in the preprocessed environmental data in the attribute neighborhood; the importance of the data point is determined by analyzing the data of each data point in the preprocessed environmental data in the space neighborhood; The method for determining the initial reference value of the data point is as follows: in time series, a time series neighborhood is constructed for the i-th data point, and the missing attribute value of the j-th attribute of each data point in the time series neighborhood is analyzed to obtain the initial reference value of the j-th attribute of the i-th data point, where i and j are natural numbers; The method for correcting the initial reference value to obtain the reference value of all data points is as follows: constructing an attribute neighborhood for all attributes of the ith data point, and correcting the initial reference value of the jth attribute of the ith data point according to the correlation between the jth attribute and other attributes in the attribute neighborhood to obtain the reference value of the jth attribute of the ith data point; obtaining the reference value of the ith data point according to the reference value of the jth attribute of the ith data point and the overall missing state of the attributes of the ith data point; traversing all data points to obtain the reference value of the data point; The importance of the data point is determined by analyzing the data of each data point in the spatial neighborhood of the preprocessed environmental data, specifically including: spatially, constructing a spatial neighborhood for the yth data point, analyzing the spatial neighborhood where the data point is missing, analyzing the spatial position relationship between the complete data point and the missing data point in the spatial neighborhood, and then combining the reference value of the yth data point to obtain the importance of the yth data point, where y is a natural number; traversing all data points to obtain the importance of the data point; The weight of the data point is determined according to the reference value of the data point and the importance of the data point, specifically including: for a complete data point, the reference value of the complete data point is used as the weight of the data point; for an incomplete data point, the importance of the incomplete data point is used as the weight of the data point; The weight of the data point is used as the weight of data interpolation, and the KNN algorithm is used to interpolate and complete the preprocessed environmental data to obtain the environmental data after data cleaning.

2. A data cleaning system based on big data analysis, characterized in that: The system is used to implement a data cleaning method based on big data analysis as described in claim 1, and the system specifically includes: Data acquisition module: used to obtain environmental data, the environmental data includes multiple data points, each data point includes multiple attributes, each attribute corresponds to an attribute value, and pre-processes the environmental data according to the abnormal situation of the attribute value to obtain the pre-processed environmental data; Feature analysis module: used to analyze the pre-processed environmental data to obtain the weight of the data point; wherein the weight of the data point is determined according to the reference value of the data point and the importance of the data point; by analyzing the data of each data point in the pre-processed environmental data in the time series neighborhood, the initial reference value of the data point is determined; by analyzing the data of each data point in the pre-processed environmental data in the attribute neighborhood, the initial reference value is corrected to determine the reference value of the data point; by analyzing the data of each data point in the pre-processed environmental data in the space neighborhood, the importance of the data point is determined; Data completion module: used to use the weight of the data point as the weight of data interpolation, and use the KNN algorithm to interpolate and complete the preprocessed environmental data to obtain the environmental data after data cleaning.

3. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor calls the computer execution instructions in the memory to execute the data cleaning method based on big data analysis described in claim 1.

Images