Data cleaning effect estimation method, device, terminal equipment and storage medium

Abstract

The present invention discloses a method, device, terminal device and storage medium for estimating the effect of data cleaning, the method comprising: acquiring the original data of a device and determining the attribute type of the original data; determining the data cleaning rule corresponding to the attribute type according to the attribute type; performing data cleaning on the original data using the data cleaning rule to obtain target reported data; estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data. The present invention performs data cleaning by being accurate to the attribute, matches cleaning rules with different attribute characteristics to data with different attributes, thereby improving the efficiency of data cleaning. At the same time, the present invention can also estimate the effect of data cleaning, achieve the effect of accurately controlling resource allocation and cost, and save data storage resources of IoT devices.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, terminal equipment and storage medium for estimating data cleaning effect. Background Art

[0002] Data cleaning refers to the process of identifying and correcting abnormal data in the data files to be reported according to certain cleaning rules. By cleaning the data to be reported by a large number of IoT devices, the quality of the reported data can be improved.

[0003] At present, IoT devices have the problem of high-frequency and large-volume raw data reporting. Since these raw data contain a series of abnormal data such as abnormal values, duplicate values, and time series discontinuity, it is often necessary to configure a sufficiently large data storage space. However, existing data processing equipment cannot accurately estimate the effect of data cleaning, cannot accurately optimize resource allocation and control costs, and wastes storage resources for IoT device data.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, device, terminal device and storage medium for estimating the effect of data cleaning are provided, aiming to solve the problem that the data processing equipment in the prior art cannot accurately estimate the effect of data cleaning, cannot accurately control the resource optimization configuration and cost, and wastes the storage resources of the Internet of Things device data.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for estimating data cleaning effect, wherein the method comprises:

[0008] Acquire raw data of a device and determine the attribute type of the raw data;

[0009] According to the attribute type, determining a data cleaning rule corresponding to the attribute type;

[0010] The data cleaning rule is used to clean the original data to obtain target reported data;

[0011] The data cleaning effect of the original data is estimated according to the first data information of the original data and the second data information of the target reported data.

[0012] In one implementation, acquiring raw data of a device and determining the attribute type of the raw data includes:

[0013] Acquire raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes;

[0014] Performing feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes;

[0015] According to the characteristic information, an attribute type of each of the target cleaning attributes is determined.

[0016] In one implementation, determining, according to the attribute type, a data cleaning rule corresponding to the attribute type includes:

[0017] Input the attribute type into the data cleaning engine module for data analysis to obtain a data cleaning model corresponding to the attribute type;

[0018] According to the data cleaning model, a data cleaning rule corresponding to the attribute type is determined.

[0019] In one implementation, the adopting the data cleaning rule to clean the original data to obtain target reported data includes:

[0020] Using the data cleaning rule to clean the original data, to obtain target cleaned data of the original data;

[0021] The target reporting data is obtained according to the original data and the target cleansing data.

[0022] In one implementation, the data cleaning rule is used to clean the original data to obtain target reported data, and the method includes:

[0023] Second data information of the target reported data is obtained, where the second data information includes a data size, a reporting frequency, and the data cleaning rule of the target reported data.

[0024] In one implementation, estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data includes:

[0025] Inputting the first data information and the second data information into a preset mathematical model for data processing to obtain a data cleaning rule;

[0026] According to the data cleaning rule, the data cleaning effect of the original data is estimated.

[0027] In one implementation, after estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data, the method includes:

[0028] The data cleaning effect is used as a data sample and input into the data cleaning model for optimization training, and the data cleaning model is updated in real time.

[0029] In a second aspect, an embodiment of the present invention further provides a device for estimating data cleaning effect, wherein the device comprises:

[0030] An attribute type determination module, used to obtain raw data of a device and determine the attribute type of the raw data;

[0031] A data cleaning rule determination module, used to determine the data cleaning rule corresponding to the attribute type according to the attribute type;

[0032] A target reporting data determination module, configured to perform data cleaning on the original data using the data cleaning rule to obtain target reporting data;

[0033] The data cleaning effect estimation module is used to estimate the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data.

[0034] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device comprises a memory, a processor, and a data cleaning effect estimation program stored in the memory and executable on the processor, and when the processor executes the data cleaning effect estimation program, the steps of the data cleaning effect estimation method of any one of the above-mentioned schemes are implemented.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a data cleaning effect estimation program is stored on the computer-readable storage medium, and when the data cleaning effect estimation program is executed by a processor, the steps of the data cleaning effect estimation method described in any one of the above-mentioned schemes are implemented.

[0036] Beneficial effects: Compared with the prior art, the present invention provides a method, device, terminal device and storage medium for estimating the effect of data cleaning. First, the original data of the device is obtained, and the attribute type of the original data is determined. Then, according to the attribute type, the data cleaning rule corresponding to the attribute type is determined. Next, the original data is cleaned using the data cleaning rule to obtain the target reported data. Finally, according to the first data information of the original data and the second data information of the target reported data, the data cleaning effect of the original data is estimated. It can be seen that the present invention improves the efficiency of data cleaning by performing data cleaning accurately to the attributes and matching cleaning rules with different attribute characteristics for data with different attributes. At the same time, the present invention can also estimate the effect of data cleaning, achieve the effect of precise control of resource allocation and cost, and save data storage resources of Internet of Things devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a specific implementation of a method for estimating data cleaning effects provided by an embodiment of the present invention.

[0038] Figure 2 A functional schematic diagram of a device for estimating data cleaning effects provided in an embodiment of the present invention.

[0039] Figure 3 A functional block diagram of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0041] Those skilled in the art will appreciate that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms, and the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when an element is "connected" or "coupled" to another element, it may be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0042] The present embodiment provides a method, apparatus, terminal device and storage medium for estimating the effect of data cleaning. When implemented, the present embodiment first obtains the original data of the device and determines the attribute type of the original data. Then, according to the attribute type, the data cleaning rule corresponding to the attribute type is determined. Next, the original data is cleaned using the data cleaning rule to obtain the target reported data. Finally, according to the first data information of the original data and the second data information of the target reported data, the data cleaning effect of the original data is estimated. It can be seen that the present embodiment performs data cleaning by accurately determining the attributes, matches cleaning rules with different attribute characteristics to data with different attributes, and improves the efficiency of data cleaning. At the same time, the present embodiment can also estimate the effect of data cleaning, achieve the effect of precise control of resource allocation and cost, and save data storage resources of IoT devices.

[0043] The data cleaning effect estimation method of this embodiment can be applied to a terminal device, which can be a preset mobile terminal such as a mobile phone, a tablet, a computer, or an intelligent device such as a smart TV. In this embodiment, the terminal device can be an external device connected to the data cleaning effect estimation device, or a device built into the data cleaning effect estimation device. Figure 1 As shown in , the method for estimating the data cleaning effect of this embodiment includes the following steps:

[0044] Step S100: Acquire original data of a device and determine the attribute type of the original data.

[0045] Generally speaking, devices often store various raw data to be reported. When these devices are connected to the controller, the various raw data can be cleaned in the controller, and then the cleaned data can be reported to the corresponding data receiving platform. This can effectively reduce the bandwidth cost during data transmission, reduce the platform's data storage cost, and improve the platform's data analysis efficiency.

[0046] Among them, the original data to be cleaned received by the controller often contains data of multiple attribute types, and the current data cleaning technology cannot accurately identify single attribute data, nor can it recommend the most appropriate data cleaning rules for each attribute data. Therefore, in this embodiment, it is first necessary to separate the attributes of the original data uploaded by the acquired device to be cleaned, and obtain the attribute type of each data contained in the original data, and then find and recommend appropriate data cleaning rules for each attribute data. Preferably, the controller that receives the original data to be cleaned in this embodiment is a Shenzhen Kaihong Super Device Controller based on the KaiHongOS system.

[0047] In one implementation, step S100 of this embodiment specifically includes the following steps:

[0048] Step S101, obtaining raw data of the device, performing attribute separation on the raw data, and obtaining a number of target cleaning attributes;

[0049] Step S102, performing feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes;

[0050] Step S103: Determine the attribute type of each of the target cleaning attributes according to the characteristic information.

[0051] Since the common types of data to be reported include: numerical fluctuation data, that is, filtering invalid or slightly changing data points according to the set threshold; numerical change data / continuous growth data, that is, only retaining the cumulative value of the last state; numerical repetitive data, that is, removing duplicate records through a deduplication algorithm; character state change data / state change data, that is, merging continuous repeated states or only recording key state change events. Therefore, different data types to be reported often require different data cleaning technologies. Common data cleaning technologies include: data filtering, data deduplication and data aggregation. In the prior art, data processing equipment cannot distinguish the different attributes contained in the original data and recommend appropriate data cleaning methods respectively. Its data cleaning process needs to include a cleaning process for all abnormal data, which affects the transmission efficiency of IoT device data.

[0052] Specifically, this embodiment first obtains the original data of the device, and then separates the attributes of the original data to obtain a number of target cleaning attributes. Among them, the target cleaning attribute reflects the original data of a single attribute, that is, this step distinguishes the multi-attribute data of the original data according to different attribute types, which facilitates the data processing device to recommend appropriate data cleaning methods for data of different attributes in the subsequent process.

[0053] Furthermore, this embodiment needs to accurately determine the attribute type of each target cleaning attribute. Specifically, the basis for the division of different attribute types is mainly the characteristics and change trends of the data. The following is a brief description of the three different attribute types and the basis for distinguishing them: First, the numerical fluctuation type (Numerical Fluctuation Type), the data values ​​of this type of attribute show the characteristics of ups and downs, without obvious long-term growth or decrease trends, but may have large fluctuations in the short term. The basis for distinguishing them is: the values ​​in the data set change randomly within a certain range, highs and lows appear alternately in the short term, and the overall trend is not obvious. Second, the numerical repetition type (Numerical Repetition Type), a large number of identical or close values ​​appear in its data set, indicating that the value of the attribute does not change much and has high stability. The basis for distinguishing them is: many observations are clustered around the same value or several very close values, with a small variance, which may mean measurement errors, data entry errors, or that the attribute does not change much. Third, the numerical growth type (Numerical Growth Type), its data shows a clear upward trend over time or other variables. The distinction is based on the fact that data values ​​continue to increase over time or with changes in other control variables. Even though there may be fluctuations in the process, the long-term trend is always upward.

[0054] Therefore, in the specific implementation process of this embodiment, each target cleaning attribute can be analyzed by data analysis methods, such as calculating trend lines, viewing data distribution diagrams (histograms, box plots, etc.), and calculating statistical indicators (means, standard deviations, correlation coefficients, etc.), to obtain characteristic information of each target cleaning attribute and classify them accordingly. In addition, this embodiment can also use corresponding statistical analysis tools or visualization methods to distinguish the types of different target cleaning attributes according to the characteristics of specific data sets, which is not specifically limited in this embodiment.

[0055] Step S200: Determine the data cleaning rule corresponding to the attribute type according to the attribute type.

[0056] After the present embodiment has accurately determined the single attribute of the original data and divided the original data according to different attribute types, it will further match the original data of different attributes with their respective suitable cleaning rules.

[0057] In one implementation, this embodiment determines the data cleaning rule corresponding to the attribute type according to the attribute type, specifically including the following steps:

[0058] Step S201: input the attribute type into the data cleaning engine module for data analysis to obtain a data cleaning model corresponding to the attribute type;

[0059] Step S202: Determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.

[0060] During the specific implementation, this embodiment constructs a data cleaning engine module in the data cleaning process. The data cleaning engine module adopts a multi-dimensional data cleaning mechanism, which is mainly reflected in the following three dimensions: First, numerical fluctuation threshold filtering: by dynamically setting the threshold range, abnormal data points that exceed the normal fluctuation range are eliminated; second, numerical repetition deduplication filtering: using efficient recognition algorithms to identify redundant and repeated records; third, continuous growth aggregation filtering: for specific types of data, reasonable time window aggregation is performed to eliminate frequent and low-value data records. It can be seen that the data cleaning engine module is mainly used to receive data of different attribute types and recommend appropriate data cleaning models (filtering cleaning model, deduplication cleaning model and aggregation cleaning model) for these data. Optionally, technicians in this field can also obtain other adapted data cleaning mechanisms based on different data cleaning recommendation logics, such as: threshold setting assisted by time series prediction, similar data merging based on clustering analysis, and data integrity checking combined with business rules, etc., which are not specifically limited in this embodiment.

[0061] Therefore, by inputting the attribute type into the data cleaning engine module for data analysis, the data cleaning model corresponding to the attribute type can be obtained based on the analysis results. Then, after determining the data cleaning model corresponding to the current attribute type, the data cleaning rules corresponding to the data cleaning model can be further retrieved, and then the data cleaning of the current attribute type can be completed according to the retrieved data cleaning rules. Among them, when the data cleaning model is a filtering cleaning model, its corresponding attribute type is a value fluctuation type, and the corresponding data cleaning rule is a filtering cleaning rule. When the data cleaning model is a deduplication cleaning model, its corresponding attribute type is a numerical repetition type, and the corresponding data cleaning rule is a deduplication cleaning rule. When the data cleaning model is an aggregation cleaning model, its corresponding attribute type is a continuous growth type, and the corresponding data cleaning rule is an aggregation cleaning rule.

[0062] Step S300: clean the original data using the data cleaning rule to obtain target reported data.

[0063] Specifically, step S300 of this embodiment includes the following steps:

[0064] Step S301: clean the original data using the data cleaning rule to obtain target cleaned data of the original data;

[0065] Step S302: Obtain the target reporting data according to the original data and the target cleansing data.

[0066] In this process, this embodiment will recommend corresponding data cleaning rules based on the current attribute type of the original data, and then use the corresponding data cleaning rules to clean the original data. The so-called data cleaning process is to find the target cleaning data in the original data of each attribute type for cleaning and deletion, and then report the cleaned and deleted data, and the final reported data is called the target reported data.

[0067] At the same time, the target cleaning data and the target reporting data can also be used to reflect the data cleaning effect during the data cleaning process. For example, the target cleaning data can be obtained by subtracting the target reporting data from the original data, and the cleaning probability can be obtained by dividing the target cleaning data by the original data.

[0068] In one implementation, after the data cleaning rule is used to clean the original data to obtain the target reported data, the present embodiment includes the following steps:

[0069] Step S3001: Acquire second data information of the target reported data, where the second data information includes the data size, reporting frequency and data cleaning rule of the target reported data.

[0070] After the target reported data is obtained in this embodiment, the second data information of the target reported data is identified and stored in advance, so as to facilitate the rapid acquisition of the second data information in the subsequent process, and improve the speed of estimating the data cleaning effect of the data cleaning device on the original data with the help of the second data information in the subsequent process. Optionally, the second data information includes but is not limited to: the data size of the target reported data, the reporting frequency of the target reported data, and the data cleaning rule of the target reported data, wherein the data cleaning rule of the target reported data is used to reflect the attribute type of the target reported data.

[0071] Step S400: estimating a data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data.

[0072] In one implementation, step S400 of this embodiment specifically includes the following steps:

[0073] Step S401: input the first data information and the second data information into a preset mathematical model for data processing to obtain a data cleaning rule;

[0074] Step S402: Estimate the data cleaning effect of the original data according to the data cleaning rule.

[0075] Specifically, this embodiment outputs the first data information before cleaning and the second data information after cleaning to a preset mathematical model for data processing to obtain a data cleaning rule. That is to say, this embodiment compares and analyzes the size and frequency of the data uploaded by the device in history and the data information to be cleaned by the cleaning rule with the data size and reporting frequency of the target reported data after cleaning in the mathematical model to obtain data cleaning rules such as the reduction range (size) of the data volume and the change (number of times) of the reporting frequency in the corresponding cleaning rule, so as to accurately estimate the data reduction effect and enhance the value of subsequent data analysis.

[0076] Furthermore, this embodiment can also compare and analyze the changing trends of different types of attribute data during the cleaning process to gain in-depth insights into the effects of the cleaning strategy and its impact on the overall performance. For example, analyze the quantitative results of data cleaning, such as how many invalid records in the memory have been cleaned and how many errors have been corrected; analyze changes in data distribution to determine changes in statistical characteristics such as trends and skewness in the data set between the first data information before cleaning and the second data information after cleaning; analyze changes in the importance of specific attributes, such as changes in the contribution of certain attributes to model predictions after data cleaning. It can be seen that data cleaning rules can effectively predict the data cleaning effect of the original data.

[0077] In specific implementation, the data cleaning rules can be used to simulate the amount of data expected to be generated after applying the cleaning strategy in the actual operating environment, so as to calculate the number of messages after cleaning and the overall storage requirements. Then, the data cleaning rules will be input into the cleaning estimation model, which adopts an advanced multi-objective optimization algorithm. The multi-objective optimization algorithm can comprehensively consider multiple optimization goals such as storage space saving, network bandwidth occupancy reduction, and response speed improvement under the premise of ensuring data quality, and finally form a global optimal cleaning strategy plan and feedback output. In other words, a real-time feedback mechanism is also established in this embodiment to timely feedback and output the storage space and network bandwidth resources saved due to data cleaning, so as to accurately control resource allocation and cost, thereby saving data storage resources of IoT devices.

[0078] In this embodiment, the mathematical model is an AI model, and the AI ​​model is generated by training using a machine learning algorithm. Machine learning algorithms are a type of mathematical model and method that allows computers to learn from data and make predictions or decisions without explicit programming, mainly including: supervised learning: that is, the model learns the mapping relationship through training data (including input features and corresponding labels). Unsupervised learning: In contrast to supervised learning, the data for unsupervised learning is not labeled. This type of algorithm needs to discover the structure or pattern in the data by itself, such as cluster analysis (K-means), dimensionality reduction (PCA, t-SNE), etc. Semi-supervised learning: This is a method between supervised learning and unsupervised learning, which uses a large amount of unlabeled data and a small amount of labeled data for learning. Reinforcement learning: In this learning method, the algorithm (called an agent) learns how to take actions through interaction with the environment to maximize a certain cumulative reward. AlphaGo is a famous example of reinforcement learning application. Deep learning: As a subfield of machine learning, it uses deep neural networks (containing multiple hidden layers) to learn complex representations of data, especially in image recognition, natural language processing, etc. This embodiment can adopt a suitable machine learning algorithm according to specific needs, the nature of the data and computing resources to improve the accuracy of the generated data cleaning model.

[0079] In one implementation, after estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data, this embodiment further includes the following steps:

[0080] Step S4001: taking the data cleaning effect as a data sample, inputting it into the data cleaning model for optimization training, and updating the data cleaning model in real time.

[0081] When the cleaning prediction model uses an advanced multi-objective optimization algorithm to estimate the data cleaning rules such as the reduction (size) of the amount of data in the cleaning rules and the change (number of times) of the reporting frequency, and obtains the data cleaning effect by comparing the amount of effective data before and after data cleaning, this embodiment will also use this data cleaning effect as a data sample and input it into the data cleaning model for optimization training to update the data cleaning model in real time, including updating model parameters and optimizing cleaning strategies. Therefore, the data cleaning model of this embodiment can be trained and learned continuously, so that the AI ​​model can dynamically adjust its prediction ability and cleaning strategy recommendation ability according to real-time feedback results, forming a closed-loop self-learning and improvement process. Through the feedback loop mechanism, it ensures that the intelligent Internet of Things platform can quickly adapt and provide the most appropriate cleaning strategy when facing the ever-changing business environment and technical challenges, effectively improving the long-term stability and adaptability of the system.

[0082] In summary, this embodiment provides a method for estimating the effect of data cleaning. When it is implemented specifically, this embodiment first obtains the original data of the device and determines the attribute type of the original data. Then, according to the attribute type, the data cleaning rule corresponding to the attribute type is determined. Next, the original data is cleaned using the data cleaning rule to obtain the target reported data. Finally, according to the first data information of the original data and the second data information of the target reported data, the data cleaning effect of the original data is estimated. It can be seen that this embodiment performs data cleaning by accurately determining the attributes, and matches cleaning rules with different attribute characteristics to data with different attributes, thereby improving the efficiency of data cleaning. At the same time, this embodiment can also estimate the effect of data cleaning, achieve the effect of precise control of resource allocation and cost, and save data storage resources of IoT devices.

[0083] Based on the above embodiments, the present invention also provides a device for estimating data cleaning effect, such as Figure 2 As shown in , the data cleaning effect estimation device includes: an attribute type determination module 10, a data cleaning rule determination module 20, a target reported data determination module 30, and a data cleaning effect estimation module 40. Specifically, the attribute type determination module 10 is used to obtain the original data of the device and determine the attribute type of the original data. The data cleaning rule determination module 20 is used to determine the data cleaning rule corresponding to the attribute type according to the attribute type. The target reported data determination module 30 is used to perform data cleaning on the original data using the data cleaning rule to obtain the target reported data. The data cleaning effect estimation module 40 is used to estimate the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data.

[0084] In one implementation, the attribute type determination module 10 includes:

[0085] A target cleaning attribute determination unit is used to obtain raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes;

[0086] A characteristic information determination unit, used for performing characteristic analysis on each of the target cleaning attributes to obtain characteristic information of each of the target cleaning attributes;

[0087] The attribute type determination unit is used to determine the attribute type of each of the target cleaning attributes according to the characteristic information.

[0088] In one implementation, the data cleaning rule determination module 20 includes:

[0089] A data cleaning model determination unit, used for inputting the attribute type into a data cleaning engine module for data analysis to obtain a data cleaning model corresponding to the attribute type;

[0090] The data cleaning rule determining unit is used to determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.

[0091] In one implementation, the target reporting data determination module 30 includes:

[0092] a target cleansing data determining unit, configured to perform data cleaning on the original data using the data cleaning rule to obtain target cleansing data of the original data;

[0093] The target reporting data determining unit is used to obtain the target reporting data according to the original data and the target cleansing data.

[0094] In one implementation, after the target reporting data determination module 30, it also includes:

[0095] The second data information acquisition unit is used to acquire second data information of the target reported data, where the second data information includes the data size, reporting frequency and data cleaning rule of the target reported data.

[0096] In one implementation, the data cleaning effect estimation module 40 includes:

[0097] a data cleaning rule determining unit, configured to input the first data information and the second data information into a preset mathematical model for data processing to obtain a data cleaning rule;

[0098] The data cleaning effect estimation unit is used to estimate the data cleaning effect of the original data according to the data cleaning rule.

[0099] In one implementation, after the data cleaning effect estimation module 40, it also includes:

[0100] The data cleaning model updating unit is used to input the data cleaning effect as a data sample into the data cleaning model for optimization training, and to update the data cleaning model in real time.

[0101] The working principles of each module in the device for reasoning based on digital retina construction of knowledge graph in this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0102] Based on the above embodiments, the present invention further provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 3The terminal device may include one or more processors 100 ( Figure 3 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a data cleaning effect estimation program. When one or more processors 100 execute the computer program 102, the various steps in the data cleaning effect estimation method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the data cleaning effect estimation device embodiment can be implemented, which is not limited here.

[0103] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0104] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0105] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] In summary, the present invention discloses a method, device, terminal device and storage medium for estimating the effect of data cleaning, the method comprising: acquiring the original data of the device, and determining the attribute type of the original data; determining the data cleaning rule corresponding to the attribute type according to the attribute type; performing data cleaning on the original data using the data cleaning rule to obtain target reported data; estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data. The present invention performs data cleaning by being accurate to the attribute, and matches cleaning rules with different attribute characteristics to data with different attributes, thereby improving the efficiency of data cleaning. At the same time, the present invention can also estimate the effect of data cleaning, achieve the effect of precise control of resource allocation and cost, and save data storage resources of IoT devices.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating data cleaning effect, characterized in that: The method comprises: Acquire raw data of a device and determine the attribute type of the raw data; According to the attribute type, determining a data cleaning rule corresponding to the attribute type; The raw data is cleaned using the data cleaning rule to obtain target reported data; estimating a data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data; The acquiring of raw data of the device and determining the attribute type of the raw data includes: Acquire raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes; perform feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes; determine the attribute type of each of the target cleaning attributes according to the feature information; The target reported data refers to the data in the original data corresponding to the data cleaning rule that has been deleted after data cleaning; The estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data includes: Input the first data information and the second data information into a preset mathematical model for data processing to obtain a data cleaning rule; according to the data cleaning rule, estimate the data cleaning effect of the original data; the first data information includes the data size, reporting frequency and data information of the proposed data cleaning rule of the original data; the second data information includes the data size, reporting frequency and data cleaning rule of the target reported data; the data cleaning rule includes the reduction of the data volume and the change of the reporting frequency in the data cleaning rule; The step of determining the data cleaning rule corresponding to the attribute type according to the attribute type includes: The attribute type is input into a data cleaning engine module for data analysis to obtain a data cleaning model corresponding to the attribute type; and according to the data cleaning model, a data cleaning rule corresponding to the attribute type is determined.

2. The method for estimating data cleaning effect according to claim 1, characterized in that: The adopting the data cleaning rule to clean the original data to obtain target reported data includes: Using the data cleaning rule to clean the original data, to obtain target cleaned data of the original data; The target reporting data is obtained according to the original data and the target cleansing data.

3. The method for estimating data cleaning effect according to claim 1, characterized in that: After estimating the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data, the method includes: The data cleaning effect is used as a data sample and input into the data cleaning model for optimization training, and the data cleaning model is updated in real time.

4. A device for estimating data cleaning effect, characterized in that: The method for estimating the data cleaning effect according to any one of claims 1 to 3 is applied to the steps, wherein the device comprises: An attribute type determination module, used to obtain raw data of a device and determine the attribute type of the raw data; A data cleaning rule determination module, used to determine the data cleaning rule corresponding to the attribute type according to the attribute type; A target reporting data determination module, configured to perform data cleaning on the original data using the data cleaning rule to obtain target reporting data; The data cleaning effect estimation module is used to estimate the data cleaning effect of the original data according to the first data information of the original data and the second data information of the target reported data.

5. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a data cleaning effect estimation program stored in the memory and executable on the processor. When the processor executes the data cleaning effect estimation program, the steps of the data cleaning effect estimation method as described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data cleaning effect estimation program, and when the data cleaning effect estimation program is executed by the processor, the steps of the data cleaning effect estimation method according to any one of claims 1 to 3 are implemented.

Images