Data cleaning rule recommendation method and device, terminal equipment and storage medium
By obtaining the original data of IoT devices, identifying their attribute types and recommending appropriate data cleaning rules, the problem of inaccurate recommendation of data cleaning methods in the prior art is solved, and data cleaning efficiency and quality are improved.
Patent Information
- Application Number
- CN202411395710.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art cannot accurately recommend suitable data cleaning methods, resulting in the impact of the data transmission efficiency of IoT devices.
By obtaining the original data of the device, determining its attribute type, and inputting the attribute type into the data cleaning engine module for analysis, we recommend data cleaning models and rules based on the analysis results, and recommending appropriate cleaning methods for different attribute types.
Improve the efficiency and quality of data cleaning, ensure that data of different attribute types matches their characteristics, and improves data transmission efficiency.
Smart Images

Figure CN118897779A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, terminal equipment and storage medium for recommending data cleaning rules. Background Art
[0002] Data cleaning refers to the process of identifying and correcting abnormal data in the data files to be reported according to certain cleaning rules. By cleaning the data to be reported by a large number of IoT devices, the quality of the reported data can be improved.
[0003] At present, IoT devices have the problem of high-frequency and large-volume raw data reporting. Since these raw data contain a series of abnormal data such as outliers, duplicate values, and time series discontinuities, and the data cleaning methods of these abnormal data are often different, the existing data processing equipment cannot accurately recommend appropriate data cleaning methods. The data cleaning process needs to include a cleaning process for all abnormal data, which affects the transmission efficiency of IoT device data.
[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention
[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, apparatus, terminal device and storage medium for recommending data cleaning rules are provided, aiming to solve the problem that the data processing equipment in the prior art cannot accurately recommend a suitable data cleaning method, thereby affecting the transmission efficiency of the data of the Internet of Things devices.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for recommending data cleaning rules, wherein the method comprises: Acquire raw data of a device and determine the attribute type of the raw data; Input the attribute type into the data cleaning engine module for data analysis, and recommend a data cleaning model corresponding to the attribute type based on the analysis result; According to the data cleaning model, a data cleaning rule corresponding to the attribute type is determined.
[0007] In one implementation, the acquiring the original data of the device and determining the attribute type of the original data includes: Acquire raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes; Performing feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes; According to the characteristic information, an attribute type of each of the target cleaning attributes is determined.
[0008] In one implementation, before the inputting the attribute type into the data cleaning engine module for data analysis and recommending the data cleaning model corresponding to the attribute type based on the analysis result, the method includes: Obtaining historical sample data of each of the target cleaning attributes; The historical sample data is input into the data cleaning engine module for training, and the data cleaning model is generated through a machine learning algorithm.
[0009] In one implementation, the inputting the attribute type into a data cleaning engine module for data analysis, and recommending a data cleaning model corresponding to the attribute type based on the analysis result, includes: When the attribute type matches the value fluctuation type in the data cleaning engine module, determining that the data cleaning model is a filtering cleaning model; When the attribute type matches the numerical repetitive type in the data cleaning engine module, determining that the data cleaning model is a deduplication cleaning model; When the attribute type matches the continuous growth type in the data cleaning engine module, it is determined that the data cleaning model is an aggregate cleaning model.
[0010] In one implementation, determining the data cleaning rule corresponding to the attribute type according to the data cleaning model includes: When the data cleaning model is the filtering cleaning model, determining that the data cleaning rule is filtering cleaning; When the data cleaning model is the deduplication cleaning model, determining that the data cleaning rule is deduplication cleaning; When the data cleaning model is the aggregate cleaning model, the data cleaning rule is determined to be aggregate cleaning.
[0011] In one implementation, after determining the data cleaning rule corresponding to the attribute type according to the data cleaning model, the method further includes: According to the data cleaning rules, determine the target cleaning data and the target reporting data; A data cleaning effect is obtained according to the target cleaning data and the target reporting data.
[0012] In one implementation, determining target cleansing data and target reporting data according to the data cleansing rule includes: When the data cleaning rule is filtering cleaning, determining a target threshold value for numerical fluctuation data; Based on the target threshold, determining first target cleaning data of the numerical fluctuation type data; Based on the first target cleaning data, first target reporting data of the numerical fluctuation data is determined.
[0013] In a second aspect, an embodiment of the present invention further provides a device for recommending data cleaning rules, wherein the device includes: An attribute type determination module, used to obtain raw data of a device and determine the attribute type of the raw data; A data cleaning model recommendation module, used to input the attribute type into the data cleaning engine module for data analysis, and recommend a data cleaning model corresponding to the attribute type based on the analysis result; The data cleaning rule determination module is used to determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.
[0014] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and a data cleaning rule recommendation program stored in the memory and executable on the processor, and when the processor executes the data cleaning rule recommendation program, the steps of the data cleaning rule recommendation method of any one of the above-mentioned schemes are implemented.
[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a data cleaning rule recommendation program is stored on the computer-readable storage medium, and when the data cleaning rule recommendation program is executed by a processor, the steps of the data cleaning rule recommendation method described in any one of the above-mentioned schemes are implemented.
[0016] Beneficial effect: Compared with the prior art, the present invention provides a method, device, terminal device and storage medium for recommending data cleaning rules. First, the original data of the device is obtained, and the attribute type of the original data is determined. Then, the attribute type is input into the data cleaning engine module for data analysis, and the data cleaning model corresponding to the attribute type is recommended based on the analysis result. Finally, according to the data cleaning model, the data cleaning rule corresponding to the attribute type is determined. Therefore, the present invention can identify multiple attribute types of the original data, and then recommend appropriate data cleaning rules for data of different attribute types. It can be seen that the present invention performs data cleaning by accurately determining the attributes, and matches cleaning rules with different attribute characteristics to data of different attributes, thereby improving the efficiency of data cleaning and the quality of data after data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flowchart of a specific implementation of a method for recommending data cleaning rules provided in an embodiment of the present invention.
[0018] Figure 2 A functional schematic diagram of a device for recommending data cleaning rules provided in an embodiment of the present invention.
[0019] Figure 3 A functional block diagram of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples.
[0021] Those skilled in the art will appreciate that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms, and the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when an element is "connected" or "coupled" to another element, it may be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0022] The present embodiment provides a method, apparatus, terminal device and storage medium for recommending data cleaning rules. In specific implementation, the present embodiment first obtains the original data of the device and determines the attribute type of the original data. Then, the attribute type is input into the data cleaning engine module for data analysis, and a data cleaning model corresponding to the attribute type is recommended based on the analysis result. Finally, according to the data cleaning model, the data cleaning rule corresponding to the attribute type is determined. Therefore, the present embodiment can identify multiple attribute types of the original data, and then recommend appropriate data cleaning rules for data of different attribute types. At the same time, the present embodiment performs data cleaning by being accurate to the attribute, and matches cleaning rules with different attribute characteristics to data of different attributes, thereby improving the efficiency of data cleaning and the quality of data after data cleaning.
[0023] The data cleaning rule recommendation method of this embodiment can be applied to a terminal device, which can be a preset mobile terminal such as a mobile phone, a tablet, a computer, or an intelligent device such as a smart TV. In this embodiment, the terminal device can be an external device connected to the data cleaning rule recommendation device, or a device built into the data cleaning rule recommendation device. Figure 1As shown in , the method for recommending data cleaning rules in this embodiment includes the following steps: Step S100: Acquire original data of a device and determine the attribute type of the original data.
[0024] Generally speaking, the raw data to be cleaned received by the controller often contains data of multiple attribute types, and the current data cleaning technology cannot accurately identify single attribute data, nor can it recommend the most appropriate data cleaning rules for each attribute data. Therefore, in this embodiment, it is first necessary to separate the attributes of the raw data uploaded by the acquired device to be cleaned, and obtain the attribute type of each data contained in the raw data, and then find and recommend appropriate data cleaning rules for each attribute data. Preferably, the controller that receives the raw data to be cleaned in this embodiment is a Shenzhen Kaihong Super Device Controller based on the KaiHongOS system.
[0025] In one implementation, step S100 of this embodiment specifically includes the following steps: Step S101, obtaining raw data of the device, performing attribute separation on the raw data, and obtaining a number of target cleaning attributes; Step S102, performing feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes; Step S103: Determine the attribute type of each of the target cleaning attributes according to the characteristic information.
[0026] Since the common types of data to be reported include: numerical fluctuation data, that is, filtering invalid or slightly changing data points according to the set threshold; numerical change data / continuous growth data, that is, only retaining the cumulative value of the last state; numerical repetitive data, that is, removing duplicate records through a deduplication algorithm; character state change data / state change data, that is, merging continuous repeated states or only recording key state change events. Therefore, different data types to be reported often require different data cleaning technologies. Common data cleaning technologies include: data filtering, data deduplication and data aggregation. In the prior art, data processing equipment cannot distinguish the different attributes contained in the original data, and thus cannot recommend their own appropriate data cleaning methods for different attributes. As a result, the cleaning process of the original data needs to include a cleaning process for all abnormal data, affecting the transmission efficiency of IoT device data.
[0027] Specifically, this embodiment first obtains the original data of the device, and then separates the attributes of the original data to obtain a number of target cleaning attributes. Among them, the target cleaning attribute reflects the original data of a single attribute, that is, this step distinguishes the multi-attribute data of the original data according to different attribute types, which facilitates the data processing device to recommend appropriate data cleaning methods for data of different attributes in the subsequent process.
[0028] Furthermore, this embodiment needs to accurately determine the attribute type of each attribute. The division of different attribute types is mainly defined based on the characteristics and change trends of the data. The following is a brief description of the three different attribute types and the basis for distinguishing them: First, the numerical fluctuation type (Numerical Fluctuation Type), the data values of this type of attribute show the characteristics of ups and downs, there is no obvious long-term growth or decrease trend, but there may be large fluctuations in the short term. The basis for distinguishing them is: the values in the data set change randomly within a certain range, high points and low points appear alternately in the short term, and the overall trend is not obvious. Second, the numerical repetition type (Numerical Repetition Type), a large number of identical or close values appear in its data set, indicating that the value of the attribute does not change much and has a high stability. The basis for distinguishing them is: many observations are clustered around the same value or several very close values, and the variance is small, which may mean measurement errors, data entry errors, or that the attribute does not change much. Third, the numerical growth type, whose data shows a clear upward trend over time or other variables. The basis for distinction is: the data value continues to increase with the passage of time or changes in other control variables. Even if there may be fluctuations in the process, the long-term trend is always upward growth.
[0029] Therefore, in the specific implementation process of this embodiment, each target cleaning attribute can be analyzed by data analysis methods, such as calculating trend lines, viewing data distribution diagrams (histograms, box plots, etc.), and calculating statistical indicators (means, standard deviations, correlation coefficients, etc.), to obtain characteristic information of each target cleaning attribute and classify them accordingly. In addition, this embodiment can also use corresponding statistical analysis tools or visualization methods to distinguish the types of different target cleaning attributes according to the characteristics of specific data sets, which is not specifically limited in this embodiment.
[0030] Step S200: input the attribute type into a data cleaning engine module for data analysis, and recommend a data cleaning model corresponding to the attribute type based on the analysis result.
[0031] Specifically, the data cleaning process of this embodiment is carried out on a data processing platform, and the data processing platform includes three parts: a data receiving module, a data cleaning engine module, and a data output module. At the same time, the data cleaning engine module of this scheme adopts a multi-dimensional data cleaning mechanism, which is mainly reflected in the following three dimensions: first, numerical fluctuation threshold filtering: by dynamically setting the threshold range, abnormal data points that exceed the normal fluctuation range are eliminated; second, numerical repetition type de-duplication filtering: using an efficient recognition algorithm to identify redundant and repeated records; third, continuous growth type aggregation filtering: for specific types of data, reasonable time window aggregation is performed to eliminate frequent and low-value data records. Therefore, after the attribute type is input into the data cleaning engine module for data analysis in this embodiment, the data cleaning mechanism corresponding to the attribute type can be recommended based on the analysis results. Optionally, those skilled in the art can also obtain other adapted data cleaning mechanisms based on different data cleaning recommendation logics, such as: threshold setting assisted by time series prediction, similar data merging based on cluster analysis, and data integrity check based on business rules, etc., which are not specifically limited in this embodiment.
[0032] It should be noted that the multi-dimensional data cleaning mechanism contained in the data cleaning engine module in this embodiment is not limited to filtering cleaning models, deduplication cleaning models and aggregation cleaning models, but should also include data cleaning models generated by training corresponding to data of various different data types.
[0033] Since this embodiment first needs to identify different types of data, such as numerical data, character data, etc., when inputting the attribute type into the data cleaning engine module for data analysis, and then further determine the cleaning model corresponding to the current type of data. For numerical data (including integers int, long integers long, floating point numbers float and double precision double), this embodiment determines the filtering cleaning model, deduplication cleaning model and aggregation cleaning model according to its internal rules and behavior patterns. For character data, including text text, Boolean value boolean and enumeration enum type, this embodiment also designs corresponding cleaning schemes according to their characteristics, for example, frequent state switching type: for attributes such as switch state and operation mode that frequently switch between multiple states, the system will propose a special cleaning strategy when it recognizes the existence of continuous repeated states, such as merging adjacent repeated states or recording state change events, so that users can more intuitively understand and trace the evolution of the working state of the device. Static data deduplication: For attribute information that always remains consistent, such as device names and aliases, the system will actively prompt to remove the redundant storage of such unchanged data to keep the database simple and efficient. Potential association mining type: The system can also perform association analysis on character data to explore the potential connections hidden between different attributes, such as the relationship between equipment fault codes and other operating parameters, thereby generating more targeted data cleaning and integration rules.
[0034] In one implementation, before the attribute type is input into the data cleaning engine module for data analysis and a data cleaning model corresponding to the attribute type is recommended based on the analysis result, the embodiment includes the following steps: Step S2001, obtaining historical sample data of each target cleaning attribute; Step S2002: input the historical sample data into the data cleaning engine module for training, and generate the data cleaning model through a machine learning algorithm.
[0035] Specifically, the multi-dimensional data cleaning mechanism included in the data cleaning engine module is trained and generated based on historical sample data of several target cleaning attributes. At the same time, in this embodiment, sample data generated during the cleaning process of newly generated data with different attributes is continuously received to train and update the generated data cleaning model in real time to ensure the data cleaning effect of the data cleaning model.
[0036] Preferably, in this embodiment, a data cleaning model with different attributes is trained and generated by a machine learning algorithm. Machine learning algorithms are a type of mathematical model and method that allows computers to learn from data and make predictions or decisions without explicit programming, mainly including: supervised learning: that is, the model learns the mapping relationship through training data (including input features and corresponding labels). Common supervised learning algorithms include linear regression, logistic regression, support vector machine, decision tree, random forest and neural network. Unsupervised learning: In contrast to supervised learning, the data of unsupervised learning has no labels. This type of algorithm needs to discover the structure or pattern in the data by itself, such as cluster analysis (K-means), dimensionality reduction (PCA, t-SNE), etc. Semi-supervised learning: This is a method between supervised learning and unsupervised learning, which uses a large amount of unlabeled data and a small amount of labeled data for learning. Reinforcement learning: In this learning method, the algorithm (called an agent) learns how to take actions through interaction with the environment to maximize a certain cumulative reward. AlphaGo is a famous case of reinforcement learning application. Deep learning: As a subfield of machine learning, it uses deep neural networks (including multiple hidden layers) to learn complex representations of data, especially in image recognition, natural language processing, etc. This embodiment can intelligently adopt appropriate machine learning algorithms according to specific needs, the nature of the data, and computing resources to improve the accuracy of the generated data cleaning model.
[0037] In one implementation, step S200 of this embodiment specifically includes the following steps: Step S201: when the attribute type matches the value fluctuation type in the data cleaning engine module, determining that the data cleaning model is a filtering cleaning model; Step S202: when the attribute type matches the numerical repetitive type in the data cleaning engine module, determining that the data cleaning model is a deduplication cleaning model; Step S203: When the attribute type matches the continuous growth type in the data cleaning engine module, it is determined that the data cleaning model is an aggregate cleaning model.
[0038] Specifically, this embodiment will sequentially input each attribute type into the data cleaning engine module for data analysis, that is, match each attribute type with the model attribute in the data cleaning engine module, and then intelligently recommend the data cleaning model of the corresponding attribute type for each attribute type of data based on the matching results. Therefore, this embodiment can accurately perform data cleaning on the attribute, match the cleaning rules of different attribute characteristics for the data of different attributes, improve the efficiency of data cleaning, and also improve the quality of data after data cleaning. Among them, the data cleaning model corresponding to the attribute type of numerical fluctuation is the filtering cleaning model; the data cleaning model corresponding to the attribute type of numerical repetition is the deduplication cleaning model; the data cleaning model corresponding to the attribute type of continuous growth is the aggregation cleaning model.
[0039] Step S300: Determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.
[0040] In one implementation, step S300 of this embodiment specifically includes the following steps: Step S301: When the data cleaning model is the filtering cleaning model, determining that the data cleaning rule is filtering cleaning; Step S302: When the data cleaning model is the deduplication cleaning model, determining that the data cleaning rule is deduplication cleaning; Step S303: When the data cleaning model is the aggregate cleaning model, determine that the data cleaning rule is aggregate cleaning.
[0041] Specifically, the data cleaning rules in each data cleaning model are set specifically for the data attribute type. Since the filtering cleaning model is the best way to clean numerical fluctuation data, and the best cleaning rule for numerical fluctuation data is filtering cleaning, when the data cleaning model is the filtering cleaning model, the data cleaning rule is filtering cleaning. Similarly, the deduplication cleaning model, numerical repetitive data, and deduplication cleaning correspond to each other; the aggregation cleaning model, continuous growth data, and aggregation cleaning correspond to each other.
[0042] In one implementation, after determining the data cleaning rule corresponding to the attribute type according to the data cleaning model, this embodiment specifically includes the following steps: Step S400: Determine target cleansing data and target reporting data according to the data cleaning rule; Step S500: obtaining a data cleaning effect according to the target cleaning data and the target reporting data.
[0043] In this embodiment, data cleaning is performed accurately according to the attributes. After recommending cleaning rules with different attribute characteristics for data with different attributes, data cleaning is completed according to the recommended data cleaning rules corresponding to each attribute. The so-called data cleaning process is to determine the target cleaning data in the data of each attribute type for cleaning and deletion, and then report the data after cleaning and deletion. The final reported data is called target reported data. At the same time, the target cleaning data and target reported data can also reflect the data cleaning effect in the data cleaning process. For example, the target cleaning data can be obtained by subtracting the target reported data from the original data, and the cleaning probability can be obtained by dividing the target cleaning data by the original data.
[0044] In one implementation, step S400 of this embodiment specifically includes the following steps: Step S401: when the data cleaning rule is filtering cleaning, determining a target threshold value of numerical fluctuation data; Step S402: determining first target cleansing data of the numerical fluctuation data based on the target threshold; Step S403: Determine first target reporting data of the numerical fluctuation data based on the first target cleaning data.
[0045] Specifically, data cleaning is completed according to the recommended data cleaning rules corresponding to each attribute. When the data cleaning rule is filtering cleaning, the filtering cleaning implementation method is used to determine the target cleaning data and target reporting data, that is, first determine the target threshold of the numerical fluctuation data, and then divide the cleaning data according to the target threshold. For example, the data in the original data that exceeds the target threshold is determined as the first target cleaning data for cleaning and deletion, and the data that does not exceed the target threshold (the original data except the first target cleaning data) is determined as the first target reporting data for reporting.
[0046] In addition, when the data cleaning rule is deduplication cleaning, the implementation method of deduplication cleaning is used to determine the second target cleaning data and the second target reported data, that is, first determine the deduplication attribute of the numerical repeated data, and then divide the cleaning data according to the deduplication attribute, for example, the data reported repeatedly in the same time period in the original data is cleaned and deleted. When the data cleaning rule is aggregation cleaning, the implementation method of aggregation cleaning is used to determine the third target cleaning data and the third target reported data, for example, first determine the aggregation attribute and aggregation time of the continuously growing data.
[0047] That is to say, if this embodiment determines that the attribute types contained in the original data include two types: numerical fluctuation data and numerical repetitive data, then the target cleaning data of the original data is "first target cleaning data + second target cleaning data", and the target reporting data of the original data is "first target reporting data + second target reporting data".
[0048] In summary, this embodiment provides a method for recommending data cleaning rules. When it is implemented specifically, this embodiment first obtains the original data of the device and determines the attribute type of the original data. Then, the attribute type is input into the data cleaning engine module for data analysis, and the data cleaning model corresponding to the attribute type is recommended based on the analysis result. Finally, according to the data cleaning model, the data cleaning rule corresponding to the attribute type is determined. Therefore, this embodiment can identify multiple attribute types of the original data, and then recommend appropriate data cleaning rules for data of different attribute types. At the same time, this embodiment performs data cleaning by being accurate to the attribute, and matches cleaning rules with different attribute characteristics for data of different attributes, thereby improving the efficiency of data cleaning and the quality of data after data cleaning.
[0049] Based on the above embodiments, the present invention also provides a device for recommending data cleaning rules, such as Figure 2 As shown in , the data cleaning rule recommendation device includes: an attribute type determination module 10, a data cleaning model recommendation module 20, and a data cleaning rule determination module 30. Specifically, the attribute type determination module 10 is used to obtain the original data of the device and determine the attribute type of the original data. The data cleaning model recommendation module 20 is used to input the attribute type into the data cleaning engine module for data analysis, and recommend the data cleaning model corresponding to the attribute type based on the analysis result. The data cleaning rule determination module 30 is used to determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.
[0050] In one implementation, the attribute type determination module 10 includes: A target cleaning attribute determination unit is used to obtain raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes; A characteristic information determination unit, used for performing characteristic analysis on each of the target cleaning attributes to obtain characteristic information of each of the target cleaning attributes; The attribute type determination unit is used to determine the attribute type of each of the target cleaning attributes according to the characteristic information.
[0051] In one implementation, before inputting the attribute type into the data cleaning engine module for data analysis and recommending a data cleaning model corresponding to the attribute type based on the analysis result, the method includes: A historical sample data acquisition unit, used to acquire historical sample data of each target cleaning attribute; The data cleaning model generating unit is used to input the historical sample data into the data cleaning engine module for training, and generate the data cleaning model through a machine learning algorithm.
[0052] In one implementation, the data cleaning model recommendation module 20 includes: A filtering cleaning model determining unit, configured to determine that the data cleaning model is a filtering cleaning model when the attribute type matches the value fluctuation type in the data cleaning engine module; a deduplication cleaning model determining unit, configured to determine that the data cleaning model is a deduplication cleaning model when the attribute type matches a numerical repetitive type in a data cleaning engine module; The aggregate cleaning model determining unit is used to determine that the data cleaning model is an aggregate cleaning model when the attribute type matches the continuous growth type in the data cleaning engine module.
[0053] In one implementation, the data cleaning rule determination module 30 includes: a filtering and cleaning rule determining unit, configured to determine that the data cleaning rule is filtering and cleaning when the data cleaning model is the filtering and cleaning model; a deduplication cleaning rule determining unit, configured to determine that the data cleaning rule is deduplication cleaning when the data cleaning model is the deduplication cleaning model; The aggregation cleaning rule determining unit is used to determine that the data cleaning rule is aggregation cleaning when the data cleaning model is the aggregation cleaning model.
[0054] In one implementation, after determining the data cleaning rule corresponding to the attribute type according to the data cleaning model, the method further includes: A target cleaning data and target reporting data determining unit, configured to determine the target cleaning data and the target reporting data according to the data cleaning rule; The data cleaning effect determining unit is used to obtain the data cleaning effect according to the target cleaning data and the target reporting data.
[0055] In one implementation, the target cleaning data and target reporting data determination unit includes: A target threshold determination subunit, used for determining a target threshold of numerical fluctuation data when the data cleaning rule is filtering cleaning; A first target cleaning data determining subunit, configured to determine first target cleaning data of the numerical fluctuation type data based on the target threshold; The first target reporting data determining subunit is used to determine the first target reporting data of the numerical fluctuation data based on the first target cleaning data.
[0056] The working principles of each module in the data cleaning rule recommendation device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.
[0057] Based on the above embodiments, the present invention further provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 3 The terminal device may include one or more processors 100 ( Figure 3 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a data cleaning rule recommendation program. When one or more processors 100 execute the computer program 102, the various steps in the data cleaning rule recommendation method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the data cleaning rule recommendation device embodiment can be implemented, which is not limited here.
[0058] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0059] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.
[0060] Those skilled in the art will understand that Figure 3The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0061] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0062] In summary, the present invention discloses a method, device, terminal device and storage medium for recommending data cleaning rules, the method comprising: obtaining the original data of the device, and determining the attribute type of the original data; inputting the attribute type into the data cleaning engine module for data analysis, and recommending the data cleaning model corresponding to the attribute type based on the analysis result; and determining the data cleaning rule corresponding to the attribute type according to the data cleaning model. The present invention can identify multiple attribute types of the original data, and then recommend appropriate data cleaning rules for data of different attribute types. It can be seen that the present invention performs data cleaning by being accurate to the attributes, and matches cleaning rules with different attribute characteristics to data of different attributes, thereby improving the efficiency of data cleaning and the quality of data after data cleaning.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for recommending data cleaning rules. It is characterized in that The method comprises: Acquire raw data of a device and determine the attribute type of the raw data; Input the attribute type into the data cleaning engine module for data analysis, and recommend a data cleaning model corresponding to the attribute type based on the analysis result; According to the data cleaning model, a data cleaning rule corresponding to the attribute type is determined.
2. The method for recommending data cleaning rules according to claim 1, It is characterized in that The acquiring of raw data of the device and determining the attribute type of the raw data includes: Acquire raw data of the device, perform attribute separation on the raw data, and obtain a number of target cleaning attributes; Performing feature analysis on each of the target cleaning attributes to obtain feature information of each of the target cleaning attributes; According to the characteristic information, an attribute type of each of the target cleaning attributes is determined.
3. The method for recommending data cleaning rules according to claim 2, It is characterized in that Before inputting the attribute type into the data cleaning engine module for data analysis and recommending the data cleaning model corresponding to the attribute type based on the analysis result, the method includes: Obtaining historical sample data of each of the target cleaning attributes; The historical sample data is input into the data cleaning engine module for training, and the data cleaning model is generated through a machine learning algorithm.
4. The method for recommending data cleaning rules according to claim 3, It is characterized in that The inputting the attribute type into the data cleaning engine module for data analysis, and recommending a data cleaning model corresponding to the attribute type based on the analysis result, includes: When the attribute type matches the value fluctuation type in the data cleaning engine module, determining that the data cleaning model is a filtering cleaning model; When the attribute type matches the numerical repetitive type in the data cleaning engine module, determining that the data cleaning model is a deduplication cleaning model; When the attribute type matches the continuous growth type in the data cleaning engine module, it is determined that the data cleaning model is an aggregate cleaning model.
5. The method for recommending data cleaning rules according to claim 4, It is characterized in that Determining the data cleaning rule corresponding to the attribute type according to the data cleaning model includes: When the data cleaning model is the filtering cleaning model, determining that the data cleaning rule is filtering cleaning; When the data cleaning model is the deduplication cleaning model, determining that the data cleaning rule is deduplication cleaning; When the data cleaning model is the aggregate cleaning model, the data cleaning rule is determined to be aggregate cleaning.
6. The method for recommending data cleaning rules according to claim 5, It is characterized in that After determining the data cleaning rule corresponding to the attribute type according to the data cleaning model, the method further comprises: According to the data cleaning rules, determine the target cleaning data and the target reporting data; A data cleaning effect is obtained according to the target cleaning data and the target reporting data.
7. The method for recommending data cleaning rules according to claim 6, It is characterized in that Determining target cleaning data and target reporting data according to the data cleaning rule includes: When the data cleaning rule is filtering cleaning, determining a target threshold value for numerical fluctuation data; Based on the target threshold, determining first target cleaning data of the numerical fluctuation type data; Based on the first target cleaning data, first target reporting data of the numerical fluctuation data is determined.
8. A device for recommending data cleaning rules, It is characterized in that The device comprises: An attribute type determination module, used to obtain raw data of a device and determine the attribute type of the raw data; A data cleaning model recommendation module, used to input the attribute type into the data cleaning engine module for data analysis, and recommend a data cleaning model corresponding to the attribute type based on the analysis result; The data cleaning rule determination module is used to determine the data cleaning rule corresponding to the attribute type according to the data cleaning model.
9. A terminal device, It is characterized in that The terminal device includes a memory, a processor, and a data cleaning rule recommendation program stored in the memory and executable on the processor. When the processor executes the data cleaning rule recommendation program, the steps of the data cleaning rule recommendation method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a data cleaning rule recommendation program, and when the data cleaning rule recommendation program is executed by a processor, the steps of the data cleaning rule recommendation method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Data cleaning system and data cleaning method
CN118363950A
Base data cleaning method and device based on rule engine, equipment and medium
CN118467522A
Automatic data matching method, electronic device and computer-readable storage medium
WO2019024231A1
Cited By
Data quality improvement method for data processing
CN119336745A
