Data cleaning method, device and equipment based on data attributes and storage medium

Abstract

The invention relates to the technical field of data cleaning, in particular to a data cleaning method and device based on data attributes, equipment and a storage medium. According to the method, real-time data, collected each time, of all attributes serves as a real-time data set, then whether the real-time data of the attributes meet the attribute cleaning conditions or not is judged based on the attribute cleaning conditions, if the real-time data of the attributes meet the attribute cleaning conditions, the real-time data of the attributes are removed, however, only the real-time data of the attributes are removed, and the real-time data of the attributes are removed. And the whole real-time data group is not removed, so that the real-time data of other attributes in the real-time data group can be stored. According to the analysis, the attribute data needing to be cleaned can be removed, other attribute data not needing to be cleaned can be reserved, and therefore the data cleaning quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and in particular to a data cleaning method, device, equipment and storage medium based on data attributes. Background Art

[0002] Data cleaning is to remove redundant data to improve data quality. Collect the attribute data of each individual (the individual can be electromechanical equipment), and collect all the attribute data of the individual each time to form the data group collected this time (that is, the whole data composed of all attribute data). Once one or several attribute data in the data group are determined to be data that needs to be cleaned, even if other attribute data in the data group are not determined to be data that needs to be cleaned, the data group will be directly cleaned, resulting in the cleaning of important attribute data, which in turn leads to the loss of valuable attribute data.

[0003] In summary, existing data cleaning methods are prone to mistakenly cleaning important attribute data.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a data cleaning method, device, equipment and storage medium based on data attributes, which solves the problem that important attribute data is easily mistakenly cleaned by existing data cleaning methods.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a data cleaning method based on data attributes, which includes:

[0008] Acquire a real-time data group, the real-time data group including real-time data corresponding to each attribute of the target individual;

[0009] Filtering out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group;

[0010] The real-time data group is saved after the data to be cleaned is removed.

[0011] In one implementation, obtaining the real-time data group includes:

[0012] Filter out the attribute with the worst stability from each of the attributes;

[0013] Get the shortest stable duration corresponding to the attribute with the worst stability;

[0014] Determining a collection frequency of a collection device according to the shortest stable duration, wherein the collection device is used to collect attributes of the target individual;

[0015] The acquisition device is controlled to acquire the real-time data group at the acquisition frequency, and the real-time data group is acquired through the acquisition device.

[0016] In one implementation, the step of selecting the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group, includes:

[0017] Determine the attribute filtering condition and the attribute deduplication condition in the attribute cleaning condition;

[0018] The data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition are screened out from the real-time data corresponding to each of the attributes, and the data to be filtered and the data to be deduplicated are used as the data to be cleaned, and the data to be filtered and the data to be deduplicated are removed from the real-time data group.

[0019] In one implementation, the filtering out the data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, and using the data to be filtered and the data to be deduplicated as the data to be cleaned, and removing the data to be filtered and the data to be deduplicated from the real-time data group, includes:

[0020] Determining a first total number of attributes corresponding to the attribute filtering condition;

[0021] Determining a second total number of attributes corresponding to the attribute deduplication condition;

[0022] When the first total number is greater than or equal to the second total number, filtering out the data to be filtered that meets the attribute filtering condition from the real-time data corresponding to each of the attributes, and removing the data to be filtered from the real-time data in the real-time data group to obtain a filtered data group;

[0023] The data to be deduplicated that meets the attribute deduplication condition is screened out from the real-time data in the filtered data group, and the data to be deduplicated is removed from the real-time data in the filtered data group.

[0024] In one implementation, the filtering out the data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, and using the data to be filtered and the data to be deduplicated as the data to be cleaned, and removing the data to be filtered and the data to be deduplicated from the real-time data group, further includes:

[0025] When the first total number is less than the second total number, filtering out the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, and removing the data to be deduplicated from the real-time data in the real-time data group to obtain a deduplicated data group;

[0026] The data to be filtered that meets the attribute filtering condition is screened out from the real-time data in the deduplicated data group, and the data to be filtered is removed from the real-time data in the deduplicated data group.

[0027] In one implementation, the attribute deduplication condition is that the real-time data of the attribute is the same as the historical data of the attribute, or the attribute deduplication condition is that the difference between the real-time data of the attribute and the historical data of the attribute is less than a threshold.

[0028] In one implementation, the step of saving the real-time data group after removing the data to be cleaned includes:

[0029] The real-time data group after removing the data to be cleaned is saved in a stack of the cloud platform.

[0030] In a second aspect, an embodiment of the present invention further provides a data cleaning device based on data attributes, wherein the device includes the following components:

[0031] A data reading module is used to obtain a real-time data group, which includes real-time data corresponding to various attributes of the target individual;

[0032] A data cleaning module, used to filter out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and remove the data to be cleaned in the real-time data group;

[0033] The data storage module is used to store the real-time data group after removing the data to be cleaned.

[0034] In one implementation, the data reading module includes:

[0035] An attribute screening unit, used for screening out an attribute with the worst stability from each of the attributes;

[0036] The duration reading unit is used to obtain the shortest stable duration corresponding to the attribute with the worst stability;

[0037] A frequency calculation unit, used to determine the collection frequency of a collection device according to the shortest stable duration, wherein the collection device is used to collect the attributes of the target individual;

[0038] The control unit is used to control the acquisition device to acquire the real-time data group at the acquisition frequency, and obtain the real-time data group through the acquisition device.

[0039] In one implementation, the data cleaning module includes:

[0040] A condition setting unit, used to determine an attribute filtering condition and an attribute deduplication condition in the attribute cleaning condition;

[0041] A cleaning unit is used to filter out the data to be filtered that meets the attribute filtering conditions and the data to be deduplicated that meets the attribute deduplication conditions from the real-time data corresponding to each of the attributes, and use the data to be filtered and the data to be deduplicated as the data to be cleaned, and remove the data to be filtered and the data to be deduplicated from the real-time data group.

[0042] In one implementation, the cleaning unit includes:

[0043] A first quantity calculation component, used to determine a first total quantity of attributes corresponding to the attribute filtering condition;

[0044] A second quantity calculation component is used to determine a second total quantity of attributes corresponding to the attribute deduplication condition, where the attribute corresponding to the attribute deduplication condition is different from the attribute corresponding to the attribute filtering condition;

[0045] A first filtering component is used for filtering the data to be filtered that meets the attribute filtering condition from the real-time data corresponding to each of the attributes when the first total number is greater than or equal to the second total number, and removing the data to be filtered from the real-time data in the real-time data group to obtain a filtered data group;

[0046] The first deduplication component is used to filter out the data to be deduplicated that meets the attribute deduplication condition from the real-time data in the filtered data group, and remove the data to be deduplicated from the real-time data in the filtered data group.

[0047] In one implementation, the cleaning unit further includes:

[0048] A second deduplication component is used for filtering out the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes when the first total number is less than the second total number, and removing the data to be deduplicated from the real-time data in the real-time data group to obtain a deduplicated data group;

[0049] The second filtering component is used to filter out the data to be filtered that meets the attribute filtering condition from the real-time data in the deduplicated data group, and remove the data to be filtered from the real-time data in the deduplicated data group.

[0050] In one implementation, the data storage module includes:

[0051] The data storage unit is used to store the real-time data group after removing the data to be cleaned in a stack of the cloud platform.

[0052] In a third aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and a data cleaning program based on data attributes stored in the memory and executable on the processor, and when the processor executes the data cleaning program based on data attributes, the steps of the data cleaning method based on data attributes described above are implemented.

[0053] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a data cleaning program based on data attributes is stored. When the data cleaning program based on data attributes is executed by a processor, the steps of the data cleaning method based on data attributes described above are implemented.

[0054] Beneficial effect: The present invention regards the real-time data of each attribute collected each time as a real-time data group, and then judges whether the real-time data of the attribute meets the attribute cleaning condition based on the attribute cleaning condition. If the attribute cleaning condition is met, the real-time data of the attribute is removed. However, the present invention only removes the real-time data of the attribute, and does not remove the entire real-time data group, so that the real-time data of other attributes in the real-time data group can be preserved. From the above analysis, it can be seen that the present invention can not only remove the attribute data that needs to be cleaned, but also retain other attribute data that does not need to be cleaned, thereby improving the data cleaning quality of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the overall flow chart of the present invention;

[0056] Figure 2 A structural diagram of a data cleaning device based on data attributes provided by the present invention;

[0057] Figure 3 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] According to research, data cleaning is to remove redundant data to improve data quality. Collect the attribute data of each individual (the individual can be electromechanical equipment), and collect all the attribute data of the individual each time to form the data group collected this time (that is, the whole data composed of all attribute data). Once one or several attribute data in the data group are determined to be data that needs to be cleaned, even if other attribute data in the data group are not determined to be data that needs to be cleaned, the data group will be directly cleaned, resulting in the cleaning of important attribute data, which in turn leads to the loss of valuable attribute data.

[0060] In order to solve the above technical problems, the present invention provides a data cleaning method, device, equipment and storage medium based on data attributes, which solves the problem that the existing data cleaning methods are prone to mistakenly clean important attribute data. In specific implementation, first obtain a real-time data group, which includes real-time data corresponding to each attribute of the target individual; then filter out the data to be cleaned that meets the attribute cleaning conditions from the real-time data corresponding to each of the attributes, and remove the data to be cleaned in the real-time data group; and save the real-time data group after removing the data to be cleaned.

[0061] For example, if the target individual is a human body, and the attribute data is the temperature, heart rate, blood pressure, and blood sugar of the human body, the real-time data of the above attributes of the human body is collected to obtain the real-time data set {T t ,x t ,y t ,z t}, T t is the temperature value at the current time t, x t is the heartbeat value at the current time t, y t is the blood pressure value at the current time t, z t is the blood sugar value at the current time t, where the temperature, heart rate, blood pressure and blood sugar values ​​are all real-time data. At the time after the current time t, several data sets of human attributes have been collected, namely {T t-1 ,x t-1 ,y t-1 ,z t-1}、{T t-2 ,x t-2 ,y t-2 ,z t-2}、{T t-3 ,x t-3 ,y t-3 ,z t-3}. t Greater than 42 degrees (exceeding the human body temperature range), so it can be determined as T t The temperature collector is abnormal and collects the wrong temperature, so T is directly filtered out t, and x t and x t-1 Repeat, so x t It also needs to be removed. t and x t After that, the real-time data set becomes {y t ,z t}. From the above analysis, we can see that only T t and x t It satisfies the cleaning condition, and y t and z t The cleaning conditions are not met, so this embodiment only removes T t and x t , and y t and z t Be preserved.

[0062] The data cleaning method based on data attributes of this embodiment can be applied to a terminal device, which can be a terminal product with data processing function, such as a computer. Figure 1 As shown in , the data cleaning method based on data attributes specifically includes the following steps:

[0063] S100, obtaining a real-time data group, where the real-time data group includes real-time data corresponding to various attributes of the target individual.

[0064] S200, filtering out data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group.

[0065] S300, saving the real-time data group after removing the data to be cleaned.

[0066] The data cleaning method based on steps S100 to S300 can be applied to cleaning device attribute data. The specific application scenarios are as follows:

[0067] Get the real-time data of the device's operating power value, vibration frequency value, voltage value, and thermal efficiency value. If the operating power value is greater than the threshold, for example, the device's maximum possible operating power is p, and the device's real-time operating power value is greater than p, it is possible that the abnormality of the power collector has caused the wrong operating power value to be collected, so the operating power value is directly filtered out. The voltage value is repeated with the historical voltage value obtained before, so this voltage value is also removed. That is, save the two real-time data of vibration frequency value and thermal efficiency value, together with the historical attribute real-time data of the device, and analyze the stability of the device operation. Since the real-time data collected each time is filtered and deduplicated, useless data is reduced, that is, unnecessary data is avoided from occupying memory.

[0068] Another application scenario is as follows:

[0069] The smoke sensor reports "smoke concentration" and "alarm status" every second. When the concentration is greater than 500mg / m 3 When the "alarm status" is true, otherwise it is false. In this scenario, the "smoke density" will fluctuate with environmental changes, but the "alarm status" is false for a long time under normal circumstances. In this scenario, you can specify to perform single attribute deduplication cleaning on the "alarm status". Data after data:

[0070] The first five groups of data after deduplication are: {concentration: 1, status: false}, {concentration: 0, status: false}, {concentration: 1, status: false}, {concentration: 1, status: false}, {concentration: 1, status: false}.

[0071] Five groups of data after deduplication: {concentration: 1, status: false}, {concentration: 0}, {concentration: 1}, {concentration: 1}, {concentration: 1}.

[0072] Embodiment 1: In this embodiment, step S100 includes the following specific steps S101, S102, S103, and S104:

[0073] S101, selecting the attribute with the worst stability from the attributes.

[0074] S102, obtaining the shortest stable duration corresponding to the attribute with the worst stability.

[0075] S103, according to the shortest stable time t min , determine a collection frequency f of a collection device, wherein the collection device is used to collect attributes of the target individual.

[0076] For example, if there are ten attributes, and one of them has the worst stability, that is, the data corresponding to this attribute changes frequently and is unstable. Then the collection frequency of the collection device should be determined based on the shortest stable duration that this attribute can maintain.

[0077] The acquisition frequency f is the number of times the target individual attribute data is obtained per unit time. The larger the acquisition frequency f, the more times the data is collected per unit time. The smaller the acquisition frequency f, the fewer times the data is collected per unit time. min And the acquisition frequency f satisfies the following relationship:

[0078]

[0079] Where t' is the duration threshold, a and b are both constants, and a is an integer. represents rounding up. From the above formula, we can see that when tmin When it is too large, the acquisition frequency no longer follows t min The purpose of this is to ensure that the amount of all attribute data collected is not too small, so that there is enough data for analyzing the target individual.

[0080] S104, controlling the acquisition device to acquire the real-time data group at the acquisition frequency, and obtaining the real-time data group through the acquisition device.

[0081] The collection device first collects data of various attributes of the target individual at the collection frequency calculated above to form a real-time data group. After filtering and deduplication, the real-time data group can be uploaded to the cloud platform to save the real-time data group.

[0082] Embodiment 2: In this embodiment, step S200 includes the following specific steps S201, S202, S203, S204, and S205:

[0083] S201, determining an attribute filtering condition and an attribute deduplication condition in the attribute cleaning condition.

[0084] S202: Determine a first total number of attributes corresponding to the attribute filtering condition.

[0085] S203: Determine a second total number of attributes corresponding to the attribute deduplication condition.

[0086] S204, when the first total number is greater than or equal to the second total number, filtering out the data to be filtered that meets the attribute filtering condition from the real-time data corresponding to each of the attributes, and removing the data to be filtered from the real-time data in the real-time data group to obtain a filtered data group;

[0087] S205: Filter out the data to be deduplicated that meets the attribute deduplication condition from the real-time data in the filtered data group, and remove the data to be deduplicated from the real-time data in the filtered data group.

[0088] That is, when the number of attributes to be filtered is greater than the number of attributes to be deduplicated, the filtering operation is performed first, and the deduplication operation is performed later. This is because when there are many attributes to be filtered, filtering these attributes first can reduce the amount of data, so that the subsequent deduplication can reduce the number of duplicate determinations, thereby saving deduplication time.

[0089] For example, there are ten attributes s1, s2, s3, s4, s5, s6, s7, s8, s9, and s10. Among them, the data of s1, s2, and s3 are often collected as incorrect data, so the first total number is 3, and the data of s7 and s8 are often repeated, so the second total number is 2.

[0090] In this embodiment, it may also be limited that the attribute corresponding to the attribute deduplication condition and the attribute corresponding to the attribute filtering condition are different, that is, there are no repeated attributes in the five attributes s1, s2, s3, s7 and s8.

[0091] For example, the attribute filtering condition is that s1 is greater than 6, s2 is less than 1, and the attribute deduplication condition is that the same data appears.

[0092] The first historical data set 1 collected: {s1=5, s2=3, s3=5, s4=1, s5=8, s6=7, s7=15, s8=20, s9=0.5, s10=1.1}; the second historical data set 2 collected: {s1=8, s2=3, s3=5, s4=1, s5=8, s6=8, s7=12, s8=22, s9=0.6, s10=1.2}. Since s1=8, which is greater than 6, the attribute data of s1 needs to be filtered out. At the same time, s2, s3, s4, and s5 are repeated in historical data set 1 and also need to be removed. Then, after cleaning, Historical data group 2 becomes {s6=8, s7=12, s8=22, s9=0.6, s10=1.2}; the currently collected real-time data group: {s1=5, s2=0, s3=5.5, s4=3, s5=8, s6=7, s7=15, s8=22, s9=0.5, s10=1.1}, the real-time data group is filtered to obtain the filtered data group: {s1=5, s3=5.5, s4=3, s5=8, s6=7, s7=15, s8=22, s9=0.5, s10=1.1}, and deduplication of the filtered data group gives the deduplication data group: {s3=5.5, s4=3}.

[0093] In this embodiment, when the first total number is less than the second total number, the data to be deduplicated that meets the attribute deduplication condition is screened out from the real-time data corresponding to each of the attributes, and the data to be deduplicated is removed from the real-time data in the real-time data group to obtain a deduplicated data group. The data to be filtered that meets the attribute filtering condition is screened out from the real-time data in the deduplicated data group, and the data to be filtered is removed from the real-time data in the deduplicated data group.

[0094] That is, when the number of attributes that need to be deduplicated is greater than the number of attributes that need to be filtered, the deduplication operation is performed first, and then the filtering operation is performed. This is because when there are many attributes that may have duplicate data, removing this part of the duplicate data first, then when filtering is performed, the number of operations to determine whether it is the attribute data that needs to be filtered can be reduced, that is, the efficiency of attribute filtering can be improved.

[0095] In this embodiment, the attribute deduplication condition can also be that the difference between the real-time data of the attribute and the historical data of the attribute is less than a threshold value. When the difference between the two is less than the threshold value, it means that the difference between the newly added real-time data and the historical data is not large. Therefore, even if the newly added real-time data is not saved, it will not affect the overall data analysis. In addition, not saving the newly added real-time data can also reduce memory usage.

[0096] Embodiment 3, the specific process of step 200 in this embodiment is as follows: directly filter out the data to be filtered that meet the attribute filtering conditions from the real-time data corresponding to each attribute in the real-time data group, and after filtering, filter out the data to be deduplicated that meet the attribute deduplication conditions from the remaining real-time data. When the data to be filtered and the data to be deduplicated are removed from the real-time data group, that is, the data to be cleaned is removed from the real-time data group, thereby completing the cleaning operation of the real-time data group.

[0097] For example, there are three attributes a, b, and c. Three data groups {a=20, b=10}, {a=15, b=10, c=5}, and {a=10, b=10, c=5} are collected in chronological order, where the attribute filtering condition is that a is greater than 10. After filtering, the first data group becomes: {b=10}; after filtering, the second data group becomes: {b=10, c=5}, and after deduplication, {b=10, c=5} becomes: {c=5}; a in the third data group is not greater than 10, so the third data group does not need to be filtered, and after deduplication, the third data group becomes: {a=10}.

[0098] Embodiment 4, based on embodiment 1 or embodiment 2 or embodiment 3, in this embodiment, the real-time data group after cleaning is saved in the stack of the cloud platform. As a data structure, the stack is a special linear table that can only perform insertion and deletion operations at one end. It stores data according to the principle of last in, first out. The data that enters first is pushed into the bottom of the stack, and the last data is at the top of the stack. When data needs to be read, data is popped out from the top of the stack (the last data is read out first). Therefore, using a stack to store real-time data groups of various attributes, the last collected data group can be placed on the top of the stack. The last collected data group is the latest data of the target individual attribute. Therefore, the latest data is placed on the top of the stack. When it is necessary to analyze whether the target individual is abnormal, the latest data is read first, and the target individual is judged to be abnormal based on the latest data to reflect the current status of the target individual.

[0099] Embodiment 5: Analyze the correlation between various attributes and select attribute association sets with correlation, that is, various attributes in the attribute association set are correlated with each other, and two attributes are correlated, that is, when the data of one attribute is sent and changed, the data of the other attribute will also be sent and changed, and the two will change synchronously. First, the data of any attribute in the attribute association set is determined to be repeatable. If the data is repeated, then continue to determine whether the data of other attributes in the attribute association set is repeated.

[0100] For example, the real-time data set S t ={a t ,b t ,c t}, where a, b, and c are the data of the three attributes at the current time t, and the historical data set S k ={a k ,b k ,c k}, k∈[1,t-1], if attribute a and attribute b are associated attributes, it has been determined that the historical data group has appeared and a t Duplicate data, then you need to focus on judging c t Is it duplicate data?

[0101] By establishing associated attributes, for real-time data groups containing a large number of attributes, duplicate data can be removed through associated attributes, and it is unnecessary to make duplicate judgments on each attribute one by one, thereby improving the efficiency of duplicate judgments.

[0102] In this embodiment, the following method is used to determine whether two attributes are related:

[0103] Calculate the mean and variance of the historical data of the two attributes of the historical data group respectively, then compare the means of the two attributes, and then compare the variances of the two attributes. If the means and variances of the two attributes are the same, then the two attributes are associated attributes, otherwise the two attributes are unassociated attributes.

[0104] The history of the two attributes can also be plotted into two groups of scatter plots, and linear fitting can be performed on the two groups of scatter plots to obtain two fitting curves. Then, the two fitting curves can be compared and analyzed. If the result of the comparative analysis is the same curve, it means that the two attributes are associated attributes, otherwise they are unassociated attributes.

[0105] In summary, the present invention regards the real-time data of each attribute collected each time as a real-time data group, and then judges whether the real-time data of the attribute meets the attribute cleaning condition based on the attribute cleaning condition. If the attribute cleaning condition is met, the real-time data of the attribute is removed. However, the present invention only removes the real-time data of the attribute, and does not remove the entire real-time data group, so that the real-time data of other attributes in the real-time data group can be preserved. From the above analysis, it can be seen that the present invention can not only remove the attribute data that needs to be cleaned, but also retain other attribute data that does not need to be cleaned, thereby improving the data cleaning quality of the present invention.

[0106] This embodiment also provides a data cleaning device based on data attributes, such as Figure 2 As shown, the device comprises the following components:

[0107] The data reading module 01 is used to obtain a real-time data group, which includes real-time data corresponding to each attribute of the target individual;

[0108] The data cleaning module 02 is used to filter out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each attribute, and remove the data to be cleaned in the real-time data group;

[0109] The data storage module 03 is used to store the real-time data group after removing the data to be cleaned.

[0110] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 3 As shown. The terminal device includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data cleaning method based on data attributes is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0111] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and a data cleaning program based on data attributes stored in the memory and executable on the processor. When the processor executes the data cleaning program based on data attributes, the following operation instructions are implemented:

[0113] Acquire a real-time data group, the real-time data group including real-time data corresponding to each attribute of the target individual;

[0114] Filtering out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group;

[0115] The real-time data group is saved after the data to be cleaned is removed.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data cleaning method based on data attributes, characterized in that: include: Acquire a real-time data group, the real-time data group including real-time data corresponding to each attribute of the target individual; Filtering out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group; The real-time data group is saved after the data to be cleaned is removed.

2. The data cleaning method based on data attributes according to claim 1, characterized in that: The obtaining of the real-time data group comprises: Filter out the attribute with the worst stability from each of the attributes; Get the shortest stable duration corresponding to the attribute with the worst stability; Determining a collection frequency of a collection device according to the shortest stable duration, wherein the collection device is used to collect attributes of the target individual; The acquisition device is controlled to acquire the real-time data group at the acquisition frequency, and the real-time data group is acquired through the acquisition device.

3. The data cleaning method based on data attributes according to claim 1, characterized in that: The step of selecting the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and removing the data to be cleaned in the real-time data group, includes: Determine the attribute filtering condition and the attribute deduplication condition in the attribute cleaning condition; The data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition are screened out from the real-time data corresponding to each of the attributes, and the data to be filtered and the data to be deduplicated are used as the data to be cleaned, and the data to be filtered and the data to be deduplicated are removed from the real-time data group.

4. The data cleaning method based on data attributes according to claim 3, characterized in that: The filtering out the data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, taking the data to be filtered and the data to be deduplicated as the data to be cleaned, and removing the data to be filtered and the data to be deduplicated from the real-time data group, comprises: Determining a first total number of attributes corresponding to the attribute filtering condition; Determining a second total number of attributes corresponding to the attribute deduplication condition; When the first total number is greater than or equal to the second total number, filtering out the data to be filtered that meets the attribute filtering condition from the real-time data corresponding to each of the attributes, and removing the data to be filtered from the real-time data in the real-time data group to obtain a filtered data group; The data to be deduplicated that meets the attribute deduplication condition is screened out from the real-time data in the filtered data group, and the data to be deduplicated is removed from the real-time data in the filtered data group.

5. The data cleaning method based on data attributes according to claim 4, characterized in that: The method further comprises: filtering out the data to be filtered that meets the attribute filtering condition and the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, taking the data to be filtered and the data to be deduplicated as the data to be cleaned, and removing the data to be filtered and the data to be deduplicated from the real-time data group, and further comprising: When the first total number is less than the second total number, filtering out the data to be deduplicated that meets the attribute deduplication condition from the real-time data corresponding to each of the attributes, and removing the data to be deduplicated from the real-time data in the real-time data group to obtain a deduplicated data group; The data to be filtered that meets the attribute filtering condition is screened out from the real-time data in the deduplicated data group, and the data to be filtered is removed from the real-time data in the deduplicated data group.

6. The data cleaning method based on data attributes according to any one of claims 3 to 5, characterized in that: The attribute deduplication condition is that the real-time data of the attribute is the same as the historical data of the attribute, or the attribute deduplication condition is that the difference between the real-time data of the attribute and the historical data of the attribute is less than a threshold.

7. The data cleaning method based on data attributes according to any one of claims 1 to 5, characterized in that: The step of saving the real-time data group after removing the data to be cleaned includes: The real-time data group after removing the data to be cleaned is saved in a stack of the cloud platform.

8. A data cleaning device based on data attributes, characterized in that: The device comprises the following components: A data reading module is used to obtain a real-time data group, which includes real-time data corresponding to various attributes of the target individual; A data cleaning module, used to filter out the data to be cleaned that meets the attribute cleaning condition from the real-time data corresponding to each of the attributes, and remove the data to be cleaned in the real-time data group; The data storage module is used to store the real-time data group after removing the data to be cleaned.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a data cleaning program based on data attributes stored in the memory and executable on the processor. When the processor executes the data cleaning program based on data attributes, the steps of the data cleaning method based on data attributes as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data cleaning program based on data attributes. When the data cleaning program based on data attributes is executed by a processor, the steps of the data cleaning method based on data attributes as described in any one of claims 1 to 7 are implemented.

Images