Data cleaning method and device, electronic equipment and computer program product

Abstract

The invention discloses a data cleaning method and device, electronic equipment and a computer program product. Relates to the field of artificial intelligence, and comprises the following steps: collecting M data records in a target database, and extracting data information of target data corresponding to the data records from each data record, M being a positive integer; data features of the target data are generated based on the data information, the data features are input into the decision tree model, a cleaning strategy of the target data is obtained, the data features at least comprise time features and frequency features, and the decision tree model takes each data feature as a node and takes information gain as a splitting condition of the node; and processing the target data based on the cleaning strategy to obtain a target database after data cleaning. By means of the method and device, the problem that in the related technology, when data in a database is cleaned, mistaken deletion exists is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a data cleaning method, device, electronic device, and computer program product. Background Art

[0002] With the rapid advancement of information technology, especially the advent of the big data era, databases have become indispensable infrastructure for businesses, organizations, and even personal information storage and processing. However, while databases offer convenience and efficiency, they also face the challenge of data explosion. With the in-depth expansion of various applications, the amount of data accumulated in databases has increased dramatically, placing higher demands on data management. Data cleansing is a crucial task in daily database operations and maintenance. Over time, databases inevitably accumulate a large amount of no longer needed information, such as historical transaction records and obsolete user profiles. This redundant data not only consumes expensive storage resources but also negatively impacts database read and write performance. For example, excessive historical data can slow queries because the database management system must scan and filter more records to meet query criteria. Similarly, data updates can be inefficient because the database management system may require additional processing to skip or ignore invalid data when modifying existing records.

[0003] Database cleanup strategies in related technologies rely on pre-set rules, such as timestamp-based automatic deletion mechanisms, or manual intervention and regular checks by administrators. However, this fixed-rule approach has significant limitations. On the one hand, it fails to consider the specific content and frequency of data usage, which may lead to the premature deletion of some still valuable data. On the other hand, due to the lack of intelligent analysis mechanisms, some inactive or irrelevant data may remain in the database for a long time and fail to be cleared in a timely manner. Furthermore, the manual cleanup process is time-consuming and error-prone, which not only increases the workload of operations and maintenance personnel, but also may lead to the omission or improper cleanup of critical data, thus affecting business continuity and data integrity.

[0004] With respect to the problem of accidental deletion when cleaning data in a database in related technologies, no effective solution has been proposed so far. Summary of the Invention

[0005] The main purpose of this application is to provide a data cleaning method, device, electronic device and computer program product to solve the problem of accidental deletion when cleaning data in a database in the related art.

[0006] To achieve the above objectives, according to one aspect of the present application, a data cleaning method is provided. The method comprises: collecting M data records from a target database, extracting data information of the target data corresponding to each data record from each data record, where M is a positive integer; generating data features of the target data based on the data information, inputting the data features into a decision tree model, and obtaining a cleaning strategy for the target data, where the data features include at least time features and frequency features, the decision tree model using each data feature as a node, and information gain as a node splitting condition; and processing the target data based on the cleaning strategy to obtain a cleaned target database.

[0007] Optionally, generating data characteristics of target data based on data information includes: determining the acquisition time, calculating the difference between the acquisition time and the target time, and obtaining the idle time of the target data, wherein the target time is the value closest to the acquisition time between the latest access time and the data modification time; determining a preset access period and the number of accesses to the target data at each moment within the preset access period, and calculating the average access frequency of the target data based on the number of accesses at each moment and the preset access period; and determining at least one of the following as the data characteristics of the target data: idle time, average access frequency, data size, and business weight.

[0008] Optionally, the decision tree model determines the cleaning strategy in the following manner: calculating the information entropy and conditional entropy of the target data based on the data characteristics; calculating the difference between the information entropy and the conditional entropy to obtain the information gain, and judging whether the information gain is greater than or equal to the gain threshold; when the information gain is greater than or equal to the gain threshold, determining the cleaning strategy of the target data to retain the data; when the information gain is less than the gain threshold, determining the cleaning strategy of the target data to remove the data.

[0009] Optionally, the method also includes: collecting new data from the target database every first preset time period, performing incremental training on the decision tree model based on the new data to obtain an updated decision tree model; counting the cleaning strategies of the updated decision tree model within the second preset time period, and determining the false deletion rate of the cleaning strategies within the second preset time period, wherein the false deletion rate is the probability that the cleaning strategy for data to be retained is to remove the data; when the false deletion rate is greater than or equal to the false deletion rate threshold, replacing the updated decision tree model with the decision tree model before the update.

[0010] Optionally, determining the false deletion rate of the cleaning strategy within the second preset time length includes: for target cleaning data whose cleaning strategy is to eliminate data, determining whether the target cleaning data is accessed or modified within the second preset time length; if the target cleaning data is accessed or modified within the second preset time length, determining the target cleaning data as false deletion data; calculating the ratio of the number of all false deletion data to the number of all target cleaning data to obtain the false deletion rate.

[0011] Optionally, generating data features of target data based on data information also includes: determining a data weight for each type of data information, and determining associated data with an associated relationship from multiple data information; for each group of associated data, performing a weighted sum based on the associated data and the data weight of the associated data to obtain a comprehensive feature, wherein the sum of the data weight of each group of associated data is a preset value; and determining the comprehensive feature as the data feature of the target data.

[0012] Optionally, extracting data information of the target data corresponding to each data record from the data record includes: extracting at least one of the following from the data record: creation time, data size, latest access time, data modification time, data access frequency and business weight of the target data; determining at least one of the following as the data information of the target data: creation time, data size, latest access time, data modification time, data access frequency and business weight.

[0013] To achieve the above-mentioned objectives, according to another aspect of the present application, a data cleaning device is provided. The device comprises: a collection unit for collecting M data records from a target database and extracting data information of the target data corresponding to each data record from each data record, where M is a positive integer; a generation unit for generating data features of the target data based on the data information, inputting the data features into a decision tree model, and obtaining a cleaning strategy for the target data, where the data features include at least time features and frequency features, and the decision tree model uses each data feature as a node, and uses information gain as a node splitting condition; and a processing unit for processing the target data based on the cleaning strategy to obtain a target database after data cleaning.

[0014] In an embodiment of the present application, M data records in a target database are collected, and data information of the target data corresponding to the data record is extracted from each data record, where M is a positive integer; data features of the target data are generated based on the data information, and the data features are input into a decision tree model to obtain a cleaning strategy for the target data, where the data features include at least: time features and frequency features, and the decision tree model uses each data feature as a node and information gain as a node splitting condition; the target data is processed based on the cleaning strategy to obtain a target database after data cleaning. By utilizing the intelligent analysis capability of the decision tree algorithm and combining multi-dimensional data features such as time features and frequency features to determine the cleaning strategy of the database, the purpose of automated and intelligent database cleaning and reduced error deletion rate is achieved, thereby achieving the technical effect of improving the accuracy of data cleaning, and further solving the technical problem of error deletion when cleaning data in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0016] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data cleaning method is shown;

[0017] Figure 2 is a flow chart of a data cleaning method provided according to an embodiment of the present application;

[0018] Figure 3 is a schematic diagram of a data cleaning system provided according to an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of a data cleaning device provided according to an embodiment of the present application;

[0020] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation portals for users to choose to agree or refuse the automated decision-making results; if the user chooses to refuse, the expert decision-making process will be entered.

[0024] Example 1

[0025] According to an embodiment of the present application, an embodiment of a method for data cleaning is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data cleaning method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as an MCU (Microcontroller Unit) or an FPGA (Field-Programmable Gate Array), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a USB (Universal Serial Bus) port (which may be included as one of the ports of a BUS (Business, bus)), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data cleaning method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned data cleaning method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0030] The display may be, for example, a touch screen liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0031] In the above operating environment, this application provides a data cleaning method. Figure 2 is a flow chart of a data cleaning method provided according to an embodiment of the present application, such as Figure 2 As shown, the method includes:

[0032] Step S201 , collecting M data records in a target database, and extracting data information of target data corresponding to each data record, where M is a positive integer.

[0033] In step S201, the target database may be a database of a financial institution, and data records are collected from the target database. The data information extracted from the data records may include but is not limited to data creation time create_time, data size c i , data modification time modify_time, data access frequency F i , Business Weight I i (This coefficient can be set according to the criticality of the business process to which the data belongs, for example, 1 for core business and 0.5 for auxiliary business).

[0034] Step S202: Generate data features of the target data based on the data information, input the data features into the decision tree model, and obtain a cleaning strategy for the target data, wherein the data features include at least: time features and frequency features. The decision tree model uses each data feature as a node and information gain as the node splitting condition.

[0035] In step S202, data features are extracted from the data information. The data features may include time features (such as survival time and idle time) that can reveal the timeliness of the data, and frequency features (access frequency) that reflect the activity and importance of the data. Survival time is the length of time from the time the data was created to the current time, indicating the duration of time the data has existed in the database since its creation. Idle time is the length of time from the time the data was last accessed or modified to the current time, indicating the length of time the data has not been operated recently. Access frequency is the number of times the data is accessed within a certain time window, which can reflect the activity of the data.

[0036] Time and frequency features are selected as node features for the decision tree. Using information gain as the basis for node splitting, the decision tree algorithm selects the optimal feature for splitting based on the calculated information gain. After each split, the optimal feature is again selected for the next level of splitting based on the information gain of the remaining features until a stopping condition is met (such as reaching the maximum depth, the number of samples in a node being less than a preset value, or the information gain being less than a preset threshold). A decision tree model is recursively constructed using a training dataset containing time and frequency features. Each leaf node represents a decision outcome, such as whether to retain, archive, or delete the data.

[0037] The target data is preprocessed to extract its time and frequency features, and the generated data features are input into the trained decision tree model. The model will make judgments based on the data features and output the corresponding cleaning strategy, that is, retain, archive or delete the data.

[0038] Step S203: Process the target data based on the cleaning strategy to obtain a target database after data cleaning.

[0039] In step S203, target data that meets the cleaning conditions is screened out based on the combination of data features described in the cleaning strategy. The screened target data is marked, and the cleaning operation is performed based on the mark. Such as physical deletion of data, logical deletion (such as setting is_deleted to true), archiving data (moving it to storage with low frequency of access), or performing data desensitization (removing sensitive information). Ensure that the cleaning operation does not destroy the integrity and consistency of the database, and avoid triggering database constraints such as foreign key relationships. After the cleaning operation is completed, the database is reorganized to reclaim storage space, optimize indexes, and reduce fragmentation.

[0040] The data cleaning method provided in the embodiment of the present application collects M data records in a target database, extracts data information of the target data corresponding to the data record from each data record, wherein M is a positive integer; generates data features of the target data based on the data information, inputs the data features into a decision tree model, and obtains a cleaning strategy for the target data, wherein the data features include at least: time features and frequency features, and the decision tree model uses each data feature as a node and information gain as a node splitting condition; processes the target data based on the cleaning strategy to obtain a target database after data cleaning, and determines the cleaning strategy of the database by utilizing the intelligent analysis capability of the decision tree algorithm and combining multi-dimensional data features such as time features and frequency features, thereby achieving the purpose of automated and intelligent database cleaning and reducing the error deletion rate, thereby achieving the technical effect of improving the accuracy of data cleaning, and further solving the technical problem of error deletion when cleaning data in the database.

[0041] In order to determine a more accurate data cleaning strategy, data features are extracted from the data information to input into a decision tree model to generate a data cleaning strategy. Optionally, in the data cleaning method provided in the embodiment of the present application, generating data features of the target data based on the data information includes: determining the acquisition time, calculating the difference between the acquisition time and the target time, and obtaining the idle time of the target data, wherein the target time is the value closest to the acquisition time between the latest access time and the data modification time; determining a preset access period and the number of accesses to the target data at each moment within the preset access period, and calculating the average access frequency of the target data based on the number of accesses at each moment and the preset access period; and determining at least one of the following as the data feature of the target data: idle time, average access frequency, data size, and business weight.

[0042] In some embodiments, the collection time may be a specific time point at which data feature extraction is performed. The idle time of the target data refers to the time difference from the last time it was accessed or modified (i.e., the target time) to the current collection time. The preset access period may be a fixed time period, such as one week, one month, or three months. During this period, the number of times the target data is accessed at each time point is recorded. Set a preset access period, such as 30 days; collect access records of the target data within the preset access period, and count the number of accesses at each time point (such as every day, every hour); calculate the average access frequency based on the collected number of accesses. For example, if the preset access period is 30 days, the formula for calculating the average access frequency is:

[0043]

[0044] Among them, the average access frequency F avgi , the preset access period is ΔT, then, where F ij is the number of accesses to data i at time j, where the current time is also the acquisition time. The data characteristics of the target data include at least one of the following: idle time, average access frequency, data size, or business weight. Idle time reflects the length of time that the data has not been used since it was last accessed or modified. The average access frequency indicates the access intensity of the data within a specified period. The data size can reflect the consumption of storage resources. Large data files may require priority cleaning or archiving even if they are rarely accessed. Business weight can be set based on the importance of the business process or data type to which the data belongs. Core business data or key data types will have a higher weight.

[0045] In this embodiment, by determining data features, the decision tree model can effectively make comprehensive judgments based on multiple data features of the target data and formulate a more refined and reasonable database cleaning strategy.

[0046] In order to generate a more accurate data cleaning strategy, a decision tree model needs to be trained. Optionally, in the data cleaning method provided in the embodiment of the present application, the decision tree model determines the cleaning strategy in the following manner: calculating the information entropy and conditional entropy of the target data based on the data characteristics; calculating the difference between the information entropy and the conditional entropy to obtain the information gain, and judging whether the information gain is greater than or equal to the gain threshold; when the information gain is greater than or equal to the gain threshold, determining the cleaning strategy of the target data to retain the data; when the information gain is less than the gain threshold, determining the cleaning strategy of the target data to eliminate the data.

[0047] In some embodiments, information entropy (H) represents the uncertainty or confusion of a data set. The information entropy is calculated as:

[0048]

[0049] where x is a random variable, and x i is the value taken by x, and p(x i ) is the probability of the occurrence of x i . Using the extracted eigenvalues as the node attributes of the decision tree, a decision tree model is constructed. For example, using c i , that is, the data size, s i , that is, the idle time, F avgi , that is, the average access frequency, and I i , that is, the service weight as attributes, a decision tree is constructed. Here, let x = s, y = c, F avgi and I i are included in the following formula for calculation as coefficients:

[0050]

[0051] where t i is the survival time of the data, that is, the difference between the data collection time and the creation time. H(y i ) is the information entropy of the target data. The conditional entropy is the uncertainty of (Y) given (X). The formula for calculating the conditional entropy is as follows:

[0052]

[0053] where y is another random variable. H(y|x) can be the uncertainty of the data size given the idle time of the data. Let the idle time s i >= 90 days, the long-term unaccessed subset, 30 days <= idle time s i < 90 days, the medium-term unaccessed subset, 0 <= idle time s[[ID=4));]] i < 30 days, the recently active subset. The data size can be divided into three categories: large, medium, and small according to the actual situation. Then when the data has been accessed in the last 30 days, p(large|0 <= idle time s i < 30 days) = a, p(medium|0 <= idle time s[[ID=4));]] i < 30 days) = b, p(small|0 <= idle time s i < 30 days) = d. Then H(c|0 <= idle time s i < 30 days) = H(large|0 <= idle time s i < 30 days) + H(medium|0 <= idle time s i < 30 days) + H(small|0 <= idle time s i<30 days) = -alog2a – blog2b – dlog2d. The constraint is: a + b + d = 1. Similarly, we can calculate the conditional entropy when the data has been accessed within 30-90 days and the conditional entropy when the data has not been accessed within 90 days.

[0054] Information gain: measures the ability of feature x to distinguish the target variable y. The calculation formula of information gain is as follows:

[0055] IG(y|x)=H(y)-H(y|x);

[0056] IG(y|x) stands for information gain, which indicates how much the noise in a dataset is reduced once the value of a data feature is known. The greater the information gain, the more important the feature. In a decision tree, the feature with the highest information gain is selected for node splitting. IG(cleaning decision | idle time) = H(cleaning decision) - H(cleaning decision | idle time).

[0057] Set a gain threshold, such as 0.2. If the calculated information gain is greater than or equal to this threshold, it means that the target data has a significant discriminatory effect on the data cleaning decision and the data should be evaluated as "retained." If the information gain is less than this threshold, the data features do not contribute much to the cleaning decision, indicating that the uncertainty or confusion of the target data has not been significantly reduced. Such data may be evaluated as "removed."

[0058] When a decision tree is constructed, the optimal splitting feature is recursively selected based on information gain. Each internal node represents a feature test, while a leaf node represents a cleaning decision (keep or remove). This recursive process continues until all data is classified into a leaf node or a preset stopping condition is reached, such as a maximum tree depth or a minimum number of leaf node samples. For example, if a decision tree model first splits data based on idle time, it calculates a threshold such that data with an idle time less than the threshold has different information gains than data with an idle time greater than or equal to the threshold. The model then continues searching for the optimal splitting feature on each subset until the model is complete. Once the decision tree model is constructed, it can be used to make cleaning decisions on the target data. Data features are input into the model and follow the decision tree path until a leaf node is reached. The cleaning strategy (keep or remove) stored in the leaf node is then applied to the target data. The results and feedback from the cleaning process (such as false deletion rate and cleaning efficiency) are recorded and used for subsequent model adjustments and optimization to ensure that the cleaning strategy is both effective and secure. For example, if the false deletion rate is high, the gain threshold can be adjusted or the feature importance can be reassessed to avoid similar issues.

[0059] For example, consider collecting user registration data containing multiple features such as username length, password complexity, and registration interval. Preprocess the data, for example, by normalizing numerical features like username length and registration interval to the same scale. For features like password complexity, which may be categorical, the data is encoded as 1 for simple, 2 for medium, and 3 for complex, respectively. A weighted summation method is used to calculate the comprehensive feature value of each sample. Weights are assigned to password complexity at 0.5, username length at 0.3, and registration interval at 0.2. A comprehensive feature value is calculated based on the feature values ​​of each sample, and node splitting is performed based on this comprehensive feature value. A threshold value Ba is set; features with a comprehensive feature value greater than the threshold are assigned to one category, and features with a comprehensive feature value less than the threshold are assigned to another category. Assuming there are n features x1, x2, …, xn, and corresponding weights w1, w2, …, wn, the formula for calculating the comprehensive feature value F is F = w1x1 + w2x2 + … + wnxn.

[0060] The weights of data features can be adjusted dynamically. For example, the feature importance Δw_i = α × current importance + (1-α) × historical mean is recalculated every week based on the feature value (the value of α depends on the actual situation).

[0061] When cleaning user registration information, let "Username Length" be x1 with a weight of w1; "Password Complexity" be x2 with a weight of w2; and "Registration Time Interval" be x3 with a weight of w3. The weights are configured to satisfy the conditions w2>w1>w3 and Σwi=1. First, standardize each feature. Suppose a user's "Username Length" has a standardized value of x11, "Password Complexity" has a standardized value of x22, and "Registration Time Interval" has a standardized value of x33. Then, the composite feature value F = w1 × x11 + w2 × x22 + w3 × x33. Decision tree model training parameters: Maximum tree depth: 8, Minimum number of leaf node samples: 1000. Information gain threshold: 0.05. Given the feature matrix: data access interval (normalized to 0.82); personal data integrity score (coded as 2); and privacy level (1-3 categories, Min-Max to 0.65), the fusion calculation result is: F = 0.4 * 0.82 + 0.4 * 2 / 3 + 0.2 * 0.65 = 0.328 + 0.267 + 0.13 = 0.725. Based on the classification conditions of the feature fusion calculation set above, the decision result is: 0.725 < 0.76, which means "keep".

[0062] This embodiment uses a decision tree model based on information entropy, conditional entropy, and information gain to achieve intelligent cleaning of database data, ensure high performance and high efficiency of the database, improve the accuracy of data cleaning strategies, and reduce the rate of false deletions.

[0063] In order to ensure the accuracy of the decision tree model, it is necessary to perform incremental training on the decision tree model regularly. Optionally, in the data cleaning method provided in the embodiment of the present application, the method also includes: collecting new data from the target database at intervals of a first preset time, and performing incremental training on the decision tree model based on the new data to obtain an updated decision tree model; statistically analyzing the cleaning strategies of the updated decision tree model within a second preset time, and determining the false deletion rate of the cleaning strategies within the second preset time, wherein the false deletion rate is the probability that the cleaning strategy for data to be retained is to remove the data; when the false deletion rate is greater than or equal to a false deletion rate threshold, replacing the updated decision tree model with the decision tree model before the update.

[0064] In some embodiments, the first preset time period can be one day, for example, it is set to 1 a.m. every day, and the incremental training process starts automatically. When the first preset time period is reached, all new data in the target database since the last training is automatically collected. These data include new data records and their related time features, frequency features, etc. Features related to the decision tree model are extracted from the new data, such as the creation time of the data, the last access time, the data size and the access frequency, etc. The features of the new data are fed into the existing decision tree model for incremental training. This step does not require building a model from scratch, but is based on an already trained model and only uses the new data for fine-tuning to reflect the latest changes in the data distribution in the database. Use online learning or semi-supervised learning techniques to update the parameters of the model, such as node splitting conditions, weight distribution, etc.

[0065] The second preset duration can be one week, for example, the next week, to observe and analyze the effectiveness of the cleanup policy. During the second preset duration, the updated decision tree model is used to make cleanup decisions for the data in the database. The false deletion rate refers to the proportion of data that should actually be retained but is mistakenly marked as "removed." The false deletion rate can be calculated by setting a calibration dataset containing data known to be retained; applying the updated decision tree model to predict the cleanup policy for the data in the calibration dataset; and calculating the proportion of data marked as "removed" by the model that should actually be "retained," i.e., the false deletion rate. For example, a false deletion rate threshold of 1% is set; if it exceeds this threshold, the model is considered to have an excessively high false deletion risk. If the false deletion rate statistics within the second preset duration are greater than or equal to the preset false deletion rate threshold (e.g., 1%), the decision tree model will be automatically rolled back to the pre-update version. For example, if the false deletion rate exceeds 1% for three consecutive days, the model will automatically roll back to the previous stable version, using the old model to make data cleanup decisions, ensuring the security and accuracy of the cleanup policy.

[0066] This embodiment ensures the effectiveness of the decision tree model by performing incremental training on the decision tree model, and avoids risks that may be caused by the cleanup strategy through dynamic adjustment, thereby achieving intelligent and robust database cleanup.

[0067] Optionally, in the data cleaning method provided in the embodiment of the present application, determining the false deletion rate of the cleaning strategy within the second preset time period includes: for target cleaning data whose cleaning strategy is to eliminate data, judging whether the target cleaning data is accessed or modified within the second preset time period; if the target cleaning data is accessed or modified within the second preset time period, determining the target cleaning data as false deletion data; and calculating the ratio of the number of all false deletion data to the number of all target cleaning data to obtain the false deletion rate.

[0068] In some embodiments, based on the cleaning strategy generated by the decision tree model within the second preset time period, all target cleaning data sets marked as "removed data" are identified. According to the judgment of the model, these data should be cleaned within this time period. During the second preset time period, the access and modification of the target cleaning data in the database are continuously monitored. For each data record marked as "removed data" within the second preset time period, check whether it has been accessed or modified within the time period. If a data record is accessed or modified, this indicates that the model's cleaning decision is wrong, because the data record is actually still active or important and should be retained instead of cleaned. The number of all data records identified as mistakenly deleted is counted. These data records are misjudged by the model's cleaning strategy. They should have been retained, but were mistakenly marked as "removed data" according to the model's decision. The mistaken deletion rate is the ratio of the number of all mistakenly deleted data to the number of all target cleaning data (i.e., data records marked as "removed data").

[0069] By calculating the false deletion rate, this embodiment can regularly evaluate the performance of the decision tree model, ensure that the database cleanup strategy is both effective and safe, and avoid unnecessary risks to business operations due to incorrect cleaning of important data.

[0070] The data features of the target data may also include comprehensive features. Optionally, in the data cleaning method provided in the embodiment of the present application, generating the data features of the target data based on the data information also includes: determining the data weight of each type of data information, and determining the associated data with an associated relationship from the multiple data information; for each group of associated data, performing weighted summation based on the associated data and the data weight of the associated data to obtain a comprehensive feature, wherein the sum of the data weight of each group of associated data is a preset value; and determining the comprehensive feature as the data feature of the target data.

[0071] In some embodiments, the decision tree model can split nodes based on a single feature. A multi-feature fusion strategy can also be designed. When selecting splitting features, the combined effect of multiple related features is comprehensively considered. For example, when cleaning medical data information in a database, multiple features such as "user name length", "password complexity", and "registration time interval" are combined into a new feature vector, which is converted into a comprehensive feature value that can be used for decision tree splitting through a specific fusion function (such as weighted summation, vector splicing and hashing, etc.). In this way, the correlation information between features can be captured, avoiding incomplete cleaning or erroneous cleaning problems caused by the limitations of a single feature, and is particularly suitable for processing data anomalies caused by the combined action of multiple factors.

[0072] For example, consider the feature fusion calculation process: the original value of username length is X1 = 15, the normalized value is 0.87, and the weight W1 is 0.3; the original value of password complexity is X2 = 2 (medium), the normalized value is 0.64, and the weight W2 is 0.5; the original value of registration interval is X3 = 72 hours, the normalized value is 0.91, and the weight W3 is 0.2. The combined feature value F = 0.3 × 0.87 + 0.5 × 0.64 + 0.2 × 0.91 = 0.763. The splitting condition for the decision tree model can be: if F ≥ 0.76, then the decision is "to be cleaned"; if F < 0.76, then the decision is "to be retained." The weighting principles are as follows: Password complexity is given the highest weight (w2 = 0.5) because it is strongly correlated with account security; registration interval is given the lowest weight (w3 = 0.2) because short registration times are likely normal behavior. Normalization methods include: Username length: Z-Score (zero-mean normalization) (μ = 10, σ = 5); Password complexity: Min-Max (linear scaling) normalization after segmented encoding (Simple = 1, Medium = 2, Complex = 3); Registration interval: Logarithmic transformation followed by normalization (log(x3 + 1) / max(log(x3 + 1))).

[0073] This embodiment uses the comprehensive features as the splitting conditions of the decision tree model, which can ensure that the decision tree model not only makes decisions based on single data information, but also takes into account the association relationship between data, thereby improving the accuracy and efficiency of the cleaning strategy.

[0074] In order to ensure the accuracy of the data cleaning strategy, it is necessary to extract data information from the target data. Optionally, in the data cleaning method provided in the embodiment of the present application, extracting the data information of the target data corresponding to the data record from each data record includes: extracting at least one of the following from the data record: the creation time, data size, latest access time, data modification time, data access frequency and business weight of the target data; and determining at least one of the following as the data information of the target data: creation time, data size, latest access time, data modification time, data access frequency and business weight.

[0075] In some embodiments, for each data record, a series of data information related to data retention decisions needs to be extracted. This information may include but is not limited to the following items: Creation time: This is the time point when the data first enters the database, which is used to evaluate the age of the data. Data size: The amount of storage space occupied by the data, which can help determine the storage cost of the data. Latest access time: The time when the data was last accessed, which is used to evaluate the activity of the data. Data modification time: The time when the data was last modified, which also reflects the activity and importance of the data. Data access frequency: The number of times or frequency that the data is accessed within a preset time window, which is used to quantify the frequency of data use. Business weight: The weight set according to the importance of the business process or data type to which the data belongs, reflecting the criticality of the data to business operations.

[0076] Through meticulous data information extraction and preprocessing, this embodiment can ensure that the decision tree model can make decisions based on the most relevant and representative features, thereby enhancing the practicality and accuracy of the model.

[0077] According to another embodiment of the present application, a data cleaning system is also provided. Figure 3 is a schematic diagram of a data cleaning system provided according to an embodiment of the present application, such as Figure 3 As shown, the system includes: a database real-time monitoring module, a data acquisition module, a feature extraction and preprocessing module, a decision tree model building module, a model version manager, a feedback data analyzer, a feature repository, a dynamic weight calculator, a cleaning decision module, a data cleaning executor, and a cleaning log and feedback module.

[0078] Among them, the database real-time monitoring module continuously monitors database activities, including data read and write operations, data size changes, data access patterns, etc., to timely capture the status and trends of the database. The data acquisition module collects data records from the database regularly or based on trigger events for subsequent feature extraction and model training. The feature extraction and preprocessing module extracts useful features (such as survival time, access frequency, etc.) from the collected data, and preprocesses these features (such as standardization, encoding, etc.) to meet the input requirements of the model. The decision tree model construction module uses a decision tree algorithm to build a model based on the extracted and preprocessed data features. The model is used to judge the retention value of the data and guide the cleanup strategy. The model version manager manages different versions of the decision tree model, including storage, retrieval, and switching between versions.

[0079] The feedback data analyzer analyzes the feedback data after the execution of the model cleanup strategy, such as the false deletion rate and the usage of retained data, for model optimization. The feature repository stores and manages the extracted and preprocessed data features for easy call during model training and prediction. The dynamic weight calculator dynamically adjusts the weight of the features in the decision tree model based on the real-time changes and historical statistics of the data features. The cleanup decision module makes a decision on whether to clean the data records in the database based on the output of the decision tree model. The data cleanup executor executes the data cleanup operations specified by the cleanup decision module, such as data deletion and archiving. The cleanup log and feedback module records the operation log of data cleanup, collects the execution results and potential problems of the cleanup decision, and provides feedback for model optimization and system improvement.

[0080] The feedback module deletes the corresponding data according to the cleanup plan and records the execution status. If, during the cleanup process, it is discovered that certain customer transaction records cannot be deleted due to associations, this information is fed back to the system, which can then adjust the decision tree model, such as adding data association features or adjusting the cleanup rules, to prevent similar issues in the future.

[0081] The data cleaning system of this embodiment uses a decision tree algorithm to perform intelligent analysis of database data, which can more accurately determine the data that needs to be cleaned, avoiding the problem of accidentally deleting useful data or failing to clean up useless data in a timely manner, and effectively improving the storage efficiency and performance of the database. It automatically adjusts the cleaning strategy according to the actual usage of the database, has good adaptability and scalability, and can be widely used in database management scenarios of various sizes and types, reducing the labor cost and complexity of database maintenance. By regularly cleaning the database, the interference of data redundancy and useless data on database query, update and other operations is reduced, the overall operating efficiency and data quality of the database are improved, and more reliable data support is provided for the business operations of the enterprise.

[0082] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0083] Example 2

[0084] The present application also provides a data cleaning device. It should be noted that the data cleaning device of the present application can be used to execute the data cleaning method provided in the present application. The data cleaning device provided in the present application is introduced below.

[0085] According to an embodiment of the present application, a device for implementing the above data cleaning method is also provided. Figure 4 is a schematic diagram of a data cleaning device provided according to an embodiment of the present application, such as Figure 4 As shown, the device includes:

[0086] The collection unit 401 is configured to collect M data records from a target database and extract data information of target data corresponding to each data record, where M is a positive integer.

[0087] A generating unit 402 is configured to generate data features of the target data based on the data information, input the data features into a decision tree model, and obtain a cleaning strategy for the target data, wherein the data features include at least time features and frequency features. The decision tree model uses each data feature as a node, and uses information gain as a node splitting condition.

[0088] The processing unit 403 is configured to process the target data based on the cleaning strategy to obtain a target database after data cleaning.

[0089] The data cleaning device provided in the embodiment of the present application collects M data records in the target database through the collection unit 401, and extracts data information of the target data corresponding to the data record from each data record, wherein M is a positive integer; the generation unit 402 generates data features of the target data based on the data information, inputs the data features into the decision tree model, and obtains a cleaning strategy for the target data, wherein the data features at least include: time features and frequency features, and the decision tree model uses each data feature as a node and uses information gain as a node splitting condition; the processing unit 403 processes the target data based on the cleaning strategy to obtain a target database after data cleaning, and determines the database cleaning strategy by utilizing the intelligent analysis capability of the decision tree algorithm and combining multi-dimensional data features such as time features and frequency features, thereby achieving the purpose of automated and intelligent database cleaning and reducing the error deletion rate, thereby achieving the technical effect of improving the accuracy of data cleaning, and further solving the technical problem of error deletion when cleaning data in the database.

[0090] Optionally, in the data cleaning device provided in the embodiment of the present application, the generation unit 402 includes: a first determination module, used to determine the acquisition time, calculate the difference between the acquisition time and the target time, and obtain the idle time of the target data, wherein the target time is the value closest to the acquisition time between the latest access time and the data modification time; a second determination module, used to determine the preset access cycle and the number of accesses to the target data at each moment within the preset access cycle, and calculate the average access frequency of the target data based on the number of accesses at each moment and the preset access cycle; a third determination module, used to determine at least one of the following as the data feature of the target data: idle time, average access frequency, data size and business weight.

[0091] Optionally, in the data cleaning device provided in the embodiment of the present application, the device also includes: a calculation unit, used to calculate the information entropy and conditional entropy of the target data based on data characteristics; a judgment unit, used to calculate the difference between the information entropy and the conditional entropy, obtain the information gain, and judge whether the information gain is greater than or equal to the gain threshold; a first determination unit, used to determine the cleaning strategy of the target data as retaining the data when the information gain is greater than or equal to the gain threshold; and a second determination unit, used to determine the cleaning strategy of the target data as eliminating the data when the information gain is less than the gain threshold.

[0092] Optionally, in the data cleaning device provided in the embodiment of the present application, the device also includes: a training unit, used to collect new data from the target database every first preset time period, and perform incremental training on the decision tree model based on the new data to obtain an updated decision tree model; a statistical unit, used to count the cleaning strategies of the updated decision tree model within the second preset time period, and determine the false deletion rate of the cleaning strategies within the second preset time period, wherein the false deletion rate is the probability that the cleaning strategy for data to be retained is to eliminate data; a replacement unit, used to replace the updated decision tree model with the decision tree model before the update when the false deletion rate is greater than or equal to a false deletion rate threshold.

[0093] Optionally, in the data cleaning device provided in the embodiment of the present application, the statistical unit includes: a judgment module, used to judge whether the target cleaning data whose cleaning strategy is to eliminate data has been accessed or modified within a second preset time length; a fourth determination module, used to determine the target cleaning data as erroneously deleted data if the target cleaning data has been accessed or modified within the second preset time length; and a calculation module, used to calculate the ratio of the number of all erroneously deleted data to the number of all target cleaning data to obtain the erroneous deletion rate.

[0094] Optionally, in the data cleaning device provided in the embodiment of the present application, the generation unit 402 also includes: a fifth determination module, used to determine the data weight of each data information, and determine the associated data with an associated relationship from multiple data information; a summation module, used to perform weighted summation based on the associated data and the data weight of the associated data for each group of associated data to obtain a comprehensive feature, wherein the sum of the data weight of each group of associated data is a preset value; and a sixth determination module, used to determine the comprehensive feature as the data feature of the target data.

[0095] Optionally, in the data cleaning device provided in an embodiment of the present application, the collection unit 401 includes: an extraction module for extracting at least one of the following from the data record: creation time, data size, latest access time, data modification time, data access frequency and business weight of the target data; a seventh determination module for determining at least one of the following as data information of the target data: creation time, data size, latest access time, data modification time, data access frequency and business weight.

[0096] It should be noted that the above-mentioned acquisition unit 401, generation unit 402, and processing unit 403 correspond to steps S201 to S203 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above-mentioned modules or units can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0097] Example 3

[0098] An embodiment of the present application may provide an electronic device, Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one is shown) processor 502, memory 504, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0099] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0100] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collect M data records in the target database, extract data information of the target data corresponding to the data record from each data record, where M is a positive integer; generate data features of the target data based on the data information, input the data features into the decision tree model, and obtain a cleaning strategy for the target data, where the data features include at least: time features and frequency features, the decision tree model uses each data feature as a node, and uses information gain as the splitting condition of the node; process the target data based on the cleaning strategy to obtain the target database after data cleaning.

[0101] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the acquisition time, calculate the difference between the acquisition time and the target time, and obtain the idle time of the target data, wherein the target time is the value closest to the acquisition time between the latest access time and the data modification time; determine the preset access cycle and the number of accesses to the target data at each moment within the preset access cycle, and calculate the average access frequency of the target data based on the number of accesses at each moment and the preset access cycle; determine at least one of the following as the data feature of the target data: idle time, average access frequency, data size and business weight.

[0102] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: calculating the information entropy and conditional entropy of the target data based on the data characteristics; calculating the difference between the information entropy and the conditional entropy to obtain the information gain, and judging whether the information gain is greater than or equal to the gain threshold; when the information gain is greater than or equal to the gain threshold, determining the cleaning strategy of the target data to retain the data; when the information gain is less than the gain threshold, determining the cleaning strategy of the target data to eliminate the data.

[0103] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: collect new data from the target database every first preset time period, perform incremental training on the decision tree model based on the new data, and obtain an updated decision tree model; count the cleaning strategies of the updated decision tree model within the second preset time period, and determine the false deletion rate of the cleaning strategies within the second preset time period, wherein the false deletion rate is the probability that the cleaning strategy for data to be retained is to remove the data; when the false deletion rate is greater than or equal to the false deletion rate threshold, replace the updated decision tree model with the decision tree model before the update.

[0104] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: for the target cleaning data whose cleaning strategy is to eliminate data, determine whether the target cleaning data is accessed or modified within the second preset time length; if the target cleaning data is accessed or modified within the second preset time length, determine the target cleaning data as mistakenly deleted data; calculate the ratio of the number of all mistakenly deleted data to the number of all target cleaning data to obtain the mistaken deletion rate.

[0105] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the data weight of each data information, and determine the associated data with an associated relationship from multiple data information; for each group of associated data, perform weighted summation based on the associated data and the data weight of the associated data to obtain a comprehensive feature, wherein the sum of the data weight of each group of associated data is a preset value; and determine the comprehensive feature as the data feature of the target data.

[0106] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extract at least one of the following from the data record: the creation time, data size, latest access time, data modification time, data access frequency and business weight of the target data; determine at least one of the following as the data information of the target data: creation time, data size, latest access time, data modification time, data access frequency and business weight.

[0107] According to an embodiment of the present application, a method is provided for collecting M data records in a target database, extracting data information of the target data corresponding to the data record from each data record, wherein M is a positive integer; generating data features of the target data based on the data information, inputting the data features into a decision tree model, and obtaining a cleaning strategy for the target data, wherein the data features include at least: time features and frequency features, and the decision tree model uses each data feature as a node and information gain as a node splitting condition; processing the target data based on the cleaning strategy, and obtaining a solution for the target database after data cleaning. By utilizing the intelligent analysis capability of the decision tree algorithm and combining multi-dimensional data features such as time features and frequency features to determine the cleaning strategy of the database, the purpose of automating and intelligentizing database cleaning and reducing the error deletion rate is achieved, thereby achieving the technical effect of improving the accuracy of data cleaning, and further solving the technical problem of the existence of error deletion when cleaning data in the database.

[0108] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 5 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 5 Different configurations shown.

[0109] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0110] Example 4

[0111] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data cleaning method provided in the first embodiment.

[0112] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0113] The present application also provides a computer program product, which is suitable for executing the steps of the data cleaning method when executed on a data processing device.

[0114] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0117] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0120] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data cleaning method, characterized in that: include: Collect M data records from the target database, and extract data information of the target data corresponding to each data record from the data record, where M is a positive integer; Generating data features of the target data based on the data information, inputting the data features into a decision tree model to obtain a cleaning strategy for the target data, wherein the data features include at least time features and frequency features, and the decision tree model uses each data feature as a node and uses information gain as a node splitting condition; The target data is processed based on the cleaning strategy to obtain a target database after data cleaning.

2. The method according to claim 1, characterized in that Generating the data features of the target data based on the data information includes: Determine the acquisition time, calculate the difference between the acquisition time and the target time, and obtain the idle time of the target data, wherein the target time is the value closest to the acquisition time between the latest access time and the data modification time; Determining a preset access period and the number of times the target data is accessed at each moment within the preset access period, and calculating an average access frequency of the target data based on the number of accesses at each moment and the preset access period; At least one of the following is determined as the data feature of the target data: the idle time, the average access frequency, the data size, and the service weight.

3. The method according to claim 1, characterized in that The decision tree model determines the cleaning strategy in the following way: Calculating the information entropy and conditional entropy of the target data based on the data features; Calculating the difference between the information entropy and the conditional entropy to obtain information gain, and determining whether the information gain is greater than or equal to a gain threshold; In a case where the information gain is greater than or equal to the gain threshold, determining the cleaning strategy of the target data to retain data; When the information gain is less than the gain threshold, the cleaning strategy of the target data is determined to be data removal.

4. The method according to claim 1, wherein The method further comprises: Collecting new data from the target database at intervals of a first preset time, and performing incremental training on the decision tree model based on the new data to obtain an updated decision tree model; Counting the cleaning strategies of the updated decision tree model within a second preset time period, and determining a false deletion rate of the cleaning strategies within the second preset time period, wherein the false deletion rate is a probability that the cleaning strategy for the data to be retained is to remove the data; When the false deletion rate is greater than or equal to a false deletion rate threshold, the updated decision tree model is replaced with the decision tree model before the update.

5. The method according to claim 4, characterized in that Determining the false deletion rate of the cleaning strategy within the second preset time period includes: For target cleaning data whose cleaning policy is to remove data, determining whether the target cleaning data has been accessed or modified within the second preset time period; If the target cleaning data is accessed or modified within the second preset time period, determining the target cleaning data as accidentally deleted data; The ratio of the number of all mistakenly deleted data to the number of all target cleaned data is calculated to obtain the mistaken deletion rate.

6. The method according to claim 1, characterized in that Generating the data features of the target data based on the data information also includes: Determine the data weight of each data information, and determine the associated data with associated relationships from multiple data information; For each set of associated data, a weighted sum is performed based on the associated data and the data weights of the associated data to obtain a comprehensive feature, wherein the sum of the data weights of each set of associated data is a preset value; The comprehensive feature is determined as the data feature of the target data.

7. The method according to claim 1, characterized in that Extracting the target data corresponding to each data record from the data record includes: Extract at least one of the following from the data record: creation time, data size, latest access time, data modification time, data access frequency, and service weight of the target data; At least one of the following is determined as the data information of the target data: the creation time, the data size, the latest access time, the data modification time, the data access frequency, and the business weight.

8. A data cleaning device, characterized in that: include: a collection unit, configured to collect M data records from a target database and extract data information of target data corresponding to each data record, wherein M is a positive integer; a generating unit, configured to generate data features of the target data based on the data information, input the data features into a decision tree model, and obtain a cleaning strategy for the target data, wherein the data features include at least time features and frequency features, the decision tree model uses each data feature as a node, and uses information gain as a node splitting condition; A processing unit is used to process the target data based on the cleaning strategy to obtain a target database after data cleaning.

9. An electronic device, characterized in that: include: a memory storing an executable program; A processor is used to run the program, wherein the program executes the data cleaning method according to any one of claims 1 to 7 when running.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the data cleaning method according to any one of claims 1 to 7 are implemented.

Images