Data cleaning method and system based on AI

Abstract

The invention provides a data cleaning method and system based on AI, and belongs to the technical field of data cleaning. The method comprises the following steps: constructing a data importance judgment standard and a data cleaning model; collecting data in real time, classifying the data in real time according to the data importance judgment standard, and pasting a mark convenient for subsequent identification and analysis on the data; and obtaining the heat change of the data, and cooling and degrading or warming the data step by step into hot data, cold data, ice data and sealed data according to the data heat classification standard and the mark. The problems that in the prior art, in the data cleaning process, judgment on data importance is rigid and inaccurate, and mistakenly deleted data cannot be reproduced are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and specifically to an AI-based data cleaning method and system. Background Art

[0002] A large amount of data is generated during the use of cloud services. For example, the design and production process of creative digital content will generate data including but not limited to user information, raw data, rendering process data, final result data, and cached data. The accumulation of asset value generated by these data is a complex and continuous process. These data are not only crucial for the training of large AI models, but can also be used to analyze user behavior and achieve personalized customization. As the number of users participating in cloud services increases, the amount of data generated is also growing dramatically, including a large amount of process data and phased data, which brings great pressure to data storage, training, and learning.

[0003] At present, data cleaning is mainly based on time standards, that is, data is cleaned after a specific time period. Although this method is simple, it lacks judgment on the importance of data and may cause important or iconic data to be deleted by mistake. In addition, there are some data cleaning methods that use the number of data calls and the number of user visits to the data as standards, that is, after a specific time period, the data whose number of calls and user visits are both lower than the set value are cleaned. For example, within 10 days, if the number of calls of data A is less than 1,000 times and the number of user visits is less than 200, then data A will be deleted. Although this method takes the importance of data into consideration, the importance judgment standard is single, which may still cause important or iconic data to be deleted by mistake.

[0004] The inventors of the present invention have found that the above technology has at least the following technical problems:

[0005] 1. The data cleaning process lacks judgment on the importance of data or the judgment criteria are rigid and inaccurate, which can easily lead to accidental deletion of data;

[0006] 2. Data errors cannot be restored after being cleaned up. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the prior art and provide an AI-based data cleaning method and system to solve the problems in the prior art of rigid and inaccurate judgment of data importance and irreproducibility of accidentally deleted data in the data cleaning process.

[0008] In order to achieve the above objectives and other objectives, the present invention is implemented by including the following technical solutions: As a first aspect, the present invention proposes an AI-based data cleaning method, the method comprising the steps of:

[0009] Establish data importance judgment criteria and data popularity classification criteria;

[0010] Collect data in real time, classify the data in real time according to the data importance judgment criteria, and affix tags to facilitate subsequent identification and analysis;

[0011] The heat change of the data is obtained, and the data is gradually cooled and degraded or warmed up into hot data, cold data, frozen data and sealed data according to the data heat classification standard and the mark.

[0012] In one embodiment, the “2. Constructing a data importance judgment standard and a data cleaning model” specifically includes the following steps:

[0013] Collect historical data and use AI to classify and tag the historical data;

[0014] Through AI learning, the historical usage of the marked historical data is obtained to obtain the heat change of the historical data. According to the heat change, the historical data is cooled and degraded from hot data to cold data and ice data in sequence, and a data importance judgment standard and a data heat classification standard are formed;

[0015] AI is used to determine whether the iconic data in the ice data can be restored through model training. If so, it is deleted; if not, the iconic data is saved as archived data.

[0016] In one embodiment, during the step-by-step cooling and degradation or warming-up process of the data, the label is adjusted in real time by AI according to the criteria for judging the importance of the data.

[0017] In one embodiment, the data heat classification standard is: when at least two of the heat indicators of the data show positive growth, the data is hot data; when only one of the heat indicators of the data shows positive growth, the data is cold data; when all the heat indicators of the data show negative growth, the data is ice data; and the non-reproducible iconic data in the ice data is sealed data.

[0018] In one embodiment, the popularity index includes three or more of the number of visits, return visit rate, time or visit duration.

[0019] In one embodiment, the iconic data refers to important data that the user will frequently access during subsequent use.

[0020] In one embodiment, the similarity between the landmark data in the newly generated data and the sealed data is determined. When the landmark data in the newly generated data is similar to the sealed data, it is further determined whether there is a difference between the landmark data in the newly generated data and the sealed data. If so, the landmark data in the newly generated data is retained after marking the difference; if not, the sealed data is retained; when the landmark data in the newly generated data is not similar to the sealed data, all the data are retained at the same time.

[0021] In one embodiment, the method further comprises the step of overwriting and updating the background data when the user has new preference settings or updates user information.

[0022] As a second aspect, the present invention provides an AI-based data cleaning system, the system is applied to the method described in any one of the first aspects, the system comprising

[0023] Construction module, used to construct data importance judgment criteria and data popularity classification criteria;

[0024] The first collection module is used to collect data in real time during the use of cloud services;

[0025] A classification and marking module, used for classifying and marking the data;

[0026] The second acquisition module is used to obtain the thermal changes of the data during use.

[0027] A data degradation module, used for gradually cooling and degrading or warming up the data according to the data heat classification standard and label;

[0028] The update iteration module is used to update and iterate the data.

[0029] As a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the methods described in the first aspect.

[0030] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0031] The present invention utilizes AI technology to classify and evaluate data, and manages the data in a hierarchical manner according to the popularity and importance of the data. For process and phased transition data, its popularity is gradually reduced according to its usage, and factors such as importance and reproducibility are judged to clean and update the data. This can prevent the accidental deletion of key data to the greatest extent and ensure the security of key data. At the same time, through dynamic changes in heat indicators and real-time importance judgments, transition data whose popularity has rebounded can be retained, making data cleaning more flexible and accurate and improving data processing efficiency. In addition, through the derivative expansion of sealed data, repeated storage of the same part of the data can be avoided, thereby reducing the occupancy of data storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Shown is a flow chart of an AI-based data cleaning method of the present invention.

[0033] Figure 2 It is a schematic diagram of the process of step-by-step cooling degradation or temperature recovery of data in the present invention.

[0034] Figure 3 Shown is a schematic diagram of the process of iterative data update in the present invention.

[0035] Figure 4 Shown is a structural schematic diagram of an AI-based data cleaning system of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention provide an AI-based data cleaning method and system to solve the problems in the prior art of rigid and inaccurate judgment of data importance and irreproducibility of accidentally deleted data in the data cleaning process.

[0037] Below, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0038] The overall idea of ​​the technical solution provided by the present invention is as follows: a data cleaning method based on AI, the method comprising:

[0039] Establish data importance judgment criteria and data popularity classification criteria;

[0040] Collect data in real time, classify the data in real time according to the data importance judgment criteria, and affix tags to facilitate subsequent identification and analysis;

[0041] The heat change of the data is obtained, and the data is gradually cooled and degraded or warmed up into hot data, cold data, frozen data and sealed data according to the data heat classification standard and the mark.

[0042] The main idea is to use AI technology to classify and evaluate data, manage the data in different levels according to its popularity and importance, and gradually reduce the popularity of process and stage transition data according to its usage. In addition, the importance, reproducibility and other factors are judged to clean and update the data. This can prevent the accidental deletion of key data to the greatest extent and ensure the security of key data. At the same time, through dynamic changes in heat indicators and real-time importance judgments, transition data whose popularity has rebounded can be retained, making data cleaning more flexible and accurate, and improving data processing efficiency.

[0043] After introducing the basic principles of the present invention, various non-limiting implementation methods of the present invention will be specifically described below in conjunction with the accompanying drawings.

[0044] Embodiment 1

[0045] Please refer to the attached Figure 1-2 , an embodiment of the present invention provides an AI-based data cleaning method, the method comprising the steps of:

[0046] S1. Constructing data importance judgment criteria and data popularity classification criteria; specifically including the following steps:

[0047] S11. Collect historical data generated by users when using cloud services, and use AI algorithms to classify and tag the historical data;

[0048] Taking cloud rendering data as an example, it can be divided into iconic data and non-iconic data according to the importance of rendering data; it can be divided into design data and production data according to the type of rendering operation; it can be divided into task initial data, process data, result data, cache data, etc. according to the rendering task process; it can also be divided into background data (including user basic information, user configuration, user preference settings, etc.) and task processing data (rendering task data, rendering result data, performance monitoring data, resource scheduling data, rendering material data, etc.). Since data under different classification standards often overlap, it is necessary to subdivide and mark the data to represent the data in multiple ways. For example, the marking content can be "iconic data: importance 90 points; production data; result data".

[0049] Among them, iconic data refers to important data that users will frequently access during later use, such as rendering result data, rendering material data, etc. Non-iconic data refers to data that is not of high value and will not be specifically accessed by users in the later stage; such as cache data, performance monitoring data, etc. However, it should be noted that as the process of the rendering operation changes, the importance of the data will also change. For example, data A that is frequently accessed in the design stage is iconic data, but will no longer be accessed in the production stage. In this case, AI will adjust the data tag in real time by judging the rendering process. Specifically, when the rendering process is in the design stage, the tag content of data A is: "Iconic data: importance 50 points; design data; process data"; when the rendering process is in the production stage, the tag content of process data A is: "Non-iconic data; design data; process data".

[0050] The AI ​​algorithms include but are not limited to deep learning algorithms utilizing neural networks.

[0051] S12, learning the historical usage of the marked historical data through AI, obtaining the heat change of the historical data, and cooling and degrading the historical data from hot data to cold data and ice data in sequence according to the heat change, and forming a data importance judgment standard and data heat classification standard in the step-by-step cooling and degradation process;

[0052] The heat change refers to the quantitative change of the heat index showing a positive growth. The heat index includes the number of visits (the number of times the data is visited by others), the revisit rate (the frequency of data revisited by users), the time (the data residence time, the time between the next visit and the last visit of the data, and the time increase is negative growth), the access time (the length of time the data is accessed once), etc. At least three heat indicators need to be introduced in the step-by-step cooling degradation process, such as Figure 2 The three heat indicators of visit volume, revisit rate and time are introduced in the step-by-step cooling and degradation process of the data shown. When at least two heat indicators show positive growth, the data is identified as hot data, such as result data visited by multiple people in a short period of time, material data repeatedly revisited by users for a long period of time, etc.; when only one heat indicator shows positive growth, the data is identified as cold data, such as material data that no one has visited for a short period of time, resource scheduling data that no one has viewed for a short period of time, etc.; when all heat indicators show negative growth, the data is identified as cold data, such as rendering log data that has not been viewed for a long time and has not been revisited by users, etc.

[0053] The data importance judgment standard refers to judging whether the data is iconic based on data scarcity, data characteristics, data effects, etc. For example, the initial data of the task, result data, rendering material data, etc. are iconic data, which need to be retained when the data is cooled and degraded; the logs and error reports generated during the task processing are non-iconic data, which can be cleared when the data is cooled and degraded. According to the changes in the use process of cloud services, there are different judgment standards for iconic data at different data degradation levels, and AI is required to adjust the data labeling in real time according to the situation.

[0054] S13, judging through AI whether the landmark data in the ice data can be restored through model training, and if so, deleting it; if not, saving the landmark data as sealed data;

[0055] The iconic data in the ice data, for example, result files that have not been accessed for a long time but are of high importance.

[0056] S2. Collect data in real time, classify the data in real time according to the data importance judgment standard, and affix a mark to facilitate subsequent identification and analysis;

[0057] S3, obtaining the heat change of the data, and cooling and degrading or warming up the data step by step into hot data, cold data, frozen data and sealed data according to the data heat classification standard and the mark;

[0058] The step-by-step cooling and degradation means that when the number of data heat indexes with positive growth decreases, AI identifies and deletes the non-significant data at this stage, and the current sign data has its heat reduced by one level. The warming-up means that when the number of heat indexes of the sign data with positive growth increases, the data has its heat increased by one level.

[0059] For example: task processing process data that has not been accessed for a long time has reduced popularity and is judged as cold data. Combined with the characteristics of its process data, it is judged as non-iconic data and is therefore determined to be deleted for cleaning; data that has not been accessed for a certain period of time and has not been revisited by users but is iconic data has reduced popularity and is judged as cold data. Combined with the characteristics of its iconic data, it is judged to be important data. AI determines whether the data can be restored through model training. If the data can be reproduced after model training and no new changes will occur, it is determined that the data can be cleaned; if the iconic data cannot be restored through AI modeling learning, it will be saved in the archived database.

[0060] S4, determining the similarity between the landmark data in the newly generated data and the archived data, and performing derivative expansion of the archived data; the specific steps include:

[0061] When the landmark data in the newly generated data is similar to the archived data, further determine whether there is a difference between the landmark data in the newly generated data and the archived data, if so, mark the difference and retain the landmark data in the newly generated data; if not, retain the archived data;

[0062] When the landmark data in the newly generated data is not similar to the archived data, all the data are retained at the same time.

[0063] For example: classify task processing data according to usage scenarios, such as animation character rendering, interior design rendering, industrial design rendering, etc. Perform derivative expansion on the iconic data in the archived data (such as rendering result files, computing power scheduling plans, rendering material usage, etc.), and compare the newly generated iconic data with the existing archived data of the same usage scenario. If the data is the same, retain the original data. If the data is different but still similar, mark the derived differences and replace the new data. If the data difference is so large that there is no similar data, retain all the data at the same time to avoid repeatedly saving the same part of the data and reduce storage space.

[0064] S5. When the user has new preference settings or updates user information, the background data is overwritten and updated to facilitate the user's rendering operation.

[0065] Embodiment 2

[0066] Based on the same inventive concept as the AI-based data cleaning method in the aforementioned embodiment, the present invention further provides an AI-based data cleaning system, comprising:

[0067] Construction module 101, used to construct data importance judgment standard and data popularity classification standard;

[0068] The first collection module 102 is used to collect data in real time during the use of cloud services;

[0069] A classification and marking module 103, used for classifying and marking the data;

[0070] The second acquisition module 104 is used to obtain the thermal changes of the data during use.

[0071] The data degradation module 105 is used to perform step-by-step cooling and degradation or temperature recovery on the data according to the data heat classification standard and label;

[0072] The update iteration module 106 is used to update and iterate the data.

[0073] The foregoing Figure 1The various variations and specific examples of the AI-based data cleaning method in Example 1 are also applicable to an AI-based data cleaning system in this embodiment. Through the above detailed description of an AI-based data cleaning method, those skilled in the art can clearly know the implementation method of an AI-based data cleaning system in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.

[0074] Embodiment 3

[0075] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method of the first embodiment is implemented.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the same technology of the present invention, the present invention is also intended to include these modifications and variations.

Claims

1. A data cleaning method based on AI, characterized in that: The method comprises the steps of: Establish data importance judgment criteria and data popularity classification criteria; Collect data in real time, classify the data in real time according to the data importance judgment criteria, and affix tags to facilitate subsequent identification and analysis; The heat change of the data is obtained, and the data is gradually cooled and degraded or warmed up into hot data, cold data, frozen data and sealed data according to the data heat classification standard and the mark.

2. The AI-based data cleaning method according to claim 1, characterized in that: The "constructing data importance judgment criteria and data cleaning model" specifically includes the following steps: Collect historical data and use AI to classify and tag the historical data; Through AI learning, the historical usage of the marked historical data is obtained to obtain the heat change of the historical data. According to the heat change, the historical data is cooled and degraded from hot data to cold data and ice data in sequence, and a data importance judgment standard and a data heat classification standard are formed; AI is used to determine whether the iconic data in the ice data can be restored through model training. If so, it is deleted; if not, the iconic data is saved as archived data.

3. The AI-based data cleaning method according to claim 1 or 2, characterized in that: During the step-by-step cooling and degradation or warming-up process of the data, the mark is adjusted in real time by AI according to the criteria for judging the importance of the data.

4. The AI-based data cleaning method according to claim 1, characterized in that: The data heat classification standard is: when at least two of the heat indicators of the data show positive growth, the data is hot data; when only one of the heat indicators of the data shows positive growth, the data is cold data; when all the heat indicators of the data show negative growth, the data is ice data; the landmark data in the ice data that cannot be reproduced is sealed data.

5. The AI-based data cleaning method according to claim 4, characterized in that: The heat index includes three or more of the number of visits, return visit rate, time or visit duration.

6. The AI-based data cleaning method according to claim 2 or 4, characterized in that: The iconic data refers to important data that users will frequently access during subsequent use.

7. The AI-based data cleaning method according to claim 1, characterized in that: The method also includes the steps of: determining the similarity between the landmark data in the newly generated data and the sealed data; when the landmark data in the newly generated data is similar to the sealed data, further determining whether there is a difference between the landmark data in the newly generated data and the sealed data; if so, marking the difference and retaining the landmark data in the newly generated data; if not, retaining the sealed data; and when the landmark data in the newly generated data is not similar to the sealed data, retaining all the data at the same time.

8. The AI-based data cleaning method according to claim 1, characterized in that: The method further comprises the step of: when the user has new preference settings or updates user information, overwriting and updating the background data.

9. An AI-based data cleaning system, characterized in that: The system is applied to the method according to any one of claims 1 to 8, and the system comprises Construction module, used to construct data importance judgment criteria and data popularity classification criteria; The first collection module is used to collect data in real time during the use of cloud services; A classification and marking module, used for classifying and marking the data; The second acquisition module is used to obtain the thermal changes of the data during use. A data degradation module, used for gradually cooling and degrading or warming up the data according to the data heat classification standard and label; The update iteration module is used to update and iterate the data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images