Junk data cleaning method and device, equipment and storage medium

Abstract

The invention relates to a junk data cleaning method and device, equipment and a storage medium. The method comprises the following steps: monitoring data generated by a cloud native application; determining statistical characteristics corresponding to the data according to records corresponding to the data; extracting data features in the data according to the data type of the data; performing junk data identification on the data by using a pre-trained junk data identification model according to the statistical features and the data features, and outputting an identification result; and when the identification result shows that the data is the junk data, executing a junk data cleaning task for the data. According to the method, the data features which can be used for reflecting the data and the statistical features which can be used for reflecting the importance of the data are collected, the junk data identification model is used for self-adapting to diversified data, and the data features and the statistical features are used for identifying potential junk data, so that the problem of mistakenly deleting important data or omitting the junk data is avoided; and the problem of storing junk data in a cloud storage system is avoided from the source.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a method, device, equipment and storage medium for cleaning junk data. Background Art

[0002] Due to the rapid development of cloud-native applications, the amount of data in cloud storage systems has increased exponentially. With the continuous use of cloud storage systems, a large amount of junk data will be stored in the cloud storage systems, such as temporary files, duplicate data, invalid data, etc. If these junk data are not cleaned up in time, they will not only waste storage space, but also affect data access efficiency and security, and increase the cost of data backup and migration.

[0003] At present, static rules are basically used to clean up junk data in cloud storage systems. However, static rules are prone to mistakenly delete important data or miss junk data. The reasons for this problem are: static rules are fixed and cannot analyze the diverse data in cloud storage systems in real time, and cannot distinguish whether the data is important or not. In addition, static rules require a lot of computing resources to identify junk data, and the recognition complexity is high, which makes it easy to miss junk data during the recognition process. Summary of the invention

[0004] The present application provides a junk data cleaning method, apparatus, device and storage medium to solve the problem that using static rules to clean junk data in a cloud storage system can easily lead to accidental deletion of important data or omission of junk data.

[0005] In view of the above technical problems, the technical solution of this application is solved by the following embodiments:

[0006] An embodiment of the present application provides a method for cleaning up junk data, including: monitoring data generated by a cloud native application; determining statistical features corresponding to the data based on records corresponding to the data; extracting data features from the data based on the data type of the data; using a pre-trained junk data recognition model to identify junk data for the data based on the statistical features and the data features and outputting a recognition result; and when the recognition result indicates that the data is junk data, performing a junk data cleaning task on the data.

[0007] The statistical features include: the access frequency of the data and / or the user behavior characteristics corresponding to the data.

[0008] The extracting of data features from the data according to the data type of the data includes: preprocessing the data according to the data type of the data; acquiring a feature extraction model corresponding to the data type of the data; and extracting data features from the preprocessed data using the feature extraction model.

[0009] Among them, executing the junk data cleaning task for the data includes: displaying attribute data corresponding to the data in a preset display device; receiving data operation instructions issued for the data during the process of displaying the attribute data corresponding to the data; clearing the data when the data operation instruction is a clear operation instruction; archiving the data when the data operation instruction is an archive operation instruction.

[0010] The method further includes: receiving an identification error message issued for the data during the process of displaying attribute data corresponding to the data; wherein the identification error message indicates that the data is mistakenly identified as junk data by the junk data identification model; after receiving the identification error message, marking the data as junk data and storing the marked data as a user feedback sample in a cloud storage system; wherein the user feedback sample is used to retrain the junk data identification model.

[0011] The method further includes: in response to a preset model training trigger event, obtaining user feedback samples stored in a preset time period and obtaining incremental data generated by the cloud native application in the time period in a cloud storage system; wherein the end time of the time period is the time when the model training trigger event is detected; generating incremental data samples based on the acquired incremental data; and retraining the junk data recognition model using the user feedback samples and the incremental data samples.

[0012] Among them, the response to a preset model training trigger event includes: a model training trigger event that is triggered once every preset time length; or a model training trigger event that is triggered in response to a preset model retraining operation; or a model training trigger event that is triggered when the recognition accuracy of the junk data recognition model is less than a preset accuracy threshold.

[0013] An embodiment of the present application also provides a garbage data cleaning device, including: a monitoring module, used to monitor data generated by cloud native applications; a determination module, used to determine the statistical features corresponding to the data based on the records corresponding to the data; an extraction module, used to extract data features from the data based on the data type of the data; an identification module, used to use a pre-trained garbage data identification model to identify garbage data for the data based on the statistical features and the data features and output an identification result; and a cleaning module, used to perform a garbage data cleaning task on the data when the identification result indicates that the data is garbage data.

[0014] An embodiment of the present application also provides a garbage data cleaning device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: execute a garbage data cleaning program stored in the memory to implement any of the above-mentioned garbage data cleaning methods.

[0015] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed to implement any of the above-mentioned garbage data cleaning methods.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application can monitor the data generated by cloud native applications in the embodiment of the present application; determine the statistical features corresponding to the data according to the log corresponding to the data; extract data features in the data according to the data type of the data; use a pre-trained junk data recognition model to identify junk data for the data according to the statistical features and the data features and output the recognition result; when the recognition result indicates that the data is junk data, perform a junk data cleaning task on the data. In the process of monitoring data, the embodiment of the present application collects data features that can be used to reflect the data itself and statistical features that are used to reflect the importance of the data, so that the pre-trained junk data recognition model identifies whether the data is junk data according to the statistical features and data features of the data, uses the junk data recognition model to adapt to the diverse data and uses the data features and statistical features to identify potential junk data in the diverse data, avoiding the problem of accidentally deleting important data or missing junk data, and avoiding the problem of storing junk data in the cloud storage system from the source. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0020] Figure 1 is a flow chart of a method for cleaning junk data according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a junk data cleaning method according to an embodiment of the present application;

[0022] Figure 3 is a flow chart of the training steps of a junk data identification model according to an embodiment of the present application;

[0023] Figure 4 A flowchart of the incremental training steps of a junk data identification model according to an embodiment of the present application;

[0024] Figure 5 is a structural diagram of a junk data cleaning device according to an embodiment of the present application;

[0025] Figure 6 is a structural diagram of a garbage data cleaning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0028] The present application embodiment provides a method for cleaning junk data. Figure 1 , which is a flow chart of a garbage data cleaning method according to an embodiment of the present application.

[0029] Step S110, monitoring data generated by cloud native applications.

[0030] Cloud-native applications refer to applications deployed in the cloud environment. When users use cloud-native applications, cloud-native applications will generate data.

[0031] Step S120: determining statistical features corresponding to the data according to the records corresponding to the data.

[0032] Statistical features refer to the features used to describe the properties and usage of data.

[0033] The types of statistical features include, but are not limited to: the access frequency of the data and / or the user behavior characteristics corresponding to the data.

[0034] User behavior characteristics refer to the types and frequency of user-data interaction operations. Interaction operation types include but are not limited to: open, edit, move, and delete.

[0035] In the embodiments of the present application, statistical features can be used to reflect the importance of data. For example, a high frequency of data access can reflect that the data is more important to the user. Another example is that a high frequency of data interaction operations can reflect that the user needs to frequently operate the data, which means that the data is more important.

[0036] Step S130: extracting data features from the data according to the data type of the data.

[0037] Types of data, including but not limited to: images, audio, video, and text.

[0038] Data features refer to the features used to describe the data itself, such as the amount of data, the visual features, acoustic features, or text features of the data.

[0039] In the embodiment of the present application, when extracting data features from data, different extraction methods are used for different data types. Further, according to the data type of the data, the data is preprocessed; a feature extraction model corresponding to the data type of the data is obtained; and the data features are extracted from the preprocessed data using the feature extraction model.

[0040] In the embodiment of the present application, the data features can be used to reflect the key information of the data itself. The key information can be used to identify the relationship between data, for example, to identify whether two data are duplicate data.

[0041] Step S140: Use a pre-trained junk data identification model to identify junk data on the data according to the statistical features and the data features, and output an identification result.

[0042] The junk data identification model is a binary classification model, which is used to identify whether the data is junk data based on the statistical characteristics and data characteristics of the data and output the identification result. The junk data identification model is a pre-trained model.

[0043] The recognition result can be used to indicate whether the data is junk data or normal data. When the recognition result indicates that the data is junk data, it indicates that the data is junk data that can be cleaned up. When the recognition result indicates that the data is normal data, it indicates that the data is non-junk data.

[0044] In the embodiment of the present application, when the recognition result indicates that the data is normal data, the data can be directly archived in the cloud storage system (cloud native storage system). Archiving refers to storing the data in the cloud storage system.

[0045] In the embodiment of the present application, since the data is diverse and the importance of the data may change over time, the junk data recognition model can be incrementally trained during the use of the junk data recognition model so that the junk data recognition model always maintains a high recognition accuracy.

[0046] Step S150, when the identification result indicates that the data is junk data, a junk data cleaning task is performed on the data.

[0047] Junk data includes, for example, duplicate data, temporary files, invalid data, etc.

[0048] The junk data cleaning task is used to directly clean up the data or clean up or archive the data according to a data operation instruction.

[0049] Specifically, when the identification result indicates that the data is junk data, the attribute data corresponding to the data is displayed in a preset display device; in the process of displaying the attribute data corresponding to the data, a data operation instruction issued for the data is received; when the data operation instruction is a clear operation instruction, the data is cleared; when the data operation instruction is an archive operation instruction, the data is archived, that is, the data is archived to a cloud storage system.

[0050] Furthermore, attribute data includes but is not limited to: data source, data folder, data type (video, audio, text, log, backup data, etc.). Of course, attribute data can also include identification results, such as: whether the data is duplicate data, whether the data does not meet the preset compliance requirements.

[0051] Furthermore, the attribute data corresponding to the plurality of data can be displayed in the form of a list and data operation buttons can be set. The data operation buttons include a clear operation button and an archive operation button. The user can view the attribute data corresponding to each data in the list, and subjectively judge whether the data needs to be deleted from the cloud storage system. When it is determined that the data needs to be cleared, the user clicks the clear data button corresponding to the data, and when it is determined that the data needs to be archived, the user clicks the archive data button corresponding to the data. For each data in the list, when it is detected that the clear operation button is triggered, the data is cleared in the cloud storage system; when it is detected that the archive operation button is triggered, the data is archived in the cloud storage system.

[0052] Furthermore, when the recognition result indicates that the data is junk data, the attribute data corresponding to the data can also be input into a pre-trained instruction generation model, so that the instruction generation model can issue data operation instructions for the data based on the attribute data corresponding to the data, namely: clear operation instructions or archive operation instructions.

[0053] Furthermore, data that is incorrectly identified by the junk data recognition model can be collected based on user feedback. Specifically, in the process of displaying the attribute data corresponding to the data, an identification error message issued for the data can be received; wherein the identification error message indicates that the data is mistakenly identified as junk data by the junk data recognition model; after receiving the identification error message, the data is marked as junk data and the marked data is stored in the cloud storage system as a user feedback sample; wherein the user feedback sample is used to retrain the junk data recognition model. In the process of displaying the attribute data corresponding to the data, an identification error feedback button can be provided for the data; after viewing the attribute data, the user can subjectively judge whether the junk data recognition model has misidentified, and if so, click the identification error feedback button; after the identification error feedback button is clicked, an identification error message is issued. Retraining the junk data recognition model using user feedback samples can better improve the recognition accuracy and robustness of the junk data recognition model.

[0054] In an embodiment of the present application, the data generated by cloud native applications are monitored; based on the log corresponding to the data, the statistical features corresponding to the data are determined; based on the data type of the data, data features are extracted from the data; a pre-trained junk data recognition model is used to identify the data as junk data based on the statistical features and the data features and output the recognition result; when the recognition result indicates that the data is junk data, a junk data cleaning task is performed on the data. In the process of monitoring data, the embodiment of the present application collects data features that can be used to reflect the data itself and statistical features that are used to reflect the importance of the data, so that the pre-trained junk data recognition model identifies whether the data is junk data based on the statistical features and data features of the data, uses the junk data recognition model to adapt to diverse data and uses data features and statistical features to identify potential junk data in diverse data, thereby avoiding the problem of accidentally deleting important data or missing junk data, and avoiding the problem of storing junk data in the cloud storage system from the source.

[0055] In order to make the embodiments of the present application easier to understand, Figure 2 The schematic diagram of the junk data cleaning method according to an embodiment of the present application is shown to further describe the junk data cleaning method of the embodiment of the present application.

[0056] In the embodiment of the present application, various data generated in the cloud native application are monitored in real time. The data types of these data include but are not limited to: images, audio, video and text.

[0057] In an embodiment of the present application, in the process of real-time monitoring of each data, the statistical features corresponding to the data are determined based on the records corresponding to the data.

[0058] Specifically, the types of statistical features of data include, but are not limited to: the access frequency of the data and / or the user behavior characteristics corresponding to the data.

[0059] When determining the access frequency of data, it can be determined by counting the number of times the data is accessed and the time interval. Furthermore, log analysis can be performed to determine the access frequency of data within a preset time period (the number of accesses divided by the length of the preset time period). The access frequency of data can reveal the importance and activity of the data. Data that is not frequently accessed may be considered to be of low value, and machine learning models can use this information to assess whether the data may be junk data.

[0060] When determining the user behavior characteristics of data, it can be determined by analyzing the interaction between the user and the data, for example, the type of user interaction operation on the data (such as opening, editing, moving or deleting) and the frequency of interaction operation corresponding to each type of interaction operation. The interaction operation frequency is the ratio of the number of operations of this type of interaction operation performed on the data within a preset time period to the length of the preset time period. Among them, user behavior characteristics can provide important clues about data relevance and user preferences. For example, if a file is frequently accessed and edited by multiple users, then the file can be considered valuable. On the contrary, if a file is rarely accessed or never edited, then the file can be considered junk data.

[0061] In the embodiment of the present application, in the process of real-time monitoring of each data, in addition to determining the statistical features corresponding to the data, it is also necessary to extract the data features corresponding to the data.

[0062] Specifically, the data features of the data include, but are not limited to: the data volume, and the visual features, acoustic features, or text features extracted according to the data type of the data.

[0063] When extracting the size of the data volume, you can query the file or data item corresponding to the data, measure the data volume of the file, measure the number of bytes of the data item, and use the measurement result as the data volume size of the data. Furthermore, the cloud storage system usually provides an API, through which the storage bucket or metadata corresponding to the data is queried, and the data volume size of the data is obtained in the storage bucket or metadata. The data volume size is input into the junk data recognition model as one of the statistical features, which can assist the junk data recognition model in identifying junk data. For example, if certain types of junk data tend to have specific size patterns (such as files that are too small or too large), the junk data recognition model can learn these patterns to identify potential junk data.

[0064] When extracting visual features, acoustic features or text features of data, a feature extraction model corresponding to the data type is obtained according to the data type of the data, and the feature extraction model is used to extract data features from the data.

[0065] Furthermore, different data types correspond to the same or different feature extraction models.

[0066] Furthermore, before using different feature extraction models to extract data features from data of different data types, it is necessary to perform preprocessing on the data.

[0067] When the data type is an image type, the data is converted into a tensor. Of course, the data can also be subjected to image denoising, image scaling, etc. as needed.

[0068] When the data type of the data is an audio type, audio denoising is performed on the data.

[0069] When the data type of the data is a video type, denoising is performed on the data frame by frame.

[0070] When the data type is text type, the data is converted into a text sequence. For example, the data is segmented and stop word filtered to form a text sequence.

[0071] Furthermore, after the data is preprocessed, the data features are extracted from the preprocessed data using a feature extraction model corresponding to the data type. This is to select a suitable feature extraction model to extract data features from the data in a targeted manner according to the characteristics of different data types.

[0072] When the data type of the data is an image type, the feature extraction model is an image feature extraction model. The image feature extraction model is, for example, a convolutional neural network (CNN). The image feature extraction model can process the data into an image feature vector sequence.

[0073] When the data type of the data is a video type, the feature extraction model is a video feature extraction model. The video feature extraction model is, for example, a CNN network. The video feature extraction model can process the data into a sequence of spatiotemporal feature vectors.

[0074] When the data type of the data is an audio type, the feature extraction model is an audio feature extraction model. The audio feature extraction model is, for example, a CNN network. The audio feature extraction model can encode the data into an acoustic feature vector sequence.

[0075] When the data type of the data is a text type, the feature extraction model is a text feature extraction model. The text feature extraction model is, for example, a recurrent neural network (RNN). The text feature extraction model can encode the data into a text feature vector sequence.

[0076] In an embodiment of the present application, after obtaining the statistical characteristics and data characteristics of the data, a junk data recognition model can be used to identify the data as junk data and output the recognition result based on the statistical characteristics and data characteristics of the data. After obtaining the recognition result output by the junk data recognition model, if the recognition result indicates that the data is junk data, the attribute data corresponding to the data is displayed, and it is determined whether to specify the data to the cloud storage system according to the user operation; if the recognition result indicates that the data is normal data, the data is directly archived to the cloud storage system. After obtaining the recognition result output by the junk data recognition model, user feedback can also be received based on the recognition result, that is, receiving a recognition error message. If a recognition error message is received for the recognition result of a piece of data, it means that the data is normal data, but is misjudged as junk data. In this way, after receiving the recognition error message, the data corresponding to the recognition error message can be archived to the cloud storage system.

[0077] Furthermore, the junk data identification model is an adaptive deep learning model. The junk data identification model can dynamically adjust model parameters according to the data volume, access frequency, user behavior characteristics, visual characteristics, acoustic characteristics or text characteristics of the data to achieve accurate identification of junk data.

[0078] Furthermore, before using the junk data recognition model to recognize junk data based on the statistical characteristics and data characteristics of the data and outputting the recognition result, the junk data recognition model needs to be trained. Furthermore, the random gradient descent method can be used to train the junk data recognition model. Of course, other methods can also be used to train the junk data recognition model.

[0079] like Figure 3 , which is a flow chart of the training steps of a junk data identification model according to an embodiment of the present application.

[0080] Step S310: collect a large amount of original data from the cloud storage system, and label each original data to form a training sample corresponding to each original data.

[0081] The data types of raw data include but are not limited to: images, audio, video, and text.

[0082] After the original data is collected, it is classified, that is, junk data and normal data (non-junk data) are distinguished, and the classification results are used as the annotation information of the original data to facilitate the training of the junk data recognition model.

[0083] When classifying the original data, it can be achieved through manual labeling, rule matching, etc. Furthermore, the original data can be labeled as junk data or non-junk data using the preset duplicate data rules (duplicate data is usually junk data), source credibility rules (data sources are unreliable or data that has been tampered with is judged as junk data), and business rules (judging whether the data is junk data based on whether it complies with business rules or common sense).

[0084] For example, in a scenario where business rules are used to determine whether data is junk data, assume that the collected raw data is an order data set from an e-commerce platform, which contains fields such as order number, user ID, and order amount. Business rule: the order amount must be greater than 0. If the value in the order amount field is less than or equal to 0, the raw data may be an abnormal or erroneous data and can be marked as junk data according to the business rules.

[0085] For another example, in a scenario where common sense is used to determine whether data is junk data, assume that the collected raw data is a student age dataset of an online education platform, which includes a student age field. Common sense rule: students are usually between 5 and 30 years old. If the value in the student age field is not within the range of 5 to 30 years old, according to the common sense rule, the raw data may be an input error or abnormal data and can be marked as junk data.

[0086] In an embodiment of the present application, in a scenario with fewer training samples, data enhancement processing can be performed on the training samples, for example: rotating, flipping, scaling, etc. the original data, thereby increasing the number of training samples, increasing the diversity and quantity of the data, and improving the generalization ability and robustness of the junk data recognition model.

[0087] Step S320: Obtain a preset number of training samples from the labeled training samples.

[0088] Step S330, for each training sample, pre-process the training sample according to the data type of the training sample; and extract data features from the training sample using a feature extraction model corresponding to the data type of the training sample.

[0089] The purpose of preprocessing is to convert the data format of the training samples so that the training samples can adapt to the processing requirements of the junk data recognition model to reduce the overfitting of the junk data recognition model and the influence of noise data.

[0090] The specific preprocessing steps and data feature extraction steps can refer to the above description.

[0091] Step S340: for each training sample, determine the statistical features corresponding to the training sample.

[0092] The specific statistical feature extraction steps can refer to the above description.

[0093] Step S350: input the data features and statistical features respectively corresponding to the plurality of training samples into the junk data recognition model, and obtain the recognition result corresponding to each training sample output by the junk data recognition model.

[0094] The junk data identification model may be a deep learning model.

[0095] Step S360: determining the recognition loss value of the junk data recognition model according to the recognition results corresponding to the plurality of training samples.

[0096] Step S370, determine whether the identification loss value is less than a preset loss threshold; if yes, execute step S380; if not, execute step S390.

[0097] Step S380, determine whether the loss value of the junk data recognition model has been less than the loss threshold for a preset number of consecutive times; if so, determine that the training of the junk data recognition model is completed and end the training step; if not, execute step S390.

[0098] Step S390, adjust the parameters in the junk data identification model, and jump to step S320.

[0099] In the embodiment of the present application, accurate identification of junk data is achieved through adaptive deep learning technology. When training the deep learning model, labeled data is used for training, and the junk data identification model is optimized and adjusted to improve its performance and accuracy. During the training process, cross-validation, regularization, optimizer and other technologies can be used to avoid overfitting of the model and improve the generalization ability of the model.

[0100] In the embodiment of the present application, in order to ensure that the junk data recognition model always has a high recognition accuracy, after the junk data recognition model training is completed, the junk data recognition model can be incrementally trained during the use of the junk data recognition model. The embodiment of the present application can use the stochastic gradient descent method to perform incremental training on the junk data recognition model, and can also use other methods to perform incremental training on the junk data recognition model, for example: using an online learning algorithm based on meta-learning to train the junk data recognition model, so as to improve the learning efficiency and accuracy of the model.

[0101] like Figure 4 , which is a flow chart of the incremental training steps of a junk data identification model according to an embodiment of the present application.

[0102] Step S410, in response to a preset model training trigger event, in the cloud storage system, obtain user feedback samples stored in a preset time period and obtain incremental data generated by the cloud native application in the time period.

[0103] Model training trigger event, used to instruct to perform incremental training on the junk data identification model. Incremental training refers to retraining the junk data identification model.

[0104] Specifically, it can respond to a model training trigger event that is triggered once every preset time length; or, respond to a model training trigger event that is triggered by a preset model retraining operation; or, respond to a model training trigger event that is triggered when the recognition accuracy of the junk data recognition model is less than a preset accuracy threshold.

[0105] Furthermore, the end time of the time period is the time when the model training trigger event is detected. The duration of the preset time period is equal to the preset time length. This is equivalent to triggering a model training trigger event every preset time period.

[0106] Furthermore, the model retraining operation may be a click operation of a preset model retraining button. For example, when the user determines that incremental training of the junk data recognition model is required, the user clicks the model retraining button.

[0107] Furthermore, the recognition accuracy may be a recognition accuracy rate or a recognition accuracy number of the junk data recognition model within the preset time length. The recognition accuracy rate is equal to the ratio of the recognition accuracy number to the recognition times.

[0108] Step S420: Generate an incremental data sample based on the acquired incremental data.

[0109] After acquiring each incremental data, the type of each incremental data is distinguished, and according to the type of each incremental data, a label (junk data or normal data) is added to each incremental data to form an incremental data sample. The label indicates whether the incremental data is junk data or normal data. The specific distinction method can refer to the above description.

[0110] Step S430: Retrain the junk data recognition model using the user feedback samples and the incremental data samples.

[0111] The specific training process can refer to the above description.

[0112] In the embodiment of the present application, incremental learning technology is used to allow the junk data recognition model to be updated online without retraining from scratch, thereby improving efficiency and reducing computing resource consumption. Furthermore, the embodiment of the present application adopts a user feedback mechanism. Through user feedback on junk data results, data that is incorrectly recognized by the junk data recognition model can be collected. The junk data recognition model can use this incorrectly recognized data to perform self-correction and learning, thereby improving the recognition accuracy and robustness of the model.

[0113] In an embodiment of the present application, when the garbage data recognition model is trained normally, multimodal data is collected from the cloud storage system and classified and labeled. The data is preprocessed, and then features are extracted and the corresponding feature extraction model is selected. According to the amount of data, access frequency, user behavior characteristics and other data characteristics, the parameters of the garbage data recognition model are adjusted through adaptive deep learning. After the garbage data recognition model is trained, the garbage data recognition model is used to clean up the garbage data generated by the cloud native application. In the process of using the garbage data recognition model, the incremental learning method can be used to update the garbage data recognition model online to increase the recognition accuracy and robustness of the garbage data recognition model, and the resource consumption of model training can be reduced by incremental learning. Moreover, the data basis for online updates can be the misjudgment data of the garbage data recognition model fed back by users.

[0114] Through the embodiments of the present application, junk data can be cleaned up effectively and accurately, so that the cloud native storage space can be fully utilized, avoiding the waste of storage resources, saving costs, reducing unnecessary I / O bandwidth, CPU resources and memory resources, improving storage performance, reducing the latency of cloud native storage, and improving the stability and availability of the cloud storage system.

[0115] The present application also provides a junk data cleaning device. Figure 5 , which is a structural diagram of a garbage data cleaning device according to an embodiment of the present application.

[0116] The garbage data cleaning device comprises:

[0117] The monitoring module 510 is used to monitor the data generated by the cloud native application.

[0118] The determination module 520 is used to determine the statistical features corresponding to the data according to the records corresponding to the data.

[0119] The extraction module 530 is used to extract data features from the data according to the data type of the data.

[0120] The identification module 540 is used to use a pre-trained junk data identification model to perform junk data identification on the data according to the statistical features and the data features and output the identification result.

[0121] The cleaning module 550 is used to perform a junk data cleaning task on the data when the recognition result indicates that the data is junk data.

[0122] The functions of the device described in the embodiment of the present application have been described in the above method embodiment. Therefore, for any details not provided in the description of this embodiment, please refer to the relevant descriptions in the above embodiment and will not be repeated here.

[0123] The present application also provides a garbage data cleaning device, such as Figure 6 , which is a structural diagram of a garbage data cleaning device according to an embodiment of the present application.

[0124] The garbage data cleaning device includes: a processor 610, a communication interface 620, a memory 630 and a communication bus 640. The processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640.

[0125] The memory 630 is used to store computer programs.

[0126] In one embodiment of the present application, the processor 610, when executing the program stored on the memory 630, implements the garbage data cleaning method provided by any one of the aforementioned method embodiments, including: monitoring the data generated by the cloud native application; determining the statistical features corresponding to the data based on the records corresponding to the data; extracting data features from the data based on the data type of the data; using a pre-trained garbage data recognition model to identify garbage data for the data based on the statistical features and the data features and outputting the recognition results; when the recognition result indicates that the data is garbage data, performing a garbage data cleaning task on the data.

[0127] The statistical features include: the access frequency of the data and / or the user behavior characteristics corresponding to the data.

[0128] The extracting of data features from the data according to the data type of the data includes: preprocessing the data according to the data type of the data; acquiring a feature extraction model corresponding to the data type of the data; and extracting data features from the preprocessed data using the feature extraction model.

[0129] Among them, executing the junk data cleaning task for the data includes: displaying attribute data corresponding to the data in a preset display device; receiving data operation instructions issued for the data during the process of displaying the attribute data corresponding to the data; clearing the data when the data operation instruction is a clear operation instruction; archiving the data when the data operation instruction is an archive operation instruction.

[0130] The method further includes: receiving an identification error message issued for the data during the process of displaying attribute data corresponding to the data; wherein the identification error message indicates that the data is mistakenly identified as junk data by the junk data identification model; after receiving the identification error message, marking the data as junk data and storing the marked data as a user feedback sample in a cloud storage system; wherein the user feedback sample is used to retrain the junk data identification model.

[0131] The method further includes: in response to a preset model training trigger event, obtaining user feedback samples stored in a preset time period and obtaining incremental data generated by the cloud native application in the time period in a cloud storage system; wherein the end time of the time period is the time when the model training trigger event is detected; generating incremental data samples based on the acquired incremental data; and retraining the junk data recognition model using the user feedback samples and the incremental data samples.

[0132] Among them, the response to a preset model training trigger event includes: a model training trigger event that is triggered once every preset time length; or a model training trigger event that is triggered in response to a preset model retraining operation; or a model training trigger event that is triggered when the recognition accuracy of the junk data recognition model is less than a preset accuracy threshold.

[0133] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the garbage data cleaning method provided in any of the above method embodiments are implemented. Since the garbage data cleaning method has been described in detail above, for any details not fully described in the description of this embodiment, please refer to the relevant descriptions in the above embodiments, and no further description will be given here.

[0134] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0136] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0137] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for cleaning junk data, characterized in that: include: Monitor data generated by cloud-native applications; Determining statistical features corresponding to the data according to the records corresponding to the data; Extracting data features from the data according to the data type of the data; Using a pre-trained junk data identification model to identify junk data on the data according to the statistical features and the data features, and outputting an identification result; When the identification result indicates that the data is junk data, a junk data cleaning task is performed on the data.

2. The method according to claim 1, characterized in that The statistical features include: the access frequency of the data and / or the user behavior characteristics corresponding to the data.

3. The method according to claim 1, characterized in that The step of extracting data features from the data according to the data type of the data includes: Preprocessing the data according to the data type of the data; Acquire a feature extraction model corresponding to the data type of the data; The feature extraction model is used to extract data features from the preprocessed data.

4. The method according to claim 1, characterized in that: The performing of a garbage data cleaning task on the data includes: In a preset display device, displaying attribute data corresponding to the data; In the process of displaying the attribute data corresponding to the data, receiving a data operation instruction issued for the data; When the data operation instruction is a clear operation instruction, clear the data; In case the data operation instruction is an archiving operation instruction, the data is archived.

5. The method according to claim 4, characterized in that The method further comprises: In the process of displaying the attribute data corresponding to the data, receiving an identification error message issued for the data; wherein the identification error message indicates that the data is mistakenly identified as junk data by the junk data identification model; After receiving the recognition error message, the data is marked as junk data and the marked data is stored in a cloud storage system as a user feedback sample; wherein the user feedback sample is used to retrain the junk data recognition model.

6. The method according to claim 1, characterized in that The method further comprises: In response to a preset model training trigger event, in a cloud storage system, user feedback samples stored within a preset time period and incremental data generated by the cloud native application within the time period are obtained; wherein the end time of the time period is the time when the model training trigger event is detected; Generate an incremental data sample according to the acquired incremental data; The junk data recognition model is trained again using the user feedback samples and the incremental data samples.

7. The method according to claim 6, characterized in that The response to a preset model training trigger event includes: In response to a model training trigger event that is triggered every preset time length; or, In response to a model training trigger event triggered by a preset model retraining operation; or, A model training trigger event is triggered in response to the recognition accuracy of the junk data recognition model being less than a preset accuracy threshold.

8. A garbage data cleaning device, characterized in that: include: Monitoring module, used to monitor the data generated by cloud-native applications; A determination module, used to determine the statistical features corresponding to the data according to the records corresponding to the data; An extraction module, used for extracting data features from the data according to the data type of the data; an identification module, configured to use a pre-trained junk data identification model to identify junk data on the data according to the statistical features and the data features and output an identification result; The cleaning module is used to perform a junk data cleaning task on the data when the recognition result indicates that the data is junk data.

9. A garbage data cleaning device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: execute a garbage data cleaning program stored in the memory to implement the garbage data cleaning method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement the garbage data cleaning method according to any one of claims 1 to 7.

Images