Repeated data processing method and device and medium

Abstract

The invention provides a duplicated data processing method and device and a medium. The method is applied to a big data platform and comprises the steps that data information of a plurality of storage files in a storage system is received, the data information at least comprises a full-path file name and fingerprint data of each storage file, and the fingerprint data are file identifiers generated according to file contents of the storage files; according to a file path included in the full-path file name, aggregating the data information of the plurality of storage files to obtain a plurality of file groups; on the basis of the multiple pieces of fingerprint data included in each file group, path fingerprints corresponding to the file groups are generated, and the path fingerprints serve as file content identifiers corresponding to the file groups; by comparing the path fingerprints, determining the number of repetitions of file directories corresponding to the storage files in the file groups; and according to the number of repetition times of the file directory, generating a directory repetition report, and sending the directory repetition report to a storage system, so that the storage system deletes the repeated file directory according to the directory repetition report.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device and medium for processing duplicate data. Background Art

[0002] In order to enable the large model to accurately solve user problems, the large model will be trained before it is officially used. When training the large model, the storage system can provide the training data required for model training. When training the model, the files in the storage system will be reused, and the training set data will be directly copied during use. This results in a large amount of duplicate data in the training set in the storage system. As the model training time increases, the duplicate data in the training set will gradually increase. These duplicate data will occupy the system resources of the storage system, resulting in a waste of resources. Summary of the invention

[0003] In view of this, the present disclosure provides a method, device and medium for processing duplicate data, which can delete duplicate file directories in a file system and reduce the waste of system resources.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for processing duplicate data is provided, the method being applied to a big data platform and comprising:

[0005] Receive data information of multiple storage files in a storage system, the data information at least including: a full path file name and fingerprint data of each storage file, the fingerprint data being a file identifier generated according to the file content of the storage file;

[0006] aggregating the data information of the plurality of stored files according to the file paths included in the full path file name to obtain a plurality of file groups, wherein the file paths in each file group are the same;

[0007] Based on a plurality of fingerprint data included in each file group, a path fingerprint corresponding to the file group is generated, the path fingerprint being used as a file content identifier corresponding to the file group;

[0008] Determining the number of repetitions of a file directory corresponding to a storage file in the file group by comparing the path fingerprints, the file directory including the storage file in the file group;

[0009] A directory duplication report is generated according to the number of duplications of the file directory, and the directory duplication report is sent to the storage system, so that the storage system deletes the duplicate file directories according to the directory duplication report.

[0010] In one embodiment, receiving data information of a plurality of storage files in a storage system includes:

[0011] According to a preset data volume, the data information uploaded by the storage system is obtained in batches, and the full path file name in the data information is split into a file path and a file name.

[0012] In one embodiment, generating a path fingerprint corresponding to each file group based on multiple fingerprint data included in each file group includes:

[0013] Sort the fingerprint data in each file group in ascending or descending order to obtain a sorting result;

[0014] According to the sorting result, the fingerprint data of multiple storage files in each file group are connected to obtain a fingerprint string;

[0015] A hash value of the fingerprint string of each file group is calculated, and the hash value is determined as the path fingerprint.

[0016] In one embodiment, determining the number of repetitions of the file directory corresponding to the stored file in the file group by comparing the path fingerprint includes:

[0017] By comparing the path fingerprints, determining the number of repetitions of the path fingerprint;

[0018] The number of repetitions of the file directories corresponding to the plurality of storage files is determined according to the number of repetitions of the path fingerprint.

[0019] In one embodiment, the directory duplication report includes: the file directory, the number of duplications of the file directory, the number of files in the file directory, and the file names in the file directory.

[0020] According to a second aspect of an embodiment of the present disclosure, a device for processing duplicate data is provided, the device being applied to a big data platform, the device comprising:

[0021] A first receiving unit is used to receive data information of multiple storage files in the storage system, wherein the data information at least includes: a full path file name and fingerprint data of each storage file, wherein the fingerprint data is a file identifier generated according to the file content of the storage file;

[0022] an aggregation unit, configured to aggregate the data information of the plurality of stored files according to the file paths included in the full path file name, to obtain a plurality of file groups, wherein the file paths in each file group are the same;

[0023] A generating unit, configured to generate a path fingerprint corresponding to each file group based on a plurality of fingerprint data included in the file group, wherein the path fingerprint serves as a file content identifier corresponding to the file group;

[0024] a determination unit, configured to determine the number of repetitions of a file directory corresponding to a storage file in the file group by comparing a path fingerprint, the file directory including the storage file in the file group;

[0025] The first sending unit is used to generate a directory duplication report according to the number of duplications of the file directory, and send the directory duplication report to the storage system, so that the storage system deletes the duplicate file directory according to the directory duplication report.

[0026] In one embodiment, the first receiving unit is specifically configured to:

[0027] According to a preset data volume, the data information uploaded by the storage system is obtained in batches, and the full path file name in the data information is split into a file path and a file name.

[0028] In one embodiment, the generating unit is specifically used to:

[0029] Sort the fingerprint data in each file group in ascending or descending order to obtain a sorting result;

[0030] According to the sorting result, the fingerprint data of multiple storage files in each file group are connected to obtain a fingerprint string;

[0031] A hash value of the fingerprint string of each file group is calculated, and the hash value is determined as the path fingerprint.

[0032] In one embodiment, the determining unit is specifically configured to:

[0033] By comparing the path fingerprints, determining the number of repetitions of the path fingerprint;

[0034] The number of repetitions of the file directories corresponding to the plurality of storage files is determined according to the number of repetitions of the path fingerprint.

[0035] In one embodiment, the directory duplication report includes: the file directory, the number of duplications of the file directory, the number of files in the file directory, and the file names in the file directory.

[0036] According to a third aspect of an embodiment of the present disclosure, a data processing method is provided, the method being applied to a storage system, comprising:

[0037] Traversing the storage files in the storage system, and calculating fingerprint data according to the file content of each storage file, the fingerprint data being used as the file identifier of the storage file;

[0038] Obtaining data information of each storage file, the data information at least including: the full path file name of each storage file and the fingerprint data;

[0039] The data information of the multiple storage files is sent to a big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage files according to the data information.

[0040] In one embodiment, after sending the data information of the plurality of storage files to the big data platform, the method further includes:

[0041] Receiving a directory duplication report sent by the big data platform;

[0042] According to the directory duplication report, duplicate file directories and stored files in the duplicate file directories are deleted.

[0043] According to a fourth aspect of an embodiment of the present disclosure, a device for processing duplicate data is provided, the device being applied to a storage system, comprising:

[0044] A traversal unit, used for traversing the storage files in the storage system, and calculating fingerprint data according to the file content of each storage file, wherein the fingerprint data is used as the file identifier of the storage file;

[0045] An acquisition unit, used to obtain data information of each storage file, the data information at least including: a full path file name of each storage file and the fingerprint data;

[0046] The second sending unit is used to send the data information of multiple storage files to the big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage file according to the data information.

[0047] In one embodiment, the apparatus further comprises:

[0048] A second receiving unit, configured to receive a directory duplication report sent by the big data platform;

[0049] The deleting unit is used to delete the duplicate file directory and the stored files in the duplicate file directory according to the directory duplication report.

[0050] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first aspect or the third aspect are implemented.

[0051] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] By analyzing duplicate file directories, you can accurately determine the storage files in the storage system that have completely duplicate data. By deleting duplicate file directories, you can delete duplicate storage files without affecting the file structure of the storage system, freeing up storage system resources and reducing the waste of storage system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a method for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0054] Figure 2 is a flow chart of a method for generating a path fingerprint shown in an exemplary embodiment of the present disclosure;

[0055] Figure 3 is a flow chart showing a method of determining the number of repetitions of a file directory according to an exemplary embodiment of the present disclosure;

[0056] Figure 4 is a block diagram of a device for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0057] Figure 5 is a flow chart of another method for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0058] Figure 6 is a flowchart of deleting duplicate directories shown in an exemplary embodiment of the present disclosure;

[0059] Figure 7 is a block diagram of another apparatus for processing duplicate data shown in an exemplary embodiment of the present disclosure;

[0060] Figure 8 It is a hardware structure diagram of a big data platform or storage system shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0063] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0064] The storage system stores a large amount of training set data required for model training. When training a model, the training set data required for model training can be obtained from the storage system. The storage system is used by multiple people, and there are many directions for model training, so the training sets in the storage system are also diverse. There may be duplicate file data between different training sets, but the training set data is more important and cannot be deleted at will. Therefore, after a period of model training, the storage system will contain training set data for each model training. These data files are stored in the storage system overlapping with each other, which will lead to a lot of waste of storage system resources.

[0065] In addition, the amount of training set data in the storage system far exceeds the computing power of ordinary programs. It is difficult to identify duplicate data when running a stand-alone program on the storage system client, and it is impossible to accurately identify duplicate files stored in it on the client.

[0066] Based on this, the embodiments of the present disclosure provide a method for processing duplicate data, which can be applied to a big data platform. The computing power expansion of the big data platform, using the distributed computing and scalability of the big data platform, can solve the problem of massive computing volume caused by massive file systems calculating duplicate directories. Through the massive data computing power of the big data platform, the duplicate data files in the client storage system can be identified and analyzed to obtain the final analysis results of the duplicate directories. It should be noted that the client storage system can be used to store training data for large models, and can also be used for other data. The solution in the embodiments of the present disclosure does not need to limit the data type in the storage system. Regardless of the type of stored data, the embodiments of the present disclosure can identify whether the file directory of the stored files in the storage system is duplicated.

[0067] See also Figure 1 As shown in the flow chart, Figure 1 is a flowchart of a method for processing duplicate data shown in an exemplary embodiment of the present disclosure, and the flowchart includes:

[0068] S101, receiving data information of multiple storage files in a storage system.

[0069] The data information includes at least: the full path file name and fingerprint data of each stored file, and the fingerprint data is a file identifier generated according to the file content of the stored file. The full path file name and fingerprint data of the stored file can be collected and calculated by the storage system. It should be noted that if the fingerprint data of the stored files are the same, it means that the file data content in the stored files is the same.

[0070] In some embodiments, when the big data platform receives the data information of the storage file sent by the storage system, due to the large amount of data in the storage system, the big data platform may not be able to receive all the data information sent by the storage system at one time through a single read. If a single transmission is performed, the data information of the storage file may be lost due to the insufficient data reading capability of the big data platform.

[0071] Therefore, the big data platform can batch obtain the data information of the storage files uploaded by the storage system according to the preset data volume, and split the full path file name in the data information uploaded by the storage system into the file path and the file name, so that the storage files in the storage system can be clustered according to the file path in the future, reducing the amount of clustering calculation.

[0072] In one embodiment, the storage file can send the data information of the storage file to the big data platform in the form of a text file. There are two columns of data in the text file, one column is the full path file name of the storage file, and the other column is the fingerprint information of the storage file. The big data platform can read the data information in the text file in batches according to the preset data volume (for example, 100,000) each time, and store the read data information in the data table of the data warehouse. In the data table, the data information of the storage file can be split into three columns, one column for the file path, one column for the file name, and one column for the fingerprint data, and the subsequent identification and processing of duplicate files can be performed according to the data table.

[0073] S102, aggregating data information of the plurality of stored files according to the file paths included in the full-path file name to obtain a plurality of file groups, wherein the file paths in each file group are the same.

[0074] After receiving the data information of all storage files in the storage system, the big data platform aggregates the data information of the storage files according to the file paths in the full path file names of the storage files, and groups the multiple storage files according to the aggregation results so that the file paths of the storage files in each file group are the same. In one embodiment, when multiple storage files correspond to multiple file paths, multiple storage files can be clustered based on each file path to determine the storage files in each file path, for example, the first path / documents / reports / 2023 / quarter1 / may include three sub-files: report_01.txt, report_02.txt, and summary_01.txt. By aggregating the data information of the storage files, it can be found that there are multiple files with similar or identical content under the same path.

[0075] S103: Generate a path fingerprint corresponding to each file group based on the multiple fingerprint data included in the file group.

[0076] After the files are clustered, there will be data information of multiple stored files in each file group, that is, there are fingerprint data of multiple stored files in each file group. The fingerprint data of multiple stored files can be spliced ​​together to form a fingerprint string for each file group. By performing hash calculation on the fingerprint string, the path fingerprint of each file group can be obtained. The file content in each file group can be determined by the path fingerprint. If the path fingerprints of two file groups are the same, it means that the fingerprint data of the stored files in the two file groups are the same, that is, the content of the stored files in the two file groups is the same.

[0077] In some embodiments, the method of generating a path fingerprint can be found in Figure 2 The flowchart shown includes the following steps:

[0078] S201, sorting the fingerprint data in each file group in ascending or descending order to obtain a sorting result.

[0079] S202: According to the sorting result, fingerprint data of multiple stored files in each file group are connected to obtain a fingerprint string.

[0080] S203, calculating a hash value of the fingerprint string of each file group, and determining the hash value as a path fingerprint.

[0081] There may be fingerprint data of multiple stored files in the same file group. When concatenating the fingerprint data and calculating the path fingerprint based on the concatenated fingerprint string, the fingerprint data in each group needs to be sorted first. This is because the position of the fingerprint data will affect the final calculated path fingerprint value. For example, one fingerprint string is 123456, and the other fingerprint string is 654321. By observing the fingerprint data, it can be seen that the fingerprint data contained in the two groups are the same, but in different orders. If the fingerprint data is not sorted, the path fingerprints of the two groups will also be different. Then, when judging whether the file contents in the two file groups are the same based on the path fingerprint, it will be considered that the file contents in the two file groups are not the same, resulting in misjudgment.

[0082] Therefore, in one embodiment, for each file group, the fingerprint data of multiple stored files in the file group can be sorted first to obtain a sorting result, and the specific sorting method can be ascending or descending sorting. After the sorting is completed, the multiple fingerprint data in each file group are concatenated to obtain the fingerprint string of each file group. Finally, the fingerprint string of each file group is hashed, for example, the MD5 (Message Digest Algorithm 5) algorithm is used to calculate the MD5 value of each fingerprint string. Finally, the calculated hash value is determined as the path fingerprint of each file group.

[0083] S104, determining the number of repetitions of the file directories corresponding to the files stored in the file group by comparing the path fingerprints.

[0084] By comparing whether the path fingerprints are the same, it can be determined whether the file directories corresponding to the stored files in each file group are repeated. The number of times the path fingerprint is repeated is the number of times the file directory is repeated. A file directory (also called a folder or a file system directory) is a structure for organizing and managing files, similar to a container, that is, a file directory includes the stored files in the file group.

[0085] In one embodiment, the process of determining the number of repetitions of a file directory is shown in Figure 3 The flowchart shown includes:

[0086] S301, determining the number of repetitions of the path fingerprint by comparing the path fingerprint.

[0087] S302: Determine the number of repetitions of file directories corresponding to the plurality of storage files according to the number of repetitions of the path fingerprint.

[0088] First, compare the path fingerprint of each file group to determine whether the path fingerprint of a file group is the same as other identical path fingerprints, and determine the number of repetitions of the path fingerprint. Alternatively, use a clustering algorithm to aggregate the same path fingerprints and count the number of repetitions of each path after aggregation. After determining the number of repetitions of the path fingerprint, the number of repetitions of the path fingerprint can be determined as the number of repetitions of the file directory corresponding to the storage file. For example, if the file group is / home / user / docs / , and its corresponding fingerprint path is the same as the path fingerprint of the file group / home / user / Documents / Work / , then it means that the contents of the files stored in these two file paths are exactly the same, and these two file directories are duplicate file directories.

[0089] S105: Generate a directory duplication report according to the number of duplications of the file directory, and send the directory duplication report to the storage system, so that the storage system deletes duplicate file directories according to the directory duplication report.

[0090] After determining the number of times a file directory is repeated, a directory duplication report can be generated based on the number of times the file directory is repeated, and the directory duplication report can be sent to the storage system. The directory duplication report includes at least: the file directory, the number of times the file directory is repeated, the number of files in the file directory, and the file name in the file directory. Based on the file directory in the directory duplication report, the storage system can be informed of the file directories where the duplicate storage file data is located, as well as the number of times the file directory is repeated and the number and file name of each file directory. The storage system can determine the file directories with a large number of duplications based on the information in the directory duplication report, and prioritize deleting duplicate directories with a number of duplications greater than a preset threshold to free up more storage space. The number of files and the file name can be used to determine the content of the files stored in the file directory so that the storage system can retain the data corresponding to the preset file name to avoid accidental deletion of stored files.

[0091] It should be noted that deleting duplicate directories means finding and deleting directories with exactly the same structure and content in the file system. A directory may contain the same files, subfolders, or even the same directory structure. Deleting duplicate directories is to remove unnecessary directories, making the file system more concise and efficient. In the related art, after identifying duplicate data, the duplicate data content is usually deleted, and the file directory is not deleted and organized, which easily leads to the problem of complicated and redundant file systems.

[0092] Corresponding to the above-mentioned embodiment of the method for processing duplicate data, the present disclosure also provides an embodiment of an apparatus for processing duplicate data.

[0093] See also Figure 4 , Figure 4: is a block diagram of a duplicate data processing device provided by an exemplary embodiment of the present disclosure, wherein the device is applied to a big data platform and includes:

[0094] The first receiving unit 401 is used to receive data information of multiple storage files in the storage system, wherein the data information at least includes: a full path file name and fingerprint data of each storage file, wherein the fingerprint data is a file identifier generated according to the file content of the storage file;

[0095] An aggregation unit 402, configured to aggregate the data information of the plurality of stored files according to the file paths included in the full path file name, to obtain a plurality of file groups, wherein the file paths in each file group are the same;

[0096] A generating unit 403, configured to generate a path fingerprint corresponding to each file group based on a plurality of fingerprint data included in the file group, wherein the path fingerprint serves as a file content identifier corresponding to the file group;

[0097] A determination unit 404, configured to determine the number of repetitions of a file directory corresponding to a storage file in the file group by comparing the path fingerprint, the file directory including the storage file in the file group;

[0098] The first sending unit 405 is configured to generate a directory duplication report according to the number of duplications of the file directory, and send the directory duplication report to the storage system, so that the storage system deletes duplicate file directories according to the directory duplication report.

[0099] In one embodiment, the first receiving unit 401 is specifically configured to:

[0100] According to a preset data volume, the data information uploaded by the storage system is obtained in batches, and the full path file name in the data information is split into a file path and a file name.

[0101] In one embodiment, the generating unit 403 is specifically configured to:

[0102] Sort the fingerprint data in each file group in ascending or descending order to obtain a sorting result;

[0103] According to the sorting result, the fingerprint data of multiple storage files in each file group are connected to obtain a fingerprint string;

[0104] A hash value of the fingerprint string of each file group is calculated, and the hash value is determined as the path fingerprint.

[0105] In one embodiment, the determining unit 404 is specifically configured to:

[0106] By comparing the path fingerprints, determining the number of repetitions of the path fingerprint;

[0107] The number of repetitions of the file directories corresponding to the plurality of storage files is determined according to the number of repetitions of the path fingerprint.

[0108] In one embodiment, the directory duplication report includes: the file directory, the number of duplications of the file directory, the number of files in the file directory, and the file names in the file directory.

[0109] The present disclosure also provides a method for processing duplicate data, which can be applied to a storage system. Figure 5 As shown in the flow chart, Figure 5 FIG. 1 is a flowchart of another method for processing duplicate data according to an exemplary embodiment of the present disclosure, and the flowchart includes:

[0110] S501, traversing the storage files in the storage system, and calculating fingerprint data according to the file content of each storage file, the fingerprint data being used as the file identifier of the storage file.

[0111] The storage system may be a distributed storage client, in which all storage files in the storage system may be traversed through instruction information, and the MD5 fingerprint data of each storage file may be calculated using the MD5 algorithm, each fingerprint data may be used as a file identifier of the storage file, and the contents of the storage files with the same fingerprint data are also the same.

[0112] S502, obtaining data information of each storage file, the data information at least including: the full path file name of each storage file and the fingerprint data.

[0113] According to the full path file name and fingerprint data of each storage file, the data information of each storage file can be determined. In one embodiment, the data information of the storage file can be collected into a text file, and a separator is used in the text file to separate the full path file name and the fingerprint information.

[0114] S503, sending the data information of the multiple storage files to the big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage files according to the data information.

[0115] After the storage system collects the data information of all storage files, the data information of the storage files is sent to the big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage file according to the data information.

[0116] In one embodiment, after sending the data information of the plurality of storage files to the big data platform, the storage system may also receive a directory duplication report returned by the big data platform, and delete the file directory according to the directory duplication report. For the process of deleting duplicate directories, please refer to Figure 6 The flowchart shown includes:

[0117] S601, receiving a directory duplication report sent by the big data platform.

[0118] S602: Delete duplicate file directories and stored files in the duplicate file directories according to the directory duplication report.

[0119] The storage system first receives a directory duplication report sent by the big data platform, which includes: file directories, the number of repetitions of the file directory, the number of files in the file directory, and the file names in the file directory. Based on the directory duplication report, the storage system can determine which file directories contain duplicate storage file data, as well as the number of repetitions of the file directory and the number and file names of each file directory. The storage system can determine the file directories with more repetitions based on the information in the directory duplication report, and prioritize deleting duplicate directories with a number of repetitions greater than a preset threshold to free up more storage space. The number of files and file names can be used to determine the content of the files stored in the file directory, so that the storage system can retain the data corresponding to the preset file name to avoid accidental deletion of stored files.

[0120] Corresponding to the above-mentioned embodiment of the method for processing duplicate data, the present disclosure also provides an embodiment of a device for processing duplicate data. Figure 7 , Figure 7 : is a block diagram of another duplicate data processing device provided by an exemplary embodiment of the present disclosure, the device can be applied to a storage system, including:

[0121] A traversal unit 701 is used to traverse the storage files in the storage system and calculate fingerprint data according to the file content of each storage file, and the fingerprint data is used as the file identifier of the storage file;

[0122] The acquisition unit 702 is used to obtain data information of each storage file, wherein the data information at least includes: the full path file name of each storage file and the fingerprint data;

[0123] The second sending unit 703 is used to send the data information of multiple storage files to the big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage file according to the data information.

[0124] In one embodiment, the apparatus further comprises:

[0125] A second receiving unit, configured to receive a directory duplication report sent by the big data platform;

[0126] The deleting unit is used to delete the duplicate file directory and the stored files in the duplicate file directory according to the directory duplication report.

[0127] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0129] The device embodiment can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the big data platform or storage system where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 8 As shown, the repeated data processing device of the present disclosure is located, except Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, other hardware may also be included, which will not be described in detail.

[0130] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0131] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0132] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for processing duplicate data, characterized in that: The method is applied to a big data platform, and the method comprises: Receive data information of multiple storage files in a storage system, the data information at least including: a full path file name and fingerprint data of each storage file, the fingerprint data being a file identifier generated according to the file content of the storage file; aggregating the data information of the plurality of stored files according to the file paths included in the full path file name to obtain a plurality of file groups, wherein the file paths in each file group are the same; Based on a plurality of fingerprint data included in each file group, a path fingerprint corresponding to the file group is generated, the path fingerprint being used as a file content identifier corresponding to the file group; Determining the number of repetitions of a file directory corresponding to a storage file in the file group by comparing the path fingerprints, the file directory including the storage file in the file group; A directory duplication report is generated according to the number of duplications of the file directory, and the directory duplication report is sent to the storage system, so that the storage system deletes the duplicate file directories according to the directory duplication report.

2. The method according to claim 1, characterized in that The receiving data information of a plurality of storage files in the storage system includes: According to a preset data volume, the data information uploaded by the storage system is obtained in batches, and the full path file name in the data information is split into a file path and a file name.

3. The method according to claim 1, characterized in that The step of generating a path fingerprint corresponding to each file group based on a plurality of fingerprint data included in each file group includes: Sort the fingerprint data in each file group in ascending or descending order to obtain a sorting result; According to the sorting result, the fingerprint data of multiple storage files in each file group are connected to obtain a fingerprint string; A hash value of the fingerprint string of each file group is calculated, and the hash value is determined as the path fingerprint.

4. The method according to claim 1, characterized in that: The determining the number of repetitions of the file directory corresponding to the stored file in the file group by comparing the path fingerprints includes: By comparing the path fingerprints, determining the number of repetitions of the path fingerprints; The number of repetitions of the file directories corresponding to the plurality of storage files is determined according to the number of repetitions of the path fingerprint.

5. The method according to claim 1, characterized in that The directory duplication report includes: the file directory, the number of duplications of the file directory, the number of files in the file directory and the file names in the file directory.

6. A method for processing duplicate data, characterized in that: The method is applied to a storage system, and the method comprises: Traversing the storage files in the storage system, and calculating fingerprint data according to the file content of each storage file, the fingerprint data being used as the file identifier of the storage file; Obtaining data information of each storage file, the data information at least including: the full path file name of each storage file and the fingerprint data; The data information of the multiple storage files is sent to a big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage files according to the data information.

7. The method according to claim 6, characterized in that After sending the data information of the plurality of storage files to the big data platform, the method further includes: Receiving a directory duplication report sent by the big data platform; According to the directory duplication report, duplicate file directories and stored files in the duplicate file directories are deleted.

8. A device for processing duplicate data, characterized in that: The device is applied to a big data platform, and the device comprises: A first receiving unit is used to receive data information of multiple storage files in the storage system, wherein the data information at least includes: a full path file name and fingerprint data of each storage file, wherein the fingerprint data is a file identifier generated according to the file content of the storage file; an aggregation unit, configured to aggregate the data information of the plurality of stored files according to the file paths included in the full path file name, to obtain a plurality of file groups, wherein the file paths in each file group are the same; A generating unit, configured to generate a path fingerprint corresponding to each file group based on a plurality of fingerprint data included in the file group, wherein the path fingerprint serves as a file content identifier corresponding to the file group; a determination unit, configured to determine the number of repetitions of a file directory corresponding to a storage file in the file group by comparing a path fingerprint, the file directory including the storage file in the file group; The first sending unit is used to generate a directory duplication report according to the number of duplications of the file directory, and send the directory duplication report to the storage system, so that the storage system deletes the duplicate file directory according to the directory duplication report.

9. A device for processing duplicate data, characterized in that: The device is applied to a storage system, and comprises: A traversal unit, used for traversing the storage files in the storage system, and calculating fingerprint data according to the file content of each storage file, wherein the fingerprint data is used as the file identifier of the storage file; An acquisition unit, used to obtain data information of each storage file, the data information at least including: a full path file name of each storage file and the fingerprint data; The second sending unit is used to send the data information of multiple storage files to the big data platform, so that the big data platform determines the number of repetitions of the file directory corresponding to the storage file according to the data information.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images