Mass file data optimization storage method

Abstract

The invention relates to the technical field of mass file data optimized storage, in particular to a mass file data optimized storage method, which comprises the following steps of: acquiring redundant data blocks with various sizes in each file based on the difference between any two data blocks in each file and the difference between any two data blocks between different files; determining data redundancy based on the distribution condition of all redundant data blocks in each file; and based on the proportion of the redundant data blocks of the same size among different files and the data redundancy, in combination with the proportion of the remaining data blocks except the redundant data blocks in each file, determining the deduplication degree, and in combination with the access frequency of each type of to-be-stored files, storing all types of to-be-stored files. According to the method, the storage efficiency of mass file data is improved by analyzing the influence of redundant data on the storage priority.

Description

Technical Field

[0001] The present application relates to the technical field of optimized storage of massive file data, and in particular to a method for optimized storage of massive file data. Background Art

[0002] When faced with massive amounts of file data, optimizing storage is a very important task, especially in scenarios such as big data, cloud computing, and high concurrent access. An effective storage optimization method can help enterprises reduce storage costs, increase data access speed, and optimize the overall performance of the system. Determining data storage priority is a key step in optimizing data storage, improving access efficiency, and reducing costs. Therefore, it is necessary to classify and grade data, and set different storage strategies based on factors such as data redundancy characteristics.

[0003] At present, the storage strategy for massive data generally divides different priorities based on the functional characteristics of the data and stores them in order of priority. However, due to the large scale of data, the redundancy of the data is relatively large. The existence of redundant data will affect the division of storage priorities, which in turn will cause errors in the priority division strategy based on the functional characteristics of the data. At the same time, the existence of redundant data will increase storage costs and reduce the storage efficiency of massive file data. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a method for optimizing storage of massive file data to solve the existing problems.

[0005] A method for optimizing storage of massive file data in this application adopts the following technical solution:

[0006] An embodiment of the present application provides a method for optimizing storage of massive file data, the method comprising the following steps:

[0007] Obtain all types of files to be stored and their respective access frequencies, and allocate multiple data blocks for each file in each type of data to be stored for storage;

[0008] In various types of files to be stored, redundant data blocks of various sizes are obtained based on the difference between any two data blocks in each file and the difference between any two data blocks between each file and other files; based on the distribution of all redundant data blocks in each file, the data redundancy of each file is determined;

[0009] Based on the proportion of redundant data blocks of the same size between each file and the remaining files, determine the redundancy index between each file in each type of files to be stored and the remaining files, and determine the redundancy similarity between each file in each type of files to be stored and the remaining files in combination with the data redundancy, so as to determine the redundancy coefficient of each type of files to be stored; based on the proportion of remaining data blocks in each file except the redundant data blocks, and the redundancy coefficient, determine the deduplication degree of each file in each type of files to be stored;

[0010] Based on the access frequency of each type of files to be stored and the degree of deduplication, the priority storage coefficient of each file in each type of files to be stored is determined, and combined with the number of all data blocks in each type of files to be stored, the storage priority of each type of files to be stored is determined, and all types of files to be stored are stored.

[0011] Preferably, the method for obtaining redundant data blocks of various sizes in each file is:

[0012] The data in each data block is in the form of a binary number. In all files of various types to be stored, a bitwise XOR operation is performed on the binary numbers between any two data blocks of the same size, and any data block of the two data blocks in which all bits of the binary result of the operation are 0 is marked as a redundant data block. All data blocks are traversed to obtain all redundant data blocks in various types of files to be stored, wherein the marked redundant data blocks are not marked repeatedly, and redundant data blocks of various sizes in each file are obtained by statistics.

[0013] Preferably, the data redundancy of each file is the proportion of all redundant data blocks in all data blocks in each file.

[0014] Preferably, the method for determining the redundancy index between each file in each type of files to be stored and the other files is:

[0015] In each type of files to be stored, the ratio of the number of types of redundant data blocks of the same size between each file and the other files to the total number of types of redundant data blocks of all different sizes is used as the redundancy index between each file and the other files in each type of files to be stored.

[0016] Preferably, the expression of the redundant similarity between each file in each type of files to be stored and the other files is: In the formula, represents the redundant similarity between file i and file j in the kth category of files to be stored; represents the redundancy index between file i and file j in the kth type of files to be stored; β i , β jThey represent the data redundancy of file i and file j respectively; norm() represents the normalization function; ε represents a constant preset to be greater than 0.

[0017] Preferably, the method for determining the redundancy coefficient of each type of file to be stored is:

[0018] The mean value of the redundant similarity between each file in each type of files to be stored and all other files is calculated, and the average value of the means of the redundant similarity of all files is used as the redundant coefficient of each type of files to be stored.

[0019] Preferably, the method for determining the degree of deduplication of each file in the various types of files to be stored is:

[0020] Calculate the proportion of the remaining data blocks in all data blocks in each file of each type of files to be stored, excluding the redundant data blocks, and record it as the non-duplicate proportion of each file in each type of files to be stored;

[0021] The degree of deduplication of each file in each type of files to be stored is the ratio of the redundancy coefficient of each type of files to be stored to the non-duplicate proportion of each file in the corresponding type of files to be stored.

[0022] Preferably, the expression of the priority storage coefficient of each file in the various types of files to be stored is: In the formula, γ k,i represents the priority storage coefficient of file i in the kth category of files to be stored; β k represents the access frequency of the k-th type of files to be stored; ω k,i represents the degree of deduplication of file i in the kth category of files to be stored; norm() represents the normalization function; σ represents a constant preset to be greater than 0.

[0023] Preferably, the expression of the storage priority of each type of files to be stored is: Where U k Indicates the storage priority of the kth type of files to be stored; represents the mean value of the priority storage coefficients of all files in the kth category of files to be stored; C k Indicates the number of all data blocks in the k-th type of files to be stored.

[0024] Preferably, storing all types of files to be stored includes:

[0025] Sort the various files to be stored in the storage device in descending order of storage priority.

[0026] This application has at least the following beneficial effects:

[0027] The present application obtains redundant data blocks in each file by analyzing the differences between data blocks, and constructs data redundancy based on the proportion of all redundant data blocks in each file. The beneficial effect is that by calculating data redundancy, the repeated parts in the data can be identified, thereby removing redundant data and improving data storage efficiency; the present application constructs redundant similarity by analyzing the redundancy between different files, and the beneficial effect is that the redundancy between data can be more accurately evaluated, thereby formulating a more reasonable storage strategy, ensuring that key data is stored first and improving data storage efficiency; the present application constructs the degree of deduplication based on redundant similarity and the proportion of remaining data blocks outside redundant data blocks, and the beneficial effect is that it helps to optimize resource allocation and improve data storage efficiency; the present application constructs storage priority by comprehensively considering access frequency and deduplication degree, as well as the number of all data blocks in various types of files to be stored, and stores all types of files to be stored, and the beneficial effect is that the influence of redundant data on storage priority can be eliminated, thereby improving data storage efficiency. The present application improves the storage efficiency of massive file data by analyzing the influence of redundant data on storage priority. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flowchart of a method for optimizing storage of massive file data provided by an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a deduplication degree extraction process provided by an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a storage priority acquisition process provided for an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of a method for optimizing storage of massive file data proposed in the present application, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0033] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0034] The following is a detailed description of a method for optimizing storage of massive file data provided by the present application in conjunction with the accompanying drawings.

[0035] An embodiment of the present application provides a method for optimizing storage of massive file data. Specifically, the following method for optimizing storage of massive file data is provided. Figure 1 , the method comprises the following steps:

[0036] Step S1: Obtain all types of files to be stored and their respective access frequencies, and allocate multiple data blocks for each file in each type of data to be stored for storage.

[0037] With the popularization and application of technologies such as the Internet, the Internet of Things, and social media, the generation of massive file data is growing exponentially. Massive file data includes various types such as text files, images, videos, and audio. At present, different types of data are generally stored in storage devices in the form of binary strings, but large storage systems generally distinguish storage space, and the smallest unit of distinction is the data block. Therefore, in order to facilitate the storage of large-scale data, it is necessary to allocate data blocks for large-scale data.

[0038] Therefore, all types of files to be stored and the access frequencies of each type of files to be stored are obtained, and multiple data blocks are allocated for each type of files to be stored for storage. The data in the data blocks are in the form of binary numbers.

[0039] It should be noted that all types of files to be stored refer to files of different formats. In this embodiment, all types of files to be stored include text files, image files, video files and audio files, wherein the access frequency of each type of file to be stored refers to the access frequency of the entire type of files to be stored. The process of obtaining the access frequency is a well-known technology, and the specific acquisition process will not be repeated here.

[0040] Step S2: In each type of files to be stored, based on the difference between any two data blocks in each file and the difference between any two data blocks between each file and the remaining files, obtain redundant data blocks of various sizes in each file; based on the distribution of all redundant data blocks in each file, determine the data redundancy of each file.

[0041] When storing various types of files to be stored, the storage priority of the corresponding type of files is generally determined by analyzing the functional characteristics of the data contained in the files, such as access frequency, data importance, etc. For large-scale data with redundant characteristics, if the storage priority is determined only based on the functional characteristics of the data, the optimal storage efficiency will not be achieved.

[0042] Therefore, by analyzing the redundant characteristics of large-scale data, removing redundant data from the data, and eliminating the impact of redundant data on data storage priority, data storage efficiency can be improved. Specifically:

[0043] (1) In each type of file to be stored, based on the difference between any two data blocks in each file and the difference between any two data blocks between each file and the other files, redundant data blocks of various sizes in each file are obtained.

[0044] Internal redundancy is a common phenomenon in data storage. It is not only related to the characteristics of the file format, but also involves data backup, data integration and other aspects. For example, in a text file, some text content may be copied and pasted to different places; a video file may contain multiple identical frames; in an audio file, in order to support playback at different bit rates, multiple audio streams of the same content may be stored; these phenomena are all cases of internal data redundancy. Redundant data occupies additional storage space. Since storage resources are occupied by unnecessary data, the priority strategy based on functional characteristics may not work as expected. In addition, since users may save multiple copies of the same data in different locations, or when sharing data between different business departments or applications, duplicate data copies will be generated, resulting in external data redundancy. The existence of external redundancy also has a certain impact on storage priority.

[0045] Therefore, in order to filter out redundant data in the file and eliminate the interference of redundant data on the storage priority, the differences between any two data blocks in each file of each type of file to be stored, as well as the differences between any two data blocks between each file and the other files, are analyzed to obtain redundant data blocks of various sizes in each file, specifically:

[0046] Since the data in each data block is in the form of binary numbers, in all files of various types of files to be stored, a bitwise XOR operation is performed on the binary numbers between any two data blocks of the same size, and any data block of the two data blocks in which all bits of the binary result of the operation are 0 is marked as a redundant data block. All data blocks are traversed to obtain all redundant data blocks in various types of files to be stored, wherein the marked redundant data blocks are not marked repeatedly, and redundant data blocks of various sizes in each file are obtained by statistics.

[0047] It should be noted that since the sizes of data blocks are different, the lengths of the binary numbers stored in them are also different. Data blocks of different sizes are definitely not redundant data blocks. The redundant data blocks here refer to data blocks whose binary numbers stored in the data blocks must also have the same length, otherwise the bitwise XOR operation cannot be performed.

[0048] Among them, the calculation steps of the bitwise XOR operation are well-known technologies, and its specific calculation process will not be repeated here.

[0049] (2) Based on the distribution of all redundant data blocks in each file, determine the data redundancy of each file.

[0050] If there are more redundant data blocks in a file, it means that the data redundancy of the file is greater, and the redundant data in the file needs to be removed. Therefore, by analyzing the proportion of redundant data blocks in the file, the data redundancy is determined, and the redundancy of the file is judged. Specifically:

[0051] The data redundancy of each file is the ratio of all redundant data blocks in all data blocks in each file.

[0052] According to the data redundancy of each file, it can be understood that the more redundant data blocks there are in a file, the greater the proportion of all redundant data blocks in the file to all data blocks, that is, the greater the data redundancy of the file; conversely, the fewer redundant data blocks there are in a file, the smaller the proportion of all redundant data blocks in the file to all data blocks, that is, the smaller the data redundancy of the file.

[0053] Step S3: Based on the proportion of redundant data blocks of the same size between each file and the remaining files, determine the redundancy index between each file in each type of files to be stored and the remaining files, and determine the redundant similarity between each file in each type of files to be stored and the remaining files in combination with the data redundancy, so as to determine the redundancy coefficient of each type of files to be stored; based on the proportion of remaining data blocks in each file except redundant data blocks, and the redundancy coefficient, determine the deduplication degree of each file in each type of files to be stored.

[0054] During the data migration process, data copies of the old system may be temporarily or permanently retained to ensure a smooth transition of the migration, resulting in external data redundancy. When data is shared between different applications, duplicate data copies may be generated, which also leads to external data redundancy. For large-scale data with redundant characteristics, if the storage order is determined only based on functional characteristics, the redundant data may be mistakenly given a higher storage priority, resulting in unreasonable allocation of storage resources. Therefore, determining the priority storage order based only on the functional characteristics of the data will not achieve the optimal storage efficiency.

[0055] Therefore, by analyzing the redundancy similarity between files and eliminating the interference of redundant data between files, the storage efficiency of massive file data can be improved. Specifically:

[0056] (1) Based on the proportion of redundant data blocks of the same size between each file and the other files, determine the redundancy index between each file and the other files in each type of files to be stored, which is used to characterize the redundancy degree between the files, specifically:

[0057] In each type of files to be stored, the ratio of the number of types of redundant data blocks of the same size between each file and the other files to the total number of types of redundant data blocks of all different sizes is used as the redundancy index between each file and the other files in each type of files to be stored.

[0058] According to the redundancy index between each file and the other files in each type of files to be stored, it can be understood that if the types of redundant data blocks of the same size between files are more, the redundancy index is larger, indicating that the redundancy degree between files is greater; conversely, if the types of redundant data blocks of the same size between files are fewer, the redundancy index is smaller, indicating that the redundancy degree between files is smaller.

[0059] (2) Further, based on the proportion of redundant data blocks of the same size between each file and the other files, and in combination with the data redundancy, the redundancy similarity between each file in each type of files to be stored and the other files is determined, specifically:

[0060] The redundancy similarity between file i and file j in the kth type of files to be stored The expression is: In the formula, represents the redundant similarity between file i and file j in the kth category of files to be stored; represents the redundancy index between file i and file j in the kth type of files to be stored; β i , β j They represent the data redundancy of file i and file j respectively; norm() represents a normalization function; ε represents a constant preset to be greater than 0, which is used to prevent the denominator from being 0, wherein the value of ε is artificially set. In this embodiment, the value of ε is 0.01. Under the premise of ensuring that the denominator is not 0 and does not excessively affect the calculation results, the implementer can set it according to the specific situation. This embodiment does not impose any special restrictions.

[0061] According to the redundancy similarity between file i and file j in the kth category of files to be stored It can be understood that the larger the ratio of the number of types of redundant data blocks of the same size between file i and file j in the kth type of files to be stored to the total number of types of all different sizes in file i and file j, that is, The larger the value is, the more redundant data blocks of the same size are in file i and file j, and the greater the redundancy similarity between file i and file j is. The smaller the difference in data redundancy between file i and file j is, the greater the redundancy similarity is. Conversely, the smaller the ratio of the number of types of redundant data blocks of the same size between file i and file j in the kth type of files to be stored to the total number of types of all different sizes in file i and file j is, that is, The smaller it is, the fewer redundant data blocks of the same size file i and file j have, and the smaller the redundant similarity between file i and file j is. The greater the difference in data redundancy between file i and file j is, the smaller the redundant similarity is.

[0062] (3) Further, based on the redundancy similarity between each file in each type of files to be stored and the remaining files, the redundancy coefficient of each type of files to be stored is determined, specifically:

[0063] The mean value of the redundant similarity between each file in each type of files to be stored and all other files is calculated, and the average value of the means of the redundant similarity of all files is used as the redundant coefficient of each type of files to be stored.

[0064] According to the redundancy coefficients of various types of files to be stored, it can be understood that if the redundant similarity between files in the files to be stored is greater, that is, the mean value of the redundant similarity between all files is greater, then the redundancy coefficient of the files to be stored is greater, and the storage priority level of the data is smaller, and the file deduplication operation is more necessary; conversely, if the redundant similarity between files in the files to be stored is smaller, that is, the mean value of the redundant similarity between all files is smaller, then the redundancy coefficient of the files to be stored is smaller, and the storage priority level of the data is greater.

[0065] (4) Based on the redundancy coefficient of each type of files to be stored and the proportion of the remaining data blocks in each file except the redundant data blocks, the degree of deduplication of each file in each type of files to be stored is determined, specifically:

[0066] Calculate the proportion of the remaining data blocks in all data blocks in each file of each type of files to be stored, excluding the redundant data blocks, and record it as the non-duplicate proportion of each file in each type of files to be stored;

[0067] The degree of deduplication of each file in each type of files to be stored is the ratio of the redundancy coefficient of each type of files to be stored to the non-duplicate proportion of each file in the corresponding type of files to be stored.

[0068] According to the deduplication degree of each file in each type of files to be stored, it can be understood that the more redundant data blocks in a file, the lower its storage priority may be, and the more it needs to deduplicate the file, that is, the smaller the non-duplicate proportion of the file and the larger the redundancy coefficient, the greater the deduplication degree and the lower the storage priority of the file data; conversely, the fewer redundant data blocks in a file, the higher its storage priority may be, that is, the larger the non-duplicate proportion of the file and the smaller the redundancy coefficient, the smaller the deduplication degree and the greater the storage priority of the file data.

[0069] Preferably, the schematic diagram of the deduplication degree extraction process provided in this embodiment is as follows Figure 2 shown.

[0070] Step S4: Based on the access frequency of each type of files to be stored and the degree of deduplication, determine the priority storage coefficient of each file in each type of files to be stored, and combine the number of all data blocks in each type of files to be stored to determine the storage priority of each type of files to be stored, and store all types of files to be stored.

[0071] If the degree of deduplication of a file is greater, it means that there is more redundant data in it, and a lower storage priority can be set for it. If the data file has a low repetition rate and a low degree of deduplication, it means that most of the data is unique and its data storage priority needs to be increased.

[0072] Therefore, based on the access frequency of each type of files to be stored and the degree of deduplication, the priority storage coefficient of each file in each type of files to be stored is determined, and combined with the number of all data blocks remaining except redundant data blocks in each type of files to be stored, the storage priority of each type of files to be stored is determined, and all types of files to be stored are stored, specifically:

[0073] (1) Based on the access frequency of each type of files to be stored and the degree of deduplication, the priority storage coefficient of each file in each type of files to be stored is determined, specifically:

[0074] The priority storage coefficient γ of file i in the kth type of files to be stored k,i The expression is: In the formula, β k represents the access frequency of the k-th type of files to be stored; ω k,i It represents the degree of deduplication of file i in the kth category of files to be stored; norm() represents the normalization function; σ represents a constant preset to be greater than 0, which is used to prevent the denominator from being 0, wherein the value of σ is artificially set. In this embodiment, the value of σ is 0.01. Under the premise of ensuring that the denominator is not 0 and does not excessively affect the calculation results, the implementer can set it according to the specific situation. This embodiment does not impose any special restrictions.

[0075] According to the priority storage coefficient of each file in each type of files to be stored, it can be understood that the more redundant data blocks there are in a file, the greater the degree of deduplication of the file. If the access frequency is smaller at this time, the priority storage coefficient is lower; conversely, the fewer redundant data blocks there are in a file, the smaller the degree of deduplication of the file. If the access frequency is larger at this time, the priority storage coefficient is higher.

[0076] (2) Further, based on the priority storage coefficient of each file in each type of files to be stored and in combination with the number of all remaining data blocks in each type of files to be stored except for redundant data blocks, the storage priority of each type of files to be stored is determined, specifically:

[0077] The storage priority U of the kth type of files to be stored k The expression is: In the formula, represents the mean value of the priority storage coefficients of all files in the kth category of files to be stored; C k Indicates the number of all data blocks in the k-th type of files to be stored.

[0078] The storage device will allocate at least one data block for each file, so the denominator cannot be 0.

[0079] According to the storage priority of various types of files to be stored, it can be understood that the larger the priority storage coefficient of all files in the same type of files to be stored, the smaller the number of data blocks allocated in this type of files to be stored, indicating that the average storage priority of the files and the smaller the data size, the greater the corresponding storage priority, then the higher the storage priority of this type of files to be stored, and the sooner this type of files to be stored will be stored; conversely, the smaller the priority storage coefficient of all files in the same type of files to be stored, the larger the number of data blocks allocated in this type of files to be stored, and the lower the storage priority of this type of files to be stored.

[0080] Preferably, the schematic diagram of the storage priority acquisition process provided in this embodiment is as follows: Figure 3 shown.

[0081] (3) Further, the various types of files to be stored are stored based on their storage priority, specifically: the various types of files to be stored are sorted in descending order of storage priority and stored in the storage device.

[0082] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0084] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for optimizing storage of massive file data, characterized in that: The method comprises the following steps: Obtain all types of files to be stored and their respective access frequencies, and allocate multiple data blocks for each file in each type of data to be stored for storage; In various types of files to be stored, based on the difference between any two data blocks in each file and the difference between any two data blocks between each file and other files, redundant data blocks of various sizes in each file are obtained; based on the distribution of all redundant data blocks in each file, the data redundancy of each file is determined; Based on the proportion of redundant data blocks of the same size between each file and the remaining files, determine the redundancy index between each file in each type of files to be stored and the remaining files, and determine the redundancy similarity between each file in each type of files to be stored and the remaining files in combination with the data redundancy, so as to determine the redundancy coefficient of each type of files to be stored; based on the proportion of remaining data blocks in each file except the redundant data blocks, and the redundancy coefficient, determine the deduplication degree of each file in each type of files to be stored; Based on the access frequency of each type of files to be stored and the degree of deduplication, the priority storage coefficient of each file in each type of files to be stored is determined, and combined with the number of all data blocks in each type of files to be stored, the storage priority of each type of files to be stored is determined, and all types of files to be stored are stored.

2. The method for optimizing storage of massive file data according to claim 1, characterized in that: The method for obtaining redundant data blocks of various sizes in each file is as follows: The data in each data block is in the form of a binary number. In all files of various types to be stored, a bitwise XOR operation is performed on the binary numbers between any two data blocks of the same size, and any data block of the two data blocks in which all bits of the binary result of the operation are 0 is marked as a redundant data block. All data blocks are traversed to obtain all redundant data blocks in various types of files to be stored, wherein the marked redundant data blocks are not marked repeatedly, and redundant data blocks of various sizes in each file are obtained by statistics.

3. The method for optimizing storage of massive file data according to claim 1, characterized in that: The data redundancy of each file is the proportion of all redundant data blocks in all data blocks in each file.

4. The method for optimizing storage of massive file data according to claim 1, characterized in that: The method for determining the redundancy index between each file in each type of files to be stored and the other files is as follows: In each type of files to be stored, the ratio of the number of types of redundant data blocks of the same size between each file and the other files to the total number of types of redundant data blocks of all different sizes is used as the redundancy index between each file and the other files in each type of files to be stored.

5. The method for optimizing storage of massive file data according to claim 1, characterized in that: The expression of the redundant similarity between each file in the various types of files to be stored and the other files is: In the formula, represents the redundant similarity between file i and file j in the kth category of files to be stored; represents the redundancy index between file i and file j in the kth type of files to be stored; β i , β j They represent the data redundancy of file i and file j respectively; norm() represents the normalization function; ε represents a constant preset to be greater than 0.

6. The method for optimizing storage of massive file data according to claim 1, characterized in that: The method for determining the redundancy coefficient of each type of files to be stored is: The mean value of the redundant similarity between each file in each type of files to be stored and all other files is calculated, and the average value of the means of the redundant similarity of all files is used as the redundant coefficient of each type of files to be stored.

7. The method for optimizing storage of mass file data according to claim 1, characterized in that: The method for determining the deduplication degree of each file in the various types of files to be stored is: Calculate the proportion of the remaining data blocks in all data blocks in each file of each type of files to be stored, excluding the redundant data blocks, and record it as the non-duplicate proportion of each file in each type of files to be stored; The degree of deduplication of each file in each type of files to be stored is the ratio of the redundancy coefficient of each type of files to be stored to the non-duplicate proportion of each file in the corresponding type of files to be stored.

8. The method for optimizing storage of massive file data according to claim 1, characterized in that: The expression of the priority storage coefficient of each file in the various types of files to be stored is: In the formula, γ k,i represents the priority storage coefficient of file i in the kth category of files to be stored; β k represents the access frequency of the kth type of files to be stored; ω k,i represents the degree of deduplication of file i in the kth category of files to be stored; norm() represents the normalization function; σ represents a constant preset to be greater than 0.

9. The method for optimizing storage of mass file data according to claim 1, characterized in that: The expression of the storage priority of each type of files to be stored is: Where U k Indicates the storage priority of the kth type of files to be stored; represents the mean value of the priority storage coefficients of all files in the kth category of files to be stored; C k Indicates the number of all data blocks in the k-th type of files to be stored.

10. The method for optimizing storage of mass file data according to claim 1, characterized in that: The storing of all types of files to be stored includes: Sort the various files to be stored in the storage device in descending order of storage priority.

Images