A file clustering method, device, equipment and medium

Abstract

The present invention discloses a file clustering method, device, equipment and medium. In the method, identification information and access time of each accessed file recorded in a log file of a preset time period are obtained, the identification information is used as a primary key and the access time is used as a value, each key-value pair is determined, the key-value pairs are sorted according to the access time, and the key-value pairs belonging to each group are determined according to the interval between the access times in the sorted key-value pairs; for the key-value pairs belonging to each group, the target accessed files identified by the identification information of the files recorded in each key-value pair contained in the group are saved in the file set corresponding to the group. Since the method of clustering based on the sorted key-value pairs improves the correlation between the file clustering results and user access, the file access speed is improved, and when determining the key-value pairs belonging to each group, a bottom-up tree merging method is used for clustering, so that the time complexity of clustering is reduced.

Description

Technical Field

[0001] The present invention relates to the field of file storage, and in particular to a file clustering method, device, equipment and medium. Background Art

[0002] With the advent of the information age, emerging fields such as big data and cloud computing have emerged and developed rapidly. In this context, in file storage systems, distributed storage architecture is often used to organize data into files for storage and access.

[0003] When a distributed storage architecture is used for storage in the prior art, there are problems with the file access speed. In order to improve the file access speed, the prior art adopts file clustering based on feature extraction. However, when performing file clustering, it is necessary to determine the similarity between each file and other files, which leads to the consumption of a large amount of computing resources. The complexity of file clustering is relatively high, and the file clustering results are only related to the content features or title features of the file itself, but have poor correlation with the behavioral features of user access. It is impossible to effectively improve the file access speed and reduce the user's access delay. Therefore, how to cluster files to improve the file access speed and reduce the complexity of file clustering becomes a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a file clustering method, device, equipment and medium, which are used to solve the problem that after files are clustered in the prior art, the file access speed is low and there is a waste of resources.

[0005] The present invention provides a file clustering method, the method comprising:

[0006] Obtaining a log file within a preset time length, extracting identification information and access time of each accessed file recorded in the log file, using the identification information as a primary key and the access time as a value, and determining each key-value pair;

[0007] Sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the interval between the access times in the sorted key-value pairs;

[0008] For the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group is stored in the file set corresponding to the group.

[0009] Furthermore, after determining the key-value pairs belonging to each group, and before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group, the method further includes:

[0010] For the identification information of the file recorded in each key-value pair, determine whether the identification information recorded in the key-value pair appears in other key-value pairs. If so, determine the target group where the key-value pair where the identification information appears belongs, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

[0011] Furthermore, determining the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs includes:

[0012] For each key-value pair, if the key-value pair is the one with the highest sorting, divide the key-value pair into a group and use the group as the current activation group; if the key-value pair is the key-value pair that is not the highest sorting, determine the first time interval between the access time of the key-value pair and the access time of the adjacent key-value pair sorted before the key-value pair; if the first time interval is not greater than a predetermined first threshold, divide the key-value pair into the current activation group; if the first time interval is greater than the predetermined first threshold, recreate a group, use the created group as the current activation group, and divide the key-value pair into the current activation group.

[0013] Furthermore, the process of determining the first threshold includes:

[0014] sorting the key-value pairs according to the access time, determining a third time interval between the access times of two adjacent key-value pairs after sorting, and determining the third time interval with the largest number of occurrences as the first threshold; or

[0015] Sort the key-value pairs according to the access time of the key-value pairs, determine the third time interval between the access times of adjacent key-value pairs after the sorting, count the first number of key-value pairs corresponding to each third time interval, and create a first tuple of each third time interval and the corresponding first number; sort each first tuple according to the size of the third time interval, and for each third time interval, determine the first sum of the first numbers in the first tuple of other third time intervals whose time interval is not greater than the third time interval, take each third time interval as the horizontal coordinate value, take the first sum corresponding to each third time interval as the vertical coordinate value, fit a first rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the first rising curve as the first threshold.

[0016] Furthermore, after determining the key-value pairs belonging to each group, and before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group, the method further includes:

[0017] Determine whether the number of divided groups is greater than a preset second threshold;

[0018] If not, performing a subsequent step of storing the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in a file set corresponding to the group;

[0019] If so, determine the average access time of each group based on the access time of each key-value pair contained in each group, sort each group according to the average time, and for each group in turn, if the group is the top-ranked group, use the group as the current activation group; if the group is not the top-ranked group, determine a second time interval between the average time in the group and the average time of the adjacent group sorted before the group; if the first time interval is not greater than a predetermined third threshold, divide the key-value pairs in the group into the current activation group; if the second time interval is greater than the third threshold, use the group as the current activation group, and delete the group to which the key-value pairs belong after redetermining the group; if the number of redetermined groups is greater than the second threshold, update the average time of each group, and sort and judge again based on the updated average time until the number of redetermined groups meets the requirement of not greater than the second threshold, and then perform the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group.

[0020] Furthermore, the process of determining the third threshold includes:

[0021] sorting each group according to the average access time of the key-value pairs in each group, determining a fourth time interval between the average times of two adjacent sorted groups, counting the second number of key-value pairs corresponding to each fourth time interval, and determining the target fourth time interval when the second number is the largest as the third threshold; or

[0022] Sort the key-value pairs according to the average time of the access time of the key-value pairs in each group, determine the fourth time interval between the average time of two adjacent groups after sorting, count the second number of key-value pairs corresponding to each fourth time interval, create a second tuple of each fourth time interval and the corresponding second number, sort each second tuple according to the size of the fourth time interval, and for each fourth time interval, determine the second sum of the second quantities in the second tuple of each other fourth time interval whose time interval is not greater than the fourth time interval, take each fourth time interval as the horizontal axis value, take the second sum corresponding to each fourth time interval as the vertical axis value, fit a second rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the second rising curve as the third threshold.

[0023] Accordingly, the present invention provides a file clustering device, the device comprising:

[0024] An acquisition module is used to acquire log files within a preset time length, extract identification information and access time of each accessed file recorded in the log files, use the identification information as a primary key and the access time as a value, and determine each key-value pair;

[0025] A determination module, used to sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs;

[0026] The clustering module is used for storing, for the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group.

[0027] Furthermore, after determining the key-value pairs belonging to each group, before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file collection corresponding to the group, the determination module is also used to determine, for the identification information of the file recorded in each key-value pair, whether the identification information recorded in the key-value pair appears in other key-value pairs, and if so, determine the target group where the key-value pair where the identification information appears is located, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

[0028] Furthermore, the determination module is specifically used to, for each key-value pair, if the key-value pair is the top-ranked key-value pair, divide the key-value pair into a group and use the group as the current activation group; if the key-value pair is the top-ranked key-value pair that is not ranked, determine a first time interval between the access time of the key-value pair and the access time of an adjacent key-value pair ranked before the key-value pair, if the first time interval is not greater than a predetermined first threshold, divide the key-value pair into the current activation group, if the first time interval is greater than a predetermined first threshold, recreate a group, use the created group as the current activation group, and divide the key-value pair into the current activation group.

[0029] Furthermore, the determination module is also used to sort the key-value pairs according to the access time, determine the third time interval between the access times of two adjacent key-value pairs after sorting, and determine the third time interval with the largest number of occurrences as the first threshold; or sort the key-value pairs according to the access time of the key-value pairs, determine the third time interval between the access times of adjacent key-value pairs after sorting, count the first number of key-value pairs corresponding to each third time interval, and create a first tuple of each third time interval and the corresponding first number; sort each first tuple according to the size of the third time interval, and for each third time interval, determine the first sum of the first number in the first tuple of other third time intervals whose time interval is not greater than the third time interval, use each third time interval as the horizontal coordinate value, use the first sum corresponding to each third time interval as the vertical coordinate value, fit a first rising curve, and use the target time interval corresponding to the coordinate point with the largest slope in the first rising curve as the first threshold.

[0030] Furthermore, the device also includes:

[0031] A judgment module is used to judge whether the number of divided groups is greater than a preset second threshold; if not, trigger the clustering module to execute the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group; if so, trigger the determination module to determine the average access time of each group according to the access time of each key-value pair contained in each group, sort each group according to the average time, and for each group in turn, if the group is the top-ranked group, use the group as the current activation group; if the group is not the top-ranked group, determine the second time between the average time in the group and the average time of the adjacent group sorted before the group interval, if the first time interval is not greater than a predetermined third threshold, the key-value pairs in the group are divided into the current activation group; if the second time interval is greater than the third threshold, the group is used as the current activation group, and after the group to which the key-value pairs belong is re-determined, the group without key-value pairs is deleted; if the number of re-determined groups is greater than the second threshold, the average time of each group is updated, and the groups are sorted and judged again according to the updated average time, until the number of re-determined groups meets the requirement of not greater than the second threshold, triggering the clustering module to execute the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group.

[0032] Further, the determination module is also used to sort each group according to the average time of the access time of the key-value pairs in each group, determine the fourth time interval between the average time of two adjacent groups after sorting, count the second number of key-value pairs corresponding to each fourth time interval, and determine the target fourth time interval when the second number is the largest as the third threshold; or sort the key-value pairs in each group according to the average time of the access time of the key-value pairs, determine the fourth time interval between the average time of two adjacent groups after sorting, count the second number of key-value pairs corresponding to each fourth time interval, create a second tuple of each fourth time interval and the corresponding second number, sort each second tuple according to the size of the fourth time interval, and for each fourth time interval, determine the second sum of the second number in the second tuple of each other fourth time interval whose time interval is not greater than the fourth time interval, use each fourth time interval as the horizontal coordinate value, use the second sum corresponding to each fourth time interval as the vertical coordinate value, fit a second rising curve, and use the target time interval corresponding to the coordinate point with the largest slope in the second rising curve as the third threshold.

[0033] Accordingly, the present invention provides an electronic device, which includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the steps of any of the above-mentioned file clustering methods when executing the computer program stored in the memory.

[0034] Accordingly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned file clustering methods.

[0035] The present invention provides a file clustering method, device, equipment and medium. In the method, log files of a preset time period are obtained, identification information and access time of each accessed file recorded in the log files are extracted, identification information is used as a primary key and access time is used as a value, each key-value pair is determined, and the key-value pairs are sorted according to the access time. That is, in the present invention, key-value pairs formed based on user operation behaviors are sorted in the order of access time, and key-value pairs belonging to each group are determined according to the intervals between access times in the sorted key-value pairs; for the key-value pairs belonging to each group, target accessed files identified by identification information of files recorded in each key-value pair contained in the group are saved in a file set corresponding to the group. Since the result of clustering files based on the sorted key-value pairs conforms to the changes and diversity of user interests, the relevance of the file clustering result and user access is improved, thereby improving the file access speed. In addition, the key-value pairs belonging to each group are determined based on the intervals between access times in the sorted key-value pairs, and clustering is performed using a bottom-up tree merging method, thereby reducing the time complexity of clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A schematic diagram of a process of a file clustering method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a file collection provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of determining a group to which a key-value pair belongs provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a specific file clustering method provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of the structure of a file clustering device provided in this application;

[0042] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In order to cluster files to improve file access speed and reduce the complexity of file clustering, embodiments of the present invention provide a file clustering method, apparatus, device and medium.

[0045] Embodiment 1:

[0046] Figure 1 A schematic diagram of a process of a file clustering method provided by an embodiment of the present invention, the process includes the following steps:

[0047] S101: Obtain a log file within a preset time length, extract identification information and access time of each accessed file recorded in the log file, use the identification information as a primary key and the access time as a value, and determine each key-value pair.

[0048] In order to achieve file clustering to improve file access speed and reduce the complexity of file clustering, a file clustering method provided in an embodiment of the present invention is applied to an electronic device, wherein the electronic device can be a host, a tablet computer, a mobile terminal or other intelligent terminal, or a server; the server can be a local server or a cloud server; specifically, the embodiment of the present invention does not impose any limitation on this.

[0049] In order to improve the file access speed, the electronic device clusters the files based on the user's behavioral characteristics, so the electronic device obtains log files within a preset time length, where the preset time length can be one day, one week, or 24 hours; the log files include the system log files and operation log files of the electronic device, etc.

[0050] After obtaining the log file within a preset time length, in order to determine the operation information related to the user behavior characteristics, the electronic device extracts the identification information and access time of each accessed file recorded in the log file, where the identification information can be the storage location of the accessed file, or the index number of the accessed file, etc. The log file records the operation information of multiple different access times of the same accessed file, and the operation information of different access times of different accessed files.

[0051] For each piece of operation information, the identification information of the accessed file in the piece of operation information is used as the primary key and the access time is used as the value to obtain a key-value pair corresponding to each piece of operation information.

[0052] As a possible real-time method, in an embodiment of the present invention, the electronic device can also extract the operation type of each accessed file recorded in the log file when it is accessed, where the operation type includes (read) read operation and (write) write operation, and use the identification information of the accessed file in the operation information as the primary key, and the access time and the operation type when accessed as the value to obtain the key-value pair corresponding to each operation information.

[0053] For example, the user operation information extracted from the log file includes 9:00 read file 2, 9:01 read file 3, 9:08 read file 3, 10:00 write file 3, 9:39 read file 5, 9:02 write file 4, 9:05 read file 4, 9:30 read file 2, 15:00 read file 4, 15:03 write file 4; then the key-value pairs corresponding to each operation information include <′2′, (9:00, read)>,<′3′,(9:01,read)>,<′3′,(9:08,read)>,<′3′,(10:00,write)>,<′5′,(9:39,read)>,<′4′,( 9:02,write)>,<′4′,(9:05,read)>,<′2′,(9:30,read)>,<′4′,(15:00,read)>,<′4′,(15:03,write)>.

[0054] S102: sorting the key-value pairs according to the access time, and determining the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs.

[0055] The electronic device sorts the key-value pairs in ascending order of access time according to the access time recorded in each key-value pair, and determines the interval between the access times of every two adjacent key-value pairs according to the access time recorded in the sorted key-value pairs.

[0056] For example, taking the key-value pairs including <′2′,(9:00,read)>, <′3′,(9:01,read)>, <′3′,(9:08,read)>, <′3′,(10:00,write)>, <′5′,(9:39,read)>, <′4′,(9:02,write)>, <′4′,(9:05,read)>, <′2′,(9:30,read)>, <′4′,(15:00,read)>, <′4′,(15:03,write)> as an example, after sorting the access time recorded in the key-value pairs from small to large, the sorted key-value pairs are obtained as <′2′,(9:00,read) >, <′3′,(9:01,read)>, <′4′,(9:02,write)>, <′4′,(9:05,read)>, <′3′,(9:08,read)>, <′2′,(9:30,read)>, <′5′,(9:39,read)>, <′3′,(10:00,write)>, <′4′,(15:00,read)>, <′4′,(15:03,write)>; According to the access time recorded in the sorted key-value pairs, the interval between the access time of each adjacent key-value pairs is determined to be 1 minute, 1 minute, 3 minutes, 3 minutes, 22 minutes, 9 minutes, 21 minutes, 5 hours, and 3 minutes.

[0057] According to the intervals between the access times in the sorted key-value pairs and a predetermined time threshold, a bottom-up tree merging method is used to merge the key-value pairs whose intervals are not greater than the time threshold into one group, thereby determining the key-value pairs belonging to each group.

[0058] S103: For the key-value pairs belonging to each group, save the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group in the file set corresponding to the group.

[0059] For the key-value pairs belonging to each group, determine the identification information of the files recorded in each key-value pair contained in the group; based on the identification information of each file corresponding to the group, determine each target accessed file identified by the identification information; and save each target accessed file in the file set corresponding to the group.

[0060] For example, when the key-value pairs belong to 5 groups, the key-value pairs belonging to each group are (<′2′,(9:00,read)>, <′3′,(9:01,read)>, <′4′,(9:02,write)>, <′4′,(9:05,read)>, <′3′,(9:08,read)>), (<′2′,(9:30,read)>), (<′5′,(9:39,read)>), (<′3′,(10:00,write)>), (<′4′,(15:00,read)>, <′4′,(15:03,write)>); then the file sets corresponding to each group are (file 2, file 3, file 4, file 4, file 2, file 3), (file 2), (file 5), (file 3), (file 4, file 4).

[0061] In the embodiment of the present invention, a log file of a preset time period is obtained, identification information and access time of each accessed file recorded in the log file are extracted, identification information is used as a primary key and access time is used as a value, each key-value pair is determined, and the key-value pairs are sorted according to the access time, that is, in the present invention, key-value pairs formed based on user operation behaviors are sorted in the order of access time, and key-value pairs belonging to each group are determined according to the intervals between access times in the sorted key-value pairs; for the key-value pairs belonging to each group, target accessed files identified by identification information of files recorded in each key-value pair contained in the group are saved in a file set corresponding to the group; since the result of clustering files based on the sorted key-value pairs conforms to the changes and diversity of user interests, the relevance of file clustering results and user access is improved, thereby improving the file access speed; and the key-value pairs belonging to each group are determined based on the intervals between access times in the sorted key-value pairs, and clustering is performed using a bottom-up tree merging method, thereby reducing the time complexity of clustering.

[0062] Embodiment 2:

[0063] In order to determine the file set in which the file is finally saved, based on the above embodiment, in the embodiment of the present invention, after determining the key-value pairs belonging to each group, before saving the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group in the file set corresponding to the group, the method further includes:

[0064] For the identification information of the file recorded in each key-value pair, determine whether the identification information recorded in the key-value pair appears in other key-value pairs. If so, determine the target group where the key-value pair where the identification information appears belongs, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

[0065] Since each key-value pair may record the identification information of the same file, the same target accessed file may be saved in different file collections. In order to determine the file collection in which the file is finally saved, the electronic device determines, for the identification information of the file recorded in each key-value pair, whether the identification information recorded in other key-value pairs is the same as the identification information based on the identification information recorded in the key-value pair and the identification information of the file recorded in other key-value pairs. If so, it is determined that the identification information recorded in the key-value pair appears in other key-value pairs. If not, it is determined that the identification information recorded in the key-value pair does not appear in other key-value pairs.

[0066] If it is determined that the identification information recorded in the key-value pair appears in other key-value pairs, then determine each target group where the key-value pair recording the identification information is located, and determine the target group with the largest number of appearances based on the number of times the key-value pair recording the identification information appears in each group, and determine the target group with the largest number of appearances as the group to which the identification information ultimately belongs, delete the key-value pairs where the identification information appears in other target groups, and retain only one key-value pair recording the identification information in the target group with the largest number of appearances, and delete the other redundant key-value pairs recording the identification information.

[0067] According to the key-value pairs saved in each group, the identification information of the file recorded by any key-value pair in each group is different from the identification information of the file recorded by other key-value pairs, and the identification information of the file recorded by any key-value pair in each group is different from the identification information of the file recorded by any key-value pair in any other groups, determine the identification information of the file recorded by each key-value pair contained in the group, save the target accessed file identified by the identification information in the file set corresponding to the group, each target accessed file in the file set corresponding to the group is different, and any target accessed file in the file set corresponding to the group is different from any target accessed file in the file set corresponding to any other groups.

[0068] As a possible real-time method, in order to determine the file collection in which the file is finally saved, in an embodiment of the present invention, after the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group is saved in the file collection corresponding to the group, since the user may operate on the same accessed file at different access times, the same accessed file may appear in different file collections, and multiple identical accessed files for the same file may also appear in a file collection. For example, when the file collections are (file 2, file 3, file 4, file 4, file 2, file 3), (file 2), (file 5), (file 3), and (file 4, file 4), file 4 is respectively saved in the first collection and the fourth collection, and there are two files 4 in the first collection.

[0069] In order to ensure that each accessed file has a unique file set and that accessed files are not stored repeatedly in each file set, the electronic device, for the identification information of the file recorded in each key-value pair, if the first target accessed file identified by the identification information recorded in the key-value pair is stored in the file sets corresponding to at least two groups, then, based on the number of times the first target accessed file is stored in the file sets corresponding to the at least two groups, the target file set with the largest number of times is determined as the file set where the first target accessed file is finally located, the first target accessed file appearing in other file sets is deleted, and only one instance of the first target accessed file is stored in the target file set.

[0070] Figure 2 A schematic diagram of a file collection provided by an embodiment of the present invention, such as Figure 2 As shown, both file set 1 and file set 2 include star-shaped first target accessed files, and the number of times the first target accessed files appear in file set 1 is greater than the number of times the star-shaped first target accessed files appear in file set 2. Therefore, file set 1 is determined as the file set where the star-shaped first target accessed files are finally located, the star-shaped first target accessed files in file set 2 are deleted, and only one star-shaped first target accessed file is saved in file set 1.

[0071] Embodiment 3:

[0072] In order to determine the key-value pairs belonging to each group, based on the above embodiments, in an embodiment of the present invention, determining the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs includes:

[0073] For each key-value pair, if the key-value pair is the one with the highest sorting, divide the key-value pair into a group and use the group as the current activation group; if the key-value pair is the key-value pair that is not the highest sorting, determine the first time interval between the access time of the key-value pair and the access time of the adjacent key-value pair sorted before the key-value pair; if the first time interval is not greater than a predetermined first threshold, divide the key-value pair into the current activation group; if the first time interval is greater than the predetermined first threshold, recreate a group, use the created group as the current activation group, and divide the key-value pair into the current activation group.

[0074] For each key-value pair after sorting, first determine whether the key-value pair is the top-ranked key-value pair. If the key-value pair is the top-ranked key-value pair, first create a group and divide the key-value pair into the group separately, and use the group as the current activation group; if the key-value pair is not the top-ranked key-value pair, then determine the first time interval of the first access time minus the second access time based on the first access time of the key-value pair and the second access time of the adjacent key-value pair sorted before the key-value pair.

[0075] In order to determine the group to which each key-value pair belongs, the electronic device also stores a predetermined first threshold, wherein the first threshold is a time threshold for determining whether the time interval is large. If one wishes to reduce the number of groups to which the key-value pairs belong, the first threshold can be set larger; if one wishes to increase the number of groups to which the key-value pairs belong, the first threshold can be set smaller.

[0076] According to the first time interval and a predetermined first threshold, if the first time interval is not greater than the first threshold, it is determined that the first time interval is small, and the key-value pair is divided into the current activation group; if the first time interval is greater than the first threshold, it is determined that the first time interval is large, and a new group is created, and the created group is used as the current activation group, and the key-value pair is divided into the current activation group.

[0077] In order to determine the first threshold used in the current key-value pair partitioning process, in an embodiment of the present invention, the process of determining the first threshold includes:

[0078] sorting the key-value pairs according to the access time, determining a third time interval between the access times of two adjacent key-value pairs after sorting, and determining the third time interval with the largest number of occurrences as the first threshold; or

[0079] Sort the key-value pairs according to the access time of the key-value pairs, determine the third time interval between the access times of adjacent key-value pairs after the sorting, count the first number of key-value pairs corresponding to each third time interval, and create a first tuple of each third time interval and the corresponding first number; sort each first tuple according to the size of the third time interval, and for each third time interval, determine the first sum of the first numbers in the first tuple of other third time intervals whose time interval is not greater than the third time interval, take each third time interval as the horizontal coordinate value, take the first sum corresponding to each third time interval as the vertical coordinate value, fit a first rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the first rising curve as the first threshold.

[0080] According to the access time recorded in each key-value pair, the key-value pairs are sorted according to the access time to determine the sorted key-value pairs, and for each key-value pair after the first sorted key-value pair, according to the third access time in the key-value pair and the fourth access time of the adjacent key-value pair before the key-value pair, a third time interval of the third access time minus the fourth access time is determined, thereby determining each third time interval before the access times of every two adjacent key-value pairs.

[0081] According to each third time interval, since each third time interval includes the same third time interval, the third time interval with the largest number of occurrences is determined, and the third time interval with the largest number of occurrences is determined as the first threshold.

[0082] As a possible implementation, after determining the third time interval between the access times of sorted adjacent key-value pairs, the electronic device determines, for each third time interval, the number of key-value pairs corresponding to the third time interval. Since one time interval corresponds to two key-value pairs, based on the number of times the third interval occurs, the product of the number of occurrences and the quantity 2 is the first number of key-value pairs corresponding to the third time interval.

[0083] According to each third time interval and the corresponding first quantity, each first tuple of each third time interval and the corresponding first quantity is created; according to the third time interval in each first tuple, the first tuples are sorted in ascending order of the third time intervals to obtain sorted first tuples.

[0084] For each third time interval, determine a first tuple containing other third time intervals whose time intervals are not greater than the third time interval, add the first quantities in the first tuple containing other third time intervals, and obtain a first sum value corresponding to the third time interval.

[0085] According to each third time interval and the first sum value corresponding to each third time interval, each third time interval is used as the horizontal coordinate value, and the first sum value corresponding to each third time interval is used as the vertical coordinate value, each target coordinate point in the coordinate system is determined, and each target coordinate is fitted to obtain a first rising curve. According to the first rising curve, the coordinate point with the largest slope in the first rising curve is determined, and the target time interval corresponding to the coordinate point with the largest slope is used as the first threshold.

[0086] Embodiment 4:

[0087] In order to determine the key-value pairs belonging to each group and ensure that the number of groups meets the requirements, based on the above embodiments, in an embodiment of the present invention, after determining the key-value pairs belonging to each group, before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group, the method further includes:

[0088] Determine whether the number of divided groups is greater than a preset second threshold;

[0089] If not, performing a subsequent step of storing the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in a file set corresponding to the group;

[0090] If so, determine the average access time of each group based on the access time of each key-value pair contained in each group, sort each group according to the average time, and for each group in turn, if the group is the top-ranked group, use the group as the current activation group; if the group is not the top-ranked group, determine a second time interval between the average time in the group and the average time of the adjacent group sorted before the group; if the first time interval is not greater than a predetermined third threshold, divide the key-value pairs in the group into the current activation group; if the second time interval is greater than the third threshold, use the group as the current activation group, and delete the group to which the key-value pairs belong after redetermining the group; if the number of redetermined groups is greater than the second threshold, update the average time of each group, and sort and judge again based on the updated average time until the number of redetermined groups meets the requirement of not greater than the second threshold, and then perform the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group.

[0091] Since in the embodiment of the present invention, there is a preset upper limit on the number of file sets, which is the second threshold value. Therefore, after determining the key-value pairs belonging to each group, it is necessary to judge whether the number of groups is greater than the second threshold value based on the number of divided groups and the second threshold value. If the number of groups is not greater than the second threshold value, it is determined that there is no need to re-determine the group to which the key-value pairs belong. Therefore, the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group is performed.

[0092] If the number of groups is greater than the second threshold, it means that the number of file sets corresponding to each group is greater than the upper limit, so the group to which the key-value pair belongs needs to be re-determined. According to the access time recorded in each key-value pair contained in each group, the average access time of each group is determined, and each group is sorted in chronological order according to the average time of each group to obtain each sorted group.

[0093] For each group after sorting, first determine whether the group is the top-ranked group. If the group is the top-ranked group, then use the group as the current activation group. If the group is not the top-ranked group, then determine the second time interval of the first average time minus the second average time based on the first average time in the group and the second average time of the adjacent group sorted before the group.

[0094] In order to redetermine the group to which each key-value pair belongs, the electronic device also stores a predetermined third threshold, wherein the third threshold is a time threshold for determining whether a time interval is large. If one wishes to reduce the number of groups to which the key-value pairs belong, the third threshold can be set to a larger value; if one wishes to increase the number of groups to which the key-value pairs belong, the third threshold can be set to a smaller value, wherein the third threshold is greater than the first threshold.

[0095] According to the second time interval and a predetermined third threshold, if the second time interval is not greater than the third threshold, it is determined that the second time interval is small, and the key-value pairs in the group are divided into the current activation group; if the second time interval is greater than the third threshold, it is determined that the second time interval is large, and the group is used as the current activation group. After judging each second time interval, the group to which the key-value pair belongs is re-determined, and the group without key-value pairs is deleted.

[0096] Based on the number of groups to which the re-determined key-value pairs belong, if the number of re-determined groups is greater than the second threshold, the average time of each group is updated based on the average access time of the key-value pairs contained in each group, and each group is sorted and judged again based on the updated average time, and the above-mentioned step of re-determining the groups to which the key-value pairs belong is executed until the number of re-determined groups satisfies not greater than the second threshold, and then the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file collection corresponding to the group is executed.

[0097] In order to determine the third threshold used when re-dividing the group to which each key-value pair belongs, in an embodiment of the present invention, the process of determining the third threshold includes:

[0098] sorting each group according to the average access time of the key-value pairs in each group, determining a fourth time interval between the average times of two adjacent sorted groups, counting the second number of key-value pairs corresponding to each fourth time interval, and determining the target fourth time interval when the second number is the largest as the third threshold; or

[0099] Sort the key-value pairs according to the average time of the access time of the key-value pairs in each group, determine the fourth time interval between the average time of two adjacent groups after sorting, count the second number of key-value pairs corresponding to each fourth time interval, create a second tuple of each fourth time interval and the corresponding second number, sort each second tuple according to the size of the fourth time interval, and for each fourth time interval, determine the second sum of the second quantities in the second tuple of each other fourth time interval whose time interval is not greater than the fourth time interval, take each fourth time interval as the horizontal axis value, take the second sum corresponding to each fourth time interval as the vertical axis value, fit a second rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the second rising curve as the third threshold.

[0100] According to the average access time of the key-value pairs in each group, each group is sorted according to the average time, and each group after sorting is determined. For each group after the first sorting group, according to the first average time in the group and the second average time in the adjacent group before the group, a fourth time interval of the first average time minus the second average time is determined, thereby determining the fourth time interval of the average time of every two adjacent groups.

[0101] For each fourth time interval, determine the total number of key-value pairs in two adjacent groups corresponding to the fourth time interval; since the fourth time interval may appear multiple times, determine the sum of the total number of key-value pairs each time the fourth time interval appears, and determine the sum as the second number of key-value pairs corresponding to the fourth time interval; based on the second number corresponding to the fourth time interval, determine the target fourth time interval when the second number is the largest, and determine the target fourth time interval as the third threshold.

[0102] According to each fourth time interval and the corresponding second quantity, each second tuple of each fourth time interval and the corresponding second quantity is created; according to the fourth time interval in each second tuple, the second tuples are sorted in ascending order of the fourth time interval to obtain sorted second tuples.

[0103] For each fourth time interval, determine the second tuple of each other fourth time interval where other time intervals whose time interval is not greater than the fourth time interval are located, add the second quantities in the second tuples where other fourth time intervals are located, and obtain a second sum value corresponding to the fourth time interval.

[0104] According to each fourth time interval and the second sum value corresponding to each fourth time interval, each fourth time interval is used as the horizontal coordinate value, and the second sum value corresponding to each fourth time interval is used as the vertical coordinate value, each target coordinate point in the coordinate system is determined, each target coordinate is fitted to obtain a second rising curve, and the coordinate point with the largest slope in the second rising curve is determined according to the second rising curve, and the target time interval corresponding to the coordinate point with the largest slope is used as the third threshold.

[0105] Embodiment 5:

[0106] A file clustering method of the present invention is described below through a specific embodiment. Step 1: The electronic device obtains a log file within a preset time length, extracts identification information of each accessed file recorded in the log file, that is, the file index number (fs_id) of each accessed file, extracts the access time (time) and operation type (op_type) of each accessed file, uses fs_id as a primary key (key), uses a tuple composed of time and op_type as a value (value), and obtains each key-value pair opi corresponding to each piece of operation information, opi=<key,value> (i=1,…,n), take each key-value pair opi as an operation sequence element, and get the operation sequence set U, U={op 1 ,op 2 ,op 3 ,…,op n}; Alternatively, after determining the group to which each key-value pair belongs, the average access time of each group can be determined based on the access time of each key-value pair contained in each group, and each group can be used as an operation sequence element to obtain an operation sequence set U.

[0107] Step 2: Sort each operation sequence element in the operation sequence set U according to the time sequence to obtain the sorted key-value pair set U′. When the key-value pair is an operation sequence element, the key-value pair is sorted according to the access time sequence to obtain the sorted set U′, for example, U′={op 1 ′,op 2 ′,op 3 ′,…,op n ′}, and T opi′ ≤T opi+1′ (i=1,…,n-1), where T opi′ Indicates the key-value pair op sorted at position i i ′; it can also be that when the group to which the key-value pair belongs is an operation sequence element, after determining the group to which each key-value pair belongs, according to the access time of each key-value pair contained in each group, determine the average access time of each group, take each group as an operation sequence element, obtain the operation sequence set U, sort each operation sequence element in the operation sequence set U in chronological order according to the average time, and obtain the sorted operation sequence set U′.

[0108] Step 3: Merge the operation sequence elements in the operation sequence set U′, and regard each operation sequence element as a node of the tree. The merging process can be regarded as a bottom-up tree merge: starting from the first node, if the time interval between two adjacent nodes is less than or equal to the time threshold TI, the merge generates a parent node, and the time of the father node is the average of the child node time; the parent node continues to merge with the adjacent nodes until the merge end condition is met; starting from the first node, the adjacent nodes are merged according to the new time threshold TI until the merge end condition is met.

[0109] Figure 3 A schematic diagram of determining a group to which a key-value pair belongs provided by an embodiment of the present invention, wherein the key-value pair is used as an element of an operation sequence, such as Figure 3As shown, there are nine operation sequence elements in total, and the time interval between each two adjacent operation sequence elements is: 1s, 1s, 2s, 2s, 3s, 1s, 2s, 3s. When there are two elements left, the merge stops. If the first time threshold TI is selected as 1s, the 1st, 2nd, and 3rd elements can be merged, and the 6th and 7th elements can be merged. After the merge, there are 6 elements in total; if the second time threshold TI is selected as 3.5s, the 1st, 2nd, and 3rd elements can be merged, and the 4th, 5th, and 6th elements can be merged. After the merge, there are two elements in total. At this time, there are only two elements, and the merge stops. Then the nine key-value pairs are assigned to two groups.

[0110] Specifically, there are two processes. The first process is to determine the time threshold TI for each division of the groups to which the key-value pairs belong; the second process is to determine the group to which each key-value pair belongs.

[0111] Process 1: Based on the sorted operation sequence set U′, determine each time interval between every two adjacent operation sequence elements in the sorted operation sequence set, and the number of key-value pairs corresponding to each time interval, or the number of operation types corresponding to each time interval, wherein the number of key-value pairs is the same as the number of operation types, and each key-value pair corresponds to one operation type; if the operation sequence element is a key-value pair, determine each time interval between the access times of every two adjacent key-value pairs, and determine the number of key-value pairs corresponding to each time interval; if the operation sequence element is a group to which the key-value pair belongs, determine the time interval of the average time of the access times of two adjacent groups, and determine the number of key-value pairs corresponding to each group.

[0112] According to each time interval and the number of key-value pairs corresponding to each time interval, each tuple of each time interval and the number of corresponding key-value pairs is created, and a set G is obtained according to each tuple, G = {s 1 ,s 2 ,…,s m}, where s i It is a tuple consisting of the time interval between two adjacent operation sequence elements and the number of key-value pairs corresponding to the time interval.

[0113] According to the time interval of each tuple in the set G, sort each tuple in the order of small to large time interval to obtain the set G′, G′={s 1 ′,s 2 ′,…,s m '},and in Represents a tuple s i ' time interval, T j ∈U′, that is, the time interval between two adjacent elements in the operation sequence set U′.

[0114] Take the sorted key-value pairs <′2′,(9:00,read)>, <′3′,(9:01,read)>, <′4′,(9:02,write)>, <′4′,(9:05,read)>, <′3′,(9:08,read)>, <′2′,(9:60,read)>, <′5′,(9:39,read)>, <′3′,(10:00,write)>, <′4′,(15:00,read)>, <′4′,(15:03,write)> as an example, and put each The key-value pairs are used as operation sequence elements to obtain a sorted operation sequence set U′. According to the sorted operation sequence set U′, each time interval between every two adjacent operation sequence elements and the number of key-value pairs corresponding to each time interval are determined, where the time intervals between every two adjacent key-value pairs are 1 minute, 1 minute, 3 minutes, 3 minutes, 22 minutes, 9 minutes, 21 minutes, 5 hours, and 3 minutes, respectively. The number of key-value pairs corresponding to each time interval is 2. The number of key-value pairs corresponding to the same time interval is accumulated to determine the number of key-value pairs corresponding to each time interval.

[0115] Therefore, we can determine each tuple consisting of each time interval between two adjacent operation sequence elements and the number of key-value pairs corresponding to each time interval, that is, <L 1 ,n 1 >, <L 2 ,n 2 >,…, <L k ,n k >, and L 1 <L 2 <… <L k , where L 1 represents the first time interval, n 1 Indicates the number of key-value pairs corresponding to the first time interval. For example, each time interval includes 1 minute, 3 minutes, 22 minutes, 9 minutes, 21 minutes, and 5 hours. The number of key-value pairs corresponding to the 1-minute time interval is 4, the number of key-value pairs corresponding to the 3-minute time interval is 6, the number of key-value pairs corresponding to the 9-minute time interval is 2, the number of key-value pairs corresponding to the 21-minute time interval is 2, the number of key-value pairs corresponding to the 22-minute time interval is 2, and the number of key-value pairs corresponding to the 5-hour time interval is 2; therefore, each tuple in the determined set G′ includes (1 minute, 4), (3 minutes, 6), (9 minutes, 2), (21 minutes, 2), (22 minutes, 2), and (5 hours, 2).

[0116] According to the time interval in each tuple in the sorted set G′ and the number of corresponding key-value pairs, the threshold for dividing the key-value pairs is determined, that is, the first threshold and the third threshold in the above embodiment. Specifically, the time interval when the number of key-value pairs is the largest can be determined as the threshold, that is, the maximum number of key-value pairs n is determined. j , where n j =max(n i ), (i=1,…,k), determine n j The corresponding L j is the threshold TI; for example, when the number of key-value pairs corresponding to 3 minutes is the largest, the threshold is determined to be 3 minutes.

[0117] Alternatively, based on each tuple in the set G′, for each time interval in the tuple, determine the sum of the number of tuples in which the time interval is not greater than that of each other time interval, take each time interval as the horizontal coordinate value, take the sum corresponding to each time interval as the vertical coordinate value, fit an ascending curve, and take the target time interval corresponding to the coordinate point with the largest slope in the ascending curve as the threshold.

[0118] Specifically, for each time interval L in the tuple i , determine the sum value corresponding to each time interval N i , where N i =n i +n i-1 +…+n 1 (i=2, ..., k), that is, to determine each coordinate point including <L 1 ,N 1 >, <L 2 ,N 2 >, …, <L k ,N k >, fit each coordinate to obtain an ascending curve, and determine the target time interval corresponding to the coordinate point with the largest slope in the curve as the threshold TI.

[0119] Process 2: According to each operation sequence element in the sorted set G′ and the threshold TI, determine the group to which each operation sequence element belongs. For each operation sequence element, if the operation sequence element is the first operation sequence element in the sorting, divide the operation sequence element into a group and use the group as the current activation group. If the operation sequence element is a key-value pair, divide the key-value pair into a group and use it as the activation group. If the operation sequence tuple is the group to which the key-value pair belongs, use the group as the activation group. If the operation sequence element is not the first operation sequence element in the sorting, determine the time interval between the operation sequence element and the adjacent operation sequence element sorted before the operation sequence element, and judge whether the time interval between the operation sequence element and the adjacent operation sequence element sorted before the operation sequence element is not greater than TI, that is, if (i=2,…,k), then op i-1 ′ and op i ′ belongs to the current activation group; if (i=2,…,k) creates a group and uses the created group as the current activation group. k ′ is divided into the current activation group until the last operation sequence element in the set is reached, and finally the set H is obtained, H = {h 1 ,h 2 ,…,h m},and where h i is a subset of H, containing one or more key-value pairs, where if h i If there is only one key-value pair in If h i If there are multiple key-value pairs in

[0120] Take each subset in the current set H as each group to which the key-value pair belongs, determine whether the number m of subsets in the current set H is not greater than the second threshold value, if so, stop merging, determine the key-value pairs belonging to each group, and execute the step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group for the key-value pairs belonging to each group; if not, take H as U′, continue to execute process 1 to redetermine the threshold TI for dividing the key-value pairs, and redetermine each group to which the key-value pairs belong.

[0121] Embodiment 6:

[0122] Figure 4 A schematic diagram of a specific file clustering method provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0123] S401: Obtain log files within a preset time length, extract identification information and access time of each accessed file recorded in the log file, use the identification information as the primary key and the access time as the value, determine each key-value pair, use each key-value pair as an operation sequence element, or use each group to which the key-value pair belongs as an operation sequence element, and obtain an operation sequence set U.

[0124] S402: Sort each operation sequence element in the operation sequence set U in chronological order to obtain a sorted operation sequence set U.

[0125] S403: Determine each time interval between every two adjacent operation sequence elements in the sorted operation sequence set and the number of key-value pairs corresponding to each time interval, create each tuple of each time interval and the number of corresponding key-value pairs, and obtain a set G consisting of each tuple.

[0126] S404: According to the time interval of each tuple in the set G, each tuple is sorted in ascending order of the time interval to obtain a sorted set G.

[0127] S405: Determine a threshold T1 for dividing the key-value pairs according to the time interval in each tuple in the sorted set G and the number of corresponding key-value pairs.

[0128] S406: For each operation sequence element in the sorted operation sequence set U, if the operation sequence element is the top-ranked operation sequence element, divide the operation sequence element into a group and use the group as the current activation group; if the operation sequence element is not the top-ranked operation sequence element, determine the time interval between the operation sequence element and the adjacent operation sequence element sorted before the operation sequence element.

[0129] S407: Determine whether the time interval between the operation sequence element and the adjacent operation sequence element arranged before the operation sequence element is not greater than T1, if so, proceed to S408, if not, proceed to S409.

[0130] S408: Assign the operation sequence element to the current activation group. If the operation sequence element is a key-value pair, assign the key-value pair to the current activation group. If the operation sequence element is a group to which the key-value pair belongs, assign the key-value pair in the operation sequence element to the current activation group and proceed to S410.

[0131] S409: Create a group and use the created group as the current activation group, assign the operation sequence element to the current activation group, and delete the operation sequence element that does not include the key-value pair.

[0132] S410: Determine whether the last operation sequence element has been reached, if not, proceed to S411, if yes, proceed to S412.

[0133] S411: Update the next operation sequence element to the operation sequence element, and proceed to S407.

[0134] S412: Get a set H, which includes m subsets, each of which is a group to which a key-value pair belongs.

[0135] S413: Determine whether m is not greater than a second threshold value, if not, proceed to S414, if yes, proceed to S415.

[0136] S414: Update the set H to the sorted operation sequence set U, and proceed to S403.

[0137] S415: For the identification information of the file recorded in each key-value pair, determine whether the identification information recorded in the key-value pair appears in other key-value pairs. If so, determine the target group where the key-value pair where the identification information appears belongs, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

[0138] S416: For the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group is saved in the file set corresponding to the group.

[0139] In the embodiment of the present invention, files are clustered based on key-value pairs formed by user operations, so that the clustering results are in line with the changes and diversity of user interests, and the user access characteristics can be captured in time, so that the file clustering results are associated with the user interest characteristics, effectively improving the speed of reading and writing files and reducing the user's access delay; and a bottom-up tree merging method is used for clustering, so that the time complexity of clustering is reduced to O (logn), which is short in time and meets the application scenario of public cloud massive file storage.

[0140] Embodiment 7:

[0141] Based on the above embodiments, Figure 5 A schematic diagram of the structure of a file clustering device provided in this application, the device comprising:

[0142] The acquisition module 501 is used to acquire log files within a preset time length, extract identification information and access time of each accessed file recorded in the log files, use the identification information as a primary key and the access time as a value, and determine each key-value pair;

[0143] A determination module 502, configured to sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs;

[0144] The clustering module 503 is used for storing, for each key-value pair belonging to each group, a target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in a file set corresponding to the group.

[0145] Embodiment 8:

[0146] Figure 6 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes: a processor 601 , a communication interface 602 , a memory 603 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 communicate with each other via the communication bus 604 .

[0147] The memory 603 stores a computer program. When the program is executed by the processor 601, the processor 601 performs the following steps:

[0148] Obtaining a log file within a preset time length, extracting identification information and access time of each accessed file recorded in the log file, using the identification information as a primary key and the access time as a value, and determining each key-value pair;

[0149] Sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the interval between the access times in the sorted key-value pairs;

[0150] For the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group is stored in the file set corresponding to the group.

[0151] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0152] The communication interface 602 is used for communication between the electronic device and other devices.

[0153] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0154] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0155] Embodiment 9:

[0156] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0157] Obtaining a log file within a preset time length, extracting identification information and access time of each accessed file recorded in the log file, using the identification information as a primary key and the access time as a value, and determining each key-value pair;

[0158] Sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the interval between the access times in the sorted key-value pairs;

[0159] For the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair included in the group is stored in the file set corresponding to the group.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A file clustering method, characterized in that: The method comprises: Obtaining a log file within a preset time length, extracting identification information and access time of each accessed file recorded in the log file, using the identification information as a primary key and the access time as a value, and determining each key-value pair; Sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the interval between the access times in the sorted key-value pairs; For the key-value pairs belonging to each group, the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group is saved in the file set corresponding to the group; The step of determining the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs includes: For each key-value pair, if the key-value pair is the one with the highest sorting, divide the key-value pair into a group and use the group as the current activation group; if the key-value pair is the key-value pair that is not the highest sorting, determine the first time interval between the access time of the key-value pair and the access time of the adjacent key-value pair sorted before the key-value pair; if the first time interval is not greater than a predetermined first threshold, divide the key-value pair into the current activation group; if the first time interval is greater than the predetermined first threshold, recreate a group, use the created group as the current activation group, and divide the key-value pair into the current activation group.

2. The method according to claim 1, characterized in that After determining the key-value pairs belonging to each group, and before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group, the method further includes: For the identification information of the file recorded in each key-value pair, determine whether the identification information recorded in the key-value pair appears in other key-value pairs. If so, determine the target group where the key-value pair where the identification information appears belongs, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

3. The method according to claim 1, characterized in that: The process of determining the first threshold comprises: sorting the key-value pairs according to the access time, determining a third time interval between the access times of two adjacent key-value pairs after sorting, and determining the third time interval with the largest number of occurrences as the first threshold; or Sort the key-value pairs according to the access time of the key-value pairs, determine the third time interval between the access times of adjacent key-value pairs after the sorting, count the first number of key-value pairs corresponding to each third time interval, and create a first tuple of each third time interval and the corresponding first number; sort each first tuple according to the size of the third time interval, and for each third time interval, determine the first sum of the first numbers in the first tuple of other third time intervals whose time interval is not greater than the third time interval, take each third time interval as the horizontal coordinate value, take the first sum corresponding to each third time interval as the vertical coordinate value, fit a first rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the first rising curve as the first threshold.

4. The method according to claim 1, characterized in that: After determining the key-value pairs belonging to each group, and before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group, the method further includes: Determine whether the number of divided groups is greater than a preset second threshold; If not, performing a subsequent step of storing the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in a file set corresponding to the group; If so, determine the average access time of each group based on the access time of each key-value pair contained in each group, sort each group according to the average time, and for each group in turn, if the group is the top-ranked group, use the group as the current activation group; if the group is not the top-ranked group, determine a second time interval between the average time in the group and the average time of the adjacent group sorted before the group; if the second time interval is not greater than a predetermined third threshold, classify the key-value pairs in the group into the current activation group; if the second time interval is greater than the third threshold, use the group as the current activation group, and delete the group to which the key-value pairs belong after redetermining the group; if the number of redetermined groups is greater than the second threshold, update the average time of each group, and sort and judge again based on the updated average time until the number of redetermined groups meets the requirement of not greater than the second threshold, and then perform the subsequent step of saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file set corresponding to the group.

5. The method according to claim 4, characterized in that The process of determining the third threshold comprises: sorting each group according to the average access time of the key-value pairs in each group, determining a fourth time interval between the average times of two adjacent sorted groups, counting the second number of key-value pairs corresponding to each fourth time interval, and determining the target fourth time interval when the second number is the largest as the third threshold; or Sort the key-value pairs according to the average time of the access time of the key-value pairs in each group, determine the fourth time interval between the average time of two adjacent groups after sorting, count the second number of key-value pairs corresponding to each fourth time interval, create a second tuple of each fourth time interval and the corresponding second number, sort each second tuple according to the size of the fourth time interval, and for each fourth time interval, determine the second sum of the second quantities in the second tuple of each other fourth time interval whose time interval is not greater than the fourth time interval, take each fourth time interval as the horizontal axis value, take the second sum corresponding to each fourth time interval as the vertical axis value, fit a second rising curve, and take the target time interval corresponding to the coordinate point with the largest slope in the second rising curve as the third threshold.

6. A file clustering device, characterized in that: The device comprises: An acquisition module is used to acquire log files within a preset time length, extract identification information and access time of each accessed file recorded in the log files, use the identification information as a primary key and the access time as a value, and determine each key-value pair; A determination module, used to sort the key-value pairs according to the access time, and determine the key-value pairs belonging to each group according to the intervals between the access times in the sorted key-value pairs; A clustering module, for storing, for each key-value pair belonging to each group, a target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in a file set corresponding to the group; Among them, the determination module is specifically used to, for each key-value pair, if the key-value pair is the top-ranked key-value pair, divide the key-value pair into a group and use the group as the current activation group; if the key-value pair is the top-ranked key-value pair that is not ranked, determine the first time interval between the access time of the key-value pair and the access time of the adjacent key-value pair ranked before the key-value pair, if the first time interval is not greater than a predetermined first threshold, divide the key-value pair into the current activation group, if the first time interval is greater than a predetermined first threshold, recreate a group, use the created group as the current activation group, and divide the key-value pair into the current activation group.

7. The device according to claim 6, characterized in that After determining the key-value pairs belonging to each group, and before saving the target accessed file identified by the identification information of the file recorded in each key-value pair contained in the group in the file collection corresponding to the group, the determination module is further used to determine, for the identification information of the file recorded in each key-value pair, whether the identification information recorded in the key-value pair appears in other key-value pairs; if so, determine the target group where the key-value pair where the identification information appears belongs, count the number of times the identification information appears in each target group, determine the target group with the largest number of appearances as the group to which the identification information belongs, and delete the key-value pairs where the identification information appears in other target groups.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; A computer program is stored in the memory. When the program is executed by the processor, the processor executes the computer program stored in the memory to implement the steps of the file clustering method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the file clustering method as claimed in any one of claims 1 to 5.

Images