File processing method and device and electronic equipment
By obtaining the data file set in the key-value storage system, calculating the data cohesive values inside and between the files, and selecting the file that needs to be compacted as the file to be compacted, the problem of difficult to identify the file to be compacted in the prior art and improving query efficiency.
Patent Information
- Application Number
- CN202411836043.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-03-18
AI Technical Summary
When facing files with different data distribution characteristics such as data file key overlap and key dispersion, it is difficult to accurately identify the files to be compacted that need to be compacted most, resulting in a reduced query efficiency.
By obtaining the data file set, the first data cohesion value of each file is determined in turn, and the degree of cohesion of the keys inside the file is measured; then the first target file is selected based on the first data cohesion value, the second data cohesion value between them is calculated, and the degree of overlap between the keys between different files is measured; finally, the second target file is selected as the file to be compacted based on the second data cohesion value.
Accurately identify the files to be compacted that need to be compacted most in the data file set to improve query efficiency.
Smart Images
Figure CN119645944A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a file processing method, device and electronic device. Background Art
[0002] Key-value storage system is a database system that uses key-value pairs as basic storage units and has been widely used in many fields. Compared with traditional relational databases, each record in a key-value storage system only includes two fields: key and value. However, when faced with continuous data writing and query requests, a large amount of data will be generated in the key-value storage system. Since the data is scattered and stored in multiple files, accessing multiple files will lead to a significant decrease in read performance, so multiple files need to be compacted.
[0003] Existing file compaction methods usually select files to be compacted based on the number of files, file size and file tiering strategy, and integrate the data in the files to be compacted into a new large file through multi-way merging processing technology.
[0004] However, when facing files with different data distribution characteristics such as overlapping data file keys and scattered keys, the existing technology often finds it difficult to accurately identify the files to be compacted that need to be compacted the most, thereby reducing query efficiency. Summary of the invention
[0005] In view of this, the present disclosure provides a file processing method, device and electronic device, which can accurately identify the files to be compacted that most need to be compacted in a data file set, thereby improving query efficiency.
[0006] According to a first aspect of the present disclosure, a file processing method is provided, the method comprising:
[0007] Get the data file set;
[0008] Determine in sequence a first data cohesion value of each file in the data file set, where the first data cohesion value is used to measure the cohesion degree of the internal keys of each file;
[0009] Determining a second data cohesion value between first target files according to the first data cohesion value, the first target files being files selected from the data file set, and the second data cohesion value being used to measure a degree of overlap of keys between the first target files;
[0010] According to the second data cohesion value, a second target file is selected as the file to be compacted, and the second target file is a file selected from the first target file.
[0011] According to a second aspect of the present disclosure, there is provided a file processing device, the device comprising:
[0012] An acquisition module, used to acquire a data file set;
[0013] A first determination module, used to sequentially determine a first data cohesion value of each file in the data file set, where the first data cohesion value is used to measure the cohesion degree of the internal keys of each file;
[0014] a second determination module, configured to determine a second data cohesion value between first target files according to the first data cohesion value, wherein the first target files are files selected from the data file set, and the second data cohesion value is used to measure the degree of overlap of keys between the first target files;
[0015] A selection module is used to select a second target file as a file to be compacted according to the second data cohesion value, where the second target file is a file selected from the first target file.
[0016] According to the third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the aforementioned first aspect.
[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method of the aforementioned first aspect.
[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method of the aforementioned first aspect when executed by a processor.
[0019] The file processing method, device and electronic device provided by the present disclosure are specifically obtained by obtaining a data file set; determining the first data cohesion value of each file in the data file set in turn, the first data cohesion value is used to measure the cohesion degree of the internal key of each file; determining the second data cohesion value between the first target files according to the first data cohesion value, the first target file is a file selected from the data file set, and the second data cohesion value is used to measure the overlap degree of the keys between the first target files; according to the second data cohesion value, selecting the second target file as the file to be compacted, the second target file is a file selected from the first target file. Compared with the prior art, the present disclosure comprehensively considers the cohesion degree of the internal key of each file in the data file set and the overlap degree of the keys between the first target files selected from the data file set, so as to accurately identify the files to be compacted that most need to be compacted in the data file set, thereby improving the query efficiency.
[0020] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0023] Figure 1 A flowchart of a file processing method provided by an embodiment of the present disclosure;
[0024] Figure 2 A flowchart of a file processing method provided by an embodiment of the present disclosure;
[0025] Figure 3 A schematic diagram of the structure of a file processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.
[0027] The following describes the file processing method, device and electronic device according to the embodiments of the present disclosure with reference to the accompanying drawings.
[0028] The present disclosure provides a file processing method, device and electronic device, which can accurately identify the files to be compacted that most need to be compacted in a data file set, thereby improving query efficiency.
[0029] like Figure 1 As shown, an embodiment of the present disclosure provides a file processing method, the method comprising:
[0030] Step 101: Obtain a data file set.
[0031] Compaction is a process of merging multiple data files or data blocks into one or more larger files or blocks, while deleting invalid or redundant data to reduce data fragmentation, improve data access speed, and reclaim space for subsequent use.
[0032] The compaction operation can be divided into large compaction operation and small compaction operation according to the compaction scale. The large compaction operation can be to compact all files of a partition into a large file, and the small compaction operation can be to select some files to be compacted according to a certain strategy, and then perform multi-way merging according to the key, and remove expired data, and finally form a large file.
[0033] For the embodiment of the present disclosure, the data file set may be a dynamic file set generated within a preset time period in a key-value storage system (Key-value Store) for performing a small compaction operation.
[0034] As a possible implementation method, after determining the data file set for the small compaction operation, the key-value storage system can remove the files generated by the large compaction operation in the data file set, and can select a preset number of files from the data file set according to a preset selection strategy to perform the small compaction operation. For the specific process, see the following embodiment steps. Wherein, the preset number of files is less than or equal to the number of files in the data file set.
[0035] Step 102: sequentially determine the first data cohesion value of each file in the data file set.
[0036] The first data cohesion value may be an index for measuring the cohesion degree of the internal key of each file in the data file set.
[0037] For the embodiment of the present disclosure, as a possible implementation method, a single-file data cohesion calculation model can be used to sequentially determine the first data cohesion value of each file in the data file set. The single-file data cohesion calculation model can be a mathematical model for evaluating the cohesion degree between data elements in a single file.
[0038] By optimizing the files with a higher first data cohesion value, the number and time of input or output (IO) operations during data access can be reduced, thereby improving data access efficiency.
[0039] Specifically, as a possible implementation method, each file in the data file set may be preprocessed, which may include steps such as data cleaning and format conversion to ensure data consistency and accuracy. Features for calculating data cohesion values may be extracted from the preprocessed files, and the extracted features may be input into a single-file data cohesion calculation model, and the cohesion value may be calculated according to a first preset formula of the model to obtain a first data cohesion value for each file in the data file set.
[0040] Step 103: Determine a second data cohesion value between first target files according to the first data cohesion value, where the first target file is a file selected from the data file set.
[0041] The second data cohesion value may be an indicator for measuring the degree of overlap of keys between different files.
[0042] For the disclosed embodiment, a file with a lower first data cohesion value can be selected from the data file set as the first target file according to the first data cohesion value. As a possible implementation method, a multi-file data cohesion calculation model can be used to calculate the second data cohesion value between the first target files. The multi-file data cohesion calculation model can be a mathematical model for evaluating the cohesion degree of data elements between different files.
[0043] Specifically, as a possible implementation method, the first target file may be preprocessed, which may include steps such as data cleaning and format conversion to ensure data consistency and accuracy. Features for calculating data cohesion values may be extracted from the preprocessed first target file, and the extracted features may be input into a multi-file data cohesion calculation model, and the cohesion values between the multiple first target files may be calculated according to a second preset formula of the model to obtain a second data cohesion value between the first target files.
[0044] Step 104: According to the second data cohesion value, select a second target file as a file to be compacted, where the second target file is a file selected from the first target file.
[0045] According to the embodiment of the present disclosure, a file with a higher second data cohesion value (ie, the second target file) may be selected from the first target file as a file to be compacted according to the calculated second data cohesion value.
[0046] By optimizing the files with higher second data cohesion values and merging the same keys in the files with higher second data cohesion values into the same file, data overlap between files and merge operations (Merge) during query can be reduced, thereby improving query efficiency.
[0047] In summary, according to a file processing method disclosed in the present invention, the present invention obtains a data file set; sequentially determines the first data cohesion value of each file in the data file set, the first data cohesion value is used to measure the cohesion degree of the internal key of each file; determines the second data cohesion value between the first target files according to the first data cohesion value, the first target file is a file selected from the data file set, and the second data cohesion value is used to measure the overlap degree of the keys between the first target files; according to the second data cohesion value, selects the second target file as the file to be compacted, the second target file is a file selected from the first target file. Compared with the prior art, the present invention comprehensively considers the cohesion degree of the internal key of each file in the data file set and the overlap degree of the keys between the first target files selected from the data file set, so as to accurately identify the files to be compacted that most need to be compacted in the data file set, thereby improving the query efficiency.
[0048] based on Figure 1 The embodiment shown, Figure 2 A flowchart of a file processing method provided by the present disclosure is further shown. Figure 2 As shown, the method may include:
[0049] Step 201: Calculate binary value differences of adjacent keys of each file in the data file set in turn; determine the first data cohesion value of each file by accumulating the binary value differences of each file.
[0050] Following step 102 of the embodiment, when the single-file data cohesion calculation model is used to calculate the cohesion value of each file in the data file set according to the first preset formula of the model, as a possible implementation method, the binary value difference of adjacent keys of each file in the data file set can be calculated in sequence; by accumulating the binary value difference of each file, the first data cohesion value of each file is determined, wherein the first preset formula can be as follows:
[0051]
[0052] Where O(x)1 represents the first data cohesion value, x i Can represent the binary value of the i-th key in the file, x i+1 Can represent the binary value of the i+1th key in the file.
[0053] For the embodiments of the present disclosure, the smaller the calculated first data cohesion value is, the lower the data cohesion degree of the file is, that is, the higher the discreteness of the keys in the file is, and when performing a range query, the more files need to be read and the higher the sorting complexity is. Therefore, the smaller the first data cohesion value is, the more urgent it is to perform a compaction operation on the file, so as to reorganize the discretely distributed keys into a more compact form, thereby reducing the number of files that need to be read and the sorting complexity when performing a range query.
[0054] For example, there are 5 keys in file F0, namely 1, 2, 3, 4, and 5; there are 5 keys in file F1, namely 1, 3, 5, 7, and 9. According to the first preset formula, the first data cohesion value of file F0 is -4, and the first data cohesion value of file F1 is -8. Since the first data cohesion value of file F1 is relatively small, it means that the keys in file F1 are relatively discrete. Therefore, file F1 should be compacted first to reduce its data cohesion and improve the performance of range query.
[0055] Step 202: Mark each file with a priority according to the first data cohesion value; sort each file in order from high to low priority, and use the target number of files with the highest priority after sorting as the first target file set.
[0056] For the embodiments of the present disclosure, each file may be marked with a priority based on the first data cohesion value, wherein a higher first data cohesion value may indicate that the internal keys of the file are tighter and the compaction effect may be smaller, and thus the corresponding file priority is lower; conversely, a lower first data cohesion value may indicate that the internal keys of the file are more discrete and the compaction effect may be greater, and thus the corresponding file priority is higher.
[0057] Each file may be sorted in order from high to low priority, and the target number of files with the highest priority after sorting is used as the first target file set. The target number may be the number of files to be selected that is pre-set according to actual conditions. The first target file set may be the target number of files with the highest priority selected.
[0058] The target number may be determined according to the number of files in the data file set and the preset number of files to be compacted that are to be selected from the data file set, wherein the preset number of files is less than or equal to the number of files, and the specific determination method is shown in the following expression:
[0059] j = min(n, 2m)
[0060] Wherein, j may represent the target number, min may represent the minimum value, n may represent the number of files in the data file set, and m may represent the preset number of files.
[0061] Step 203: Determine a second data cohesion value between any two first target files in the first target file set.
[0062] Following step 103 of the embodiment, when using the data cohesion calculation model between multiple files and determining the second data cohesion value between any two first target files in the first target file set according to the second preset formula of the model, as a possible implementation method, the number of intersection elements between the key sets of any two second target files can be calculated; based on the number of intersection elements, the second data cohesion value is determined, wherein the greater the number of intersection elements, the higher the corresponding second data cohesion value. The second preset formula can be as follows:
[0063]
[0064] Where, O(x)2 can represent the second data cohesion value, |K i | can represent the number of keys in the i-th file, |K j | can represent the number of keys in the jth file, and n can represent the number of files in the data file set.
[0065] Since the second data cohesion value between any two first target files in the first target file set of the target number (j) is calculated, a total of Second-rate.
[0066] For the embodiments of the present disclosure, the larger the calculated second data cohesion value is, the higher the degree of data cohesion between files is, that is, there are a large number of identical keys in different files, which will lead to higher disk input or output and more merge operations when performing range queries. Therefore, the higher the second data cohesion value is, the more necessary it is to perform compaction operations on files to reduce the complexity and resource consumption during queries.
[0067] Exemplarily, the first target file set includes file F0, file F1 and file F3. Among them, file F0 has 5 keys, namely 1, 2, 3, 4, 5; file F1 has 5 keys, namely 1, 3, 5, 7, 9; and file F3 has 5 keys, namely 1, 3, 5, 7, 9. According to the first preset formula, the second data cohesion value of file F0 and file F1 is 3, the second data cohesion value of file F1 and file F3 is 5, and the second data cohesion value of file F0 and file F3 is 3. Therefore, if only two files are selected from these three files for compaction operation, file F1 and file F3 should be selected first to reduce data overlap between files and merge operations during query, and improve the efficiency of data storage and query.
[0068] Step 204: sort any two first target files in descending order according to their corresponding second data cohesion values; and select a preset number of second target files with the highest second data cohesion values after sorting as files to be compacted.
[0069] For the embodiments of the present disclosure, any two first target files can be sorted in order from high to low according to the corresponding second data cohesion values. Since files with high second data cohesion values carry more random I / O operations, the urgency of compacting files with high second data cohesion values is also the highest. Therefore, a preset number of second target files with the highest second data cohesion values after sorting are selected as files to be compacted to reduce the storage space occupied by files and improve data access efficiency, thereby reducing the I / O burden of the system and improving overall performance.
[0070] For the embodiments of the present disclosure, compared with the three existing strategies based on the number of files, file size and file layering, the present disclosure first abstracts the data cohesion model that affects query performance, that is, the degree of key cohesion within the file and the degree of key cohesion between files, calculates the data cohesion degree in two dimensions, both within the file and between files, and based on the two-dimensional compaction strategy, compacts the disk IO files that carry more (that is, the files that can most improve query performance), thereby maximizing the effect of compaction in optimizing query performance.
[0071] In summary, the file processing method disclosed in the present invention obtains a data file set; sequentially determines the first data cohesion value of each file in the data file set, the first data cohesion value is used to measure the cohesion degree of the internal keys of each file; determines the second data cohesion value between the first target files according to the first data cohesion value, the first target file is a file selected from the data file set, and the second data cohesion value is used to measure the overlap degree of the keys between the first target files; according to the second data cohesion value, selects the second target file as the file to be compacted, the second target file is a file selected from the first target file. Compared with the prior art, the present invention comprehensively considers the cohesion degree of the internal keys of each file in the data file set and the overlap degree of the keys between the first target files selected from the data file set, so as to accurately identify the files to be compacted that most need to be compacted in the data file set, thereby improving the query efficiency.
[0072] Based on the above Figure 1 and Figure 2 The present disclosure provides a file processing device, such as Figure 3 As shown, the device includes: an acquisition module 31, a first determination module 32, a second determination module 33, and a selection module 34;
[0073] An acquisition module 31 is used to acquire a data file set;
[0074] A first determination module 32, used to sequentially determine a first data cohesion value of each file in the data file set, wherein the first data cohesion value is used to measure the cohesion degree of the internal keys of each file;
[0075] A second determination module 33, configured to determine a second data cohesion value between first target files according to the first data cohesion value, wherein the first target files are files selected from the data file set, and the second data cohesion value is used to measure the degree of overlap of keys between the first target files;
[0076] The selection module 34 is configured to select a second target file as a file to be compacted according to the second data cohesion value, where the second target file is a file selected from the first target file.
[0077] In a specific application scenario, the first determination module 32 may be used to sequentially calculate the binary value differences of adjacent keys of each file in the data file set;
[0078] The first data cohesion value of each file is determined by performing a cumulative operation on the binary value difference of each file.
[0079] In a specific application scenario, the second determination module 33 may be used to mark the priority of each file according to the first data cohesion value, wherein the higher the first data cohesion value is, the lower the corresponding file priority is;
[0080] Sorting each of the files in descending order according to priority, and taking the target number of files with the highest priority after sorting as the first target file set;
[0081] A second data cohesion value between any two first target files in the first target file set is determined.
[0082] In a specific application scenario, the second determination module 33 may be used to calculate the number of intersection elements between the key sets of any two second target files;
[0083] Based on the number of the intersection elements, the second data cohesion value is determined, wherein the greater the number of the intersection elements, the higher the corresponding second data cohesion value.
[0084] In a specific application scenario, the selection module 34 may be used to sort any two of the first target files in order from high to low according to the corresponding second data cohesion values;
[0085] Select the second target files with the highest preset file number of second data cohesion values after sorting as the files to be compacted.
[0086] In specific application scenarios, such as Figure 3 As shown, the device further includes: a third determination module 35;
[0087] The third determination module 35 is used to determine the target number according to the number of files in the data file set and the preset number of files.
[0088] It should be noted that for other corresponding descriptions of the functional units involved in the file processing device provided by the present disclosure, please refer to Figure 1 and Figure 2 The corresponding description in will not be repeated here.
[0089] Based on the above Figure 1 and Figure 2 The method shown in the present disclosure, accordingly, also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned Figure 1 and Figure 2 The method shown.
[0090] Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present disclosure.
[0091] Based on the above Figure 1 and Figure 2 The method shown, and Figure 3 In order to achieve the above-mentioned purpose, the embodiment of the present disclosure also provides an electronic device, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.
[0092] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0093] Those skilled in the art will appreciate that the above-mentioned physical device structure provided by the present disclosure does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.
[0094] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
[0095] Through the description of the above implementation methods, the technical personnel in this field can clearly understand that the present disclosure can be implemented by means of software plus the necessary general hardware platform, or by hardware. According to the file processing method, device and electronic device provided by the present disclosure, the present disclosure obtains a data file set; sequentially determines the first data cohesion value of each file in the data file set, and the first data cohesion value is used to measure the cohesion degree of the internal key of each file; determines the second data cohesion value between the first target files according to the first data cohesion value, the first target file is a file selected from the data file set, and the second data cohesion value is used to measure the overlap degree of the keys between the first target files; according to the second data cohesion value, selects the second target file as the file to be compacted, and the second target file is a file selected from the first target file. Compared with the prior art, the present disclosure comprehensively considers the cohesion degree of the internal key of each file in the data file set and the overlap degree of the keys between the first target files selected from the data file set, so as to accurately identify the files to be compacted that most need to be compacted in the data file set, thereby improving the query efficiency.
[0096] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0097] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied herein.
Claims
1. A file processing method, characterized in that: The method comprises: Get the data file set; Determine in sequence a first data cohesion value of each file in the data file set, where the first data cohesion value is used to measure the cohesion degree of the internal keys of each file; Determining a second data cohesion value between first target files according to the first data cohesion value, the first target files being files selected from the data file set, and the second data cohesion value being used to measure a degree of overlap of keys between the first target files; According to the second data cohesion value, a second target file is selected as the file to be compacted, and the second target file is a file selected from the first target file.
2. The method according to claim 1, characterized in that The step of sequentially determining the first data cohesion value of each file in the data file set includes: Calculating binary value differences of adjacent keys of each file in the data file set in sequence; The first data cohesion value of each file is determined by performing a cumulative operation on the binary value difference of each file.
3. The method according to claim 1, characterized in that The determining the second data cohesion value between the first target files according to the first data cohesion value includes: Marking the priority of each file according to the first data cohesion value, wherein the higher the first data cohesion value is, the lower the corresponding file priority is; Sorting each of the files in descending order according to priority, and taking the target number of files with the highest priority after sorting as the first target file set; A second data cohesion value between any two first target files in the first target file set is determined.
4. The method according to claim 3, characterized in that: The determining the second data cohesion value between any two first target files in the first target file set includes: Calculate the number of intersection elements between the key sets of any two second target files; Based on the number of the intersection elements, the second data cohesion value is determined, wherein the greater the number of the intersection elements is, the higher the corresponding second data cohesion value is.
5. The method according to claim 3, characterized in that: The selecting, according to the second data cohesion value, a second target file as a file to be compacted includes: Sort any two of the first target files in descending order according to the corresponding second data cohesion values; Select the second target files with the highest preset file number of second data cohesion values after sorting as the files to be compacted.
6. The method according to claim 5, characterized in that The method further comprises: The target number is determined according to the number of files in the data file set and the preset number of files.
7. A file processing device, characterized in that: The device comprises: An acquisition module is used to acquire a data file set; A first determination module, used to sequentially determine a first data cohesion value of each file in the data file set, where the first data cohesion value is used to measure the cohesion degree of the internal keys of each file; a second determination module, configured to determine a second data cohesion value between first target files according to the first data cohesion value, wherein the first target files are files selected from the data file set, and the second data cohesion value is used to measure the degree of overlap of keys between the first target files; A selection module is used to select a second target file as a file to be compacted according to the second data cohesion value, where the second target file is a file selected from the first target file.
8. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method described in claims 1-6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in claims 1 to 6 is implemented.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.