Intelligent file merging method and device, electronic equipment and storage medium

Abstract

The invention provides an intelligent file merging method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and cloud computing, in particular to the technical direction of big data. According to the specific implementation scheme, first-class files and second-class files are recognized from a to-be-processed file set, the file size of the first-class files is smaller than a target size, and the file size of the second-class files is larger than that of the first-class files; splitting the second class file into a plurality of sub-files; determining a to-be-merged file set based on the plurality of sub-files and the first type of files; and carrying out merging operation on the to-be-merged file set.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence and cloud computing, and particularly relates to the technical field of big data. BACKGROUND

[0002] In data-intensive businesses, enterprises usually use data lakes to store and analyze massive amounts of structured and unstructured data.

[0003] However, due to the large amount of data in the data lake, the query performance is affected to a certain extent, and therefore a file merging method is needed to improve the performance of the data lake. SUMMARY

[0004] The present disclosure provides a file intelligent merging method and device, an electronic device, and a storage medium.

[0005] According to an aspect of the present disclosure, a file intelligent merging method is provided, comprising:

[0006] identifying a first type of file and a second type of file from a set of files to be processed, the file size of the first type of file being less than a target size, and the file size of the second type of file being greater than the file size of the first type of file;

[0007] splitting the second type of file into a plurality of sub-files;

[0008] determining a set of files to be merged based on the plurality of sub-files and the first type of file;

[0009] performing a merging operation on the set of files to be merged.

[0010] According to another aspect of the present disclosure, a file intelligent merging device is provided, comprising:

[0011] an identification module configured to identify a first type of file and a second type of file from a set of files to be processed, the file size of the first type of file being less than a target size, and the file size of the second type of file being greater than the file size of the first type of file;

[0012] a splitting module configured to split the second type of file into a plurality of sub-files;

[0013] a determination module configured to determine a set of files to be merged based on the plurality of sub-files and the first type of file;

[0014] a merging module configured to perform a merging operation on the set of files to be merged.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory in communication with the at least one processor; wherein

[0018] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method according to any of the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0021] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0023] Figure 1a is a scenario diagram of a file intelligent merging method according to an embodiment of the present disclosure;

[0024] Figure 1b is a flow diagram of a file intelligent merging method according to an embodiment of the present disclosure;

[0025] Figure 2 is a flow diagram of identifying a first type of file according to an embodiment of the present disclosure;

[0026] Figure 3 is a flow diagram of identifying a second type of file according to an embodiment of the present disclosure;

[0027] Figure 4 is a flow diagram of a file intelligent merging method according to another embodiment of the present disclosure;

[0028] Figure 5 is a flow diagram of a file intelligent merging method according to another embodiment of the present disclosure;

[0029] Figure 6 is a flow diagram of a file intelligent merging method according to another embodiment of the present disclosure;

[0030] Figure 7is a schematic diagram of an overall flow of a file intelligent merging method according to another embodiment of the present disclosure;

[0031] Figure 8 is a structural schematic diagram of a file intelligent merging device according to another embodiment of the present disclosure;

[0032] Figure 9 is a block diagram of an electronic device for implementing the file intelligent merging method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are set forth in order to provide an overall understanding of the embodiments of the present disclosure. It should be understood that the embodiments described herein are merely exemplary and that a variety of modifications and alterations can be made without departing from the scope of the present disclosure. Also, the description set forth in the following description of the embodiments of the present disclosure is not limitative. It should be understood that the embodiments of the present disclosure can include other modifications, alterations, and / or substitutions without departing from the scope of the present disclosure.

[0034] The terms "first", "second", and the like in the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0035] It should be noted that the operations shown in the flowcharts of the embodiments of the present disclosure can be executed in any order unless the execution order between different operations is explicitly stated or the execution order between different operations is technically necessary. The execution order between multiple operations can not be sequential, and multiple operations can be executed simultaneously.

[0036] A data lake is a system or repository that stores large amounts of raw data (i.e., data that has not been processed or transformed). The core idea is to store all data from various data sources in one place in the form of files or objects, regardless of whether it is structured data (such as data from a relational database), semi-structured data (such as data in XML (Extensible Markup Language), JSON (JavaScript Object Notation), CSV (Comma-Separated Values), etc.), unstructured data (such as audio, video, images, etc.), or binary data. However, traditional data lakes face challenges such as high data management complexity, difficulty in data quality control, and insufficient real-time query performance.

[0037] Therefore, data lake supports the introduction of "data lake table format" or "data lake table format". Among them, the representative technology is, for example, Iceberg (Apache Iceberg). As a new table format standard, Iceberg significantly improves the use experience and management ability of data lake by supporting multi-table version management, fine partition pruning, transaction processing, and Schema (schema) evolution. By supporting Iceberg, it can ensure that data from different sources can be seamlessly accessed, converted and loaded into the data lake, and efficient query and data governance capabilities can be achieved within the data lake. This has important value for enterprise data middle platform construction, real-time analysis, historical data traceability, and compliance requirements.

[0038] Therefore, the introduction of "data lake table format" or "data lake table format" can help enterprises better manage and utilize data lakes.

[0039] However, in actual use, frequent write operations often lead to the generation of a large number of small files. These small files not only affect query performance, but also often require manual intervention to merge small files, and increase the management overhead of the data lake. In addition, large files also affect data query performance to some extent.

[0040] Therefore, the present disclosure proposes a file intelligent merging method, as shown in Figure 1a The scenario diagram of the application of the method is shown. Figure 1a The server 11 and the terminal device 12 are included in the method.

[0041] The terminal device 12 and the server 11 are connected through a wireless or wired network. The terminal device 12 includes, but is not limited to, a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a smart television, and the like. The server 11 can be a single server, a server cluster composed of a plurality of servers, or a cloud computing center. The server 11 can be a single physical server, a server cluster composed of a plurality of physical servers, or a distributed system. The server 11 can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0042] The file intelligent merging method provided in the embodiments of the present disclosure can be executed by the server 11. After the server 11 completes the merging processing on the files in the data lake, the terminal device 12 can query the server 11.

[0043] A file intelligent merging method is provided in the embodiments of the present disclosure, as shown in the following table. Figure 1b The file intelligent merging method includes the following contents.

[0044] In S101, a first type of file and a second type of file are identified from a set of files to be processed. The file size of the first type of file is less than a target size, and the file size of the second type of file is greater than the file size of the first type of file.

[0045] The first type of file refers to a file with a file size less than the target size, which can be referred to as a small file. The size of this type of file is relatively small.

[0046] The parameter of the target size can be, for example, bin_capacity, which can be, for example, 128 MB. The parameter and specific value of the target size can be set based on actual conditions, and the present disclosure does not limit this.

[0047] The second type of file refers to a file with a file size greater than the file size of the first type of file, which can be referred to as a large file. The size of this type of file is relatively large.

[0048] In implementation, the writing of the data lake can be monitored in real time, for example, the writing of the Iceberg table is monitored to record the number of files generated and their distribution. For example, the monitored information includes writing time, number of files written, file size, path, and the like. The path can include the partition of the file in the metadata.

[0049] The target time period can be extracted as a set of files to be processed by writing time to determine the first type of file and the second type of file. By monitoring the number of files written, the incremental file corresponding to the number of files written can be determined as a set of files to be processed to determine the first type of file and the second type of file. By monitoring the file size to identify the first type of file and the second type of file, the path can help obtain the first type of file and the second type of file for subsequent processing.

[0050] It can be understood that the set of files to be processed is the monitored written file. For example, Iceberg can include data files and files generated by managing data lakes. The files generated by managing data lakes are illustrated by taking Iceberg as an example. When deleting data based on Iceberg, the specific data is deleted by generating and executing a delete file. For example, the delete file can define which data record to delete, or can define to delete a complete data file.

[0051] S102, the second type of file is split into a plurality of subfiles.

[0052] In implementation, the second type of file can be split according to the target size to obtain a plurality of subfiles. It can be understood that the second type of file is divided according to the target size to obtain a plurality of subfiles, and each subfile obtained is less than or equal to the target size.

[0053] For example, the target size is 128MB, and the size of the second type of file is 288MB. Then, the split subfiles include: subfile 1 is 128MB, subfile 2 is 128MB, and subfile 3 is 32MB.

[0054] S103, determining a set of files to be merged based on the plurality of subfiles and the first type of file.

[0055] The plurality of subfiles can include a subfile of a target size, or can include a subfile smaller than the target size. In the embodiment of the present disclosure, the subfile of the target size does not need to be processed subsequently, so the subfile smaller than the target size is filtered from the plurality of subfiles obtained by splitting, and the filtered subfile and the first type of file are jointly formed into a set of files to be merged.

[0056] S104, performing a merging operation on the set of files to be merged.

[0057] In implementation, the files in the set of files to be merged can come from different partitions, so the merging operation needs to be performed on the files of each partition respectively.

[0058] In addition, in the case that the data lake applicable to the embodiments of the present disclosure includes metadata, the merging operation includes merging the data in the disk and synchronizing the data in the metadata. Specifically, the merging operation is performed on the set of files to be merged in the disk, the metadata is synchronously updated, and the consistency between the metadata level and the underlying data is ensured based on this, so that the underlying data is visible to the outside and the query is consistent.

[0059] In the embodiments of the present disclosure, not only the first type of files (i.e., small files) smaller than the target size are subjected to the merging operation, but also the second type of files (i.e., large files) are subjected to splitting, and the split sub-files will construct a set of files to be merged together with the small files to perform the merging operation. By splitting the large files, the query efficiency can be improved. For example, when performing distributed queries, the split files can be processed in parallel, and different nodes or threads can process different file segments at the same time, thereby improving the query efficiency. If the query only needs part of the fields or part of the records, the split files can more accurately find the relevant data, reducing the amount of data that needs to be scanned. For example, a large file is split into multiple small files according to time or theme, and when querying data in a certain time period, only the relevant files need to be scanned, without scanning the entire large file. For systems that support partitioned or bucketed queries, splitting large files helps better allocate query tasks. For example, data is split into different files according to regions, and when querying data in a certain region, only the corresponding file needs to be queried, thereby improving the query efficiency. After splitting, the large files are also easier to manage. For example, if a large file is damaged or lost, it may be very time-consuming and difficult to recover the entire large file, while small files can be backed up and recovered individually, reducing data loss and recovery time. The metadata management of the split small files is simpler. The metadata can more detailedly describe the content, attributes and location of each small file, facilitating indexing and updating of the data lake. The split small files can be subjected to different storage strategies according to their importance and access frequency. For example, frequently accessed small files are stored in high-performance storage media, and infrequently accessed small files are stored in storage media with lower cost, thereby optimizing the storage cost. In addition, in some cases, the split small files can be more easily subjected to data consistency checks and maintenance than the large files. For example, the small files can be periodically checked to ensure the integrity and correctness of the data. In addition, the present disclosure can effectively improve the query performance by merging the small files. For example, the high I / O (Input / Output) overhead caused by scanning a large number of small files during query is reduced, and the query efficiency is improved. In addition, by merging the small files, the waste of storage resources caused by excessive small files can be reduced. In summary, the scheme proposed based on the present disclosure can realize the automatic merging and optimization strategy of the files, making the maintenance work of the data lake more efficient and stable, reducing the work burden of the management personnel, and improving the overall manageability of the data lake.

[0060] In some embodiments, identifying the first type of file from the set of files to be processed can be implemented as follows: for a first target file in the set of files to be processed, if the first target file is a target type file or meets a file size requirement, it is determined that the first target file belongs to the first type of file; the file size requirement includes that the file size of the first target file is less than a target size, and a first difference between the file size of the first target file and the target size is greater than or equal to a first threshold.

[0061] In implementation, by detecting the write operation of the set of files to be processed, the newly written file is monitored as the first target file.

[0062] The first difference can be a difference between the first target file and the target size, or a ratio of the first target file to the target size.

[0063] Taking the difference as an example, assuming that the target size is q, the first threshold is p1, and the size of the first target file is a, where q is a positive number greater than a. Then the first difference is (q-a), and in the case that the first difference (q-a) is greater than the first threshold p1, it is determined that the first target file is the first type of file, otherwise, it is not the first type of file.

[0064] Taking the ratio as an example, assuming that the target size is q, the first threshold is p1, and the first target file is a, where q is a positive number greater than a. Then the first difference is (q / a), and in the case that the first difference (q / a) is greater than the first threshold p1, it is determined that the first target file is the first type of file, otherwise, it is not the first type of file.

[0065] The target type is a deletion type. Based on the foregoing content, it can be known that in Iceberg, if the content in the data lake is deleted, a corresponding deletion file performs a deletion operation. The deletion file often has a small data amount, i.e., a small file. Therefore, these deletion files also need to be executed as the first type of file for the merging operation.

[0066] In addition, some snapshots are also stored in Iceberg. The snapshots are used to record operation records for the data lake. After the merging operation is completed, the expired snapshots can delete the original files (i.e., the first type of file and the second type of file) before the corresponding merging operation. Therefore, in the embodiments of the present disclosure, the expired snapshots can be implemented as follows: snapshots outside a specified time range starting from the current time are expired to delete the original files before the merging operation. Here, the snapshots within the specified time range are used to facilitate checking and rolling back in case of problems.

[0067] The specified time range can be set based on actual conditions, such as the last 5 days, which is not limited in the present disclosure.

[0068] In some embodiments, as shown in FIG. 2, a flowchart of identifying the first type of file in the embodiments of the present disclosure is shown, which includes the following steps: Figure 2

[0069] S201, in the case of monitoring that file A is written, file A is determined as the first target file;

[0070] S202, it is judged whether file A is a target type of file, in the case of determining that file A is a target type of file, S203 is executed; in the case of determining that file A is not a target type, S204 is executed;

[0071] S203, it is determined that file A is a first type of file;

[0072] S204, it is judged whether the file size of file A is less than the target size, in the case of determining that it is less than the target size, S205 is executed; if it is determined that it is not less than the target size, S206 is executed;

[0073] S205, it is judged whether the first difference between the file size of the first target file and the target size is greater than or equal to the first threshold, if it is greater than or equal to the first threshold, S203 is executed; if the first difference is less than the first threshold, S206 is executed;

[0074] S206, it is determined that file A is not a first type of file.

[0075] In another embodiment, it can also be preferred to determine whether the first target file is less than the target size, and then to determine whether the first target file is a target type, and the two judgment conditions can also be judged in parallel, and any condition meeting can determine that the first target file is a first type of file.

[0076] In the embodiments of the present disclosure, by positioning the first type of file based on the target type, the small files that need to be merged can be directly identified. In addition, based on the file size requirement and the difference threshold requirement, only the small files with large difference are processed, which can effectively identify the small files with small size, so as to reasonably perform the file merging operation.

[0077] In some embodiments, identifying the second type of file from the set of files to be processed can be implemented as follows: for a second target file in the set of files to be processed, the file size of the second target file is obtained; in the case that the file size of the second target file is greater than the target size, and the second difference between the file size of the second target file and the target size is greater than or equal to the second threshold, the second target file is determined as a candidate file, and a candidate file set is obtained; the candidate file set is filtered to obtain the second type of file. ​

[0078] The second difference degree is used to measure whether the second target file is a large file. Moreover, in the embodiments of the present disclosure, a suitable large file is screened out as the second type file through a splitting condition, and subsequent splitting and merging operations are performed. The splitting condition can be screened based on actual conditions, for example, a large file that is stable and will not change in the long term, and that can significantly help improve query efficiency and management performance after splitting according to the management performance of the data lake. Instead of treating all large files equally.

[0079] In some embodiments, the process of obtaining the candidate file can be described as shown in Figure 3

[0080] S301, in the case where it is monitored that the file B is written, the file B is determined as a second target file;

[0081] S302, it is judged whether the file size of the file B is greater than a target size, in the case where it is determined that the file size is not greater than the target size, S303 is executed, and in the case where it is determined that the file size is greater than the target size, S304 is executed;

[0082] S303, it is determined that the file B is not a candidate file;

[0083] S304, it is judged whether a second difference degree between the file size of the second target file and the target size is greater than or equal to a second threshold value, if the second difference degree is greater than or equal to the second threshold value, S305 is executed, and if the second difference degree is less than the second threshold value, S303 is executed;

[0084] S305, it is determined that the file B is a candidate file, and a candidate file set is formed based on at least one candidate file;

[0085] S306, a candidate file that meets a splitting condition is screened out from the candidate file set, and a second type file is obtained.

[0086] In the embodiments of the present disclosure, based on the requirements of the file size and the difference degree threshold value, only the file that meets the requirements of the second difference degree and the splitting condition is processed, which can effectively identify the large file, and help improve the efficiency and effect of the later file merging.

[0087] In some embodiments, the candidate file that meets the splitting condition is screened out from the candidate file set, and the second type file is obtained, as shown in Figure 4

[0088] S401, based on the file size of the candidate file, the candidate file in the candidate file set is scored, and a first score value of the candidate file is obtained.

[0089] The first score value is used to measure the necessity of splitting from the dimension of the file size. ​​

[0090] In some embodiments, the candidate files in the candidate file set are scored based on the file size of the candidate files to obtain a first score value of the candidate files, which can be implemented specifically as follows: a first score value range corresponding to the file size of the candidate files is determined; a first preset value corresponding to the first score value range is obtained to obtain the first score value of the candidate files; wherein the larger the file size is, the larger the corresponding first score value is.

[0091] In implementation, the corresponding first score value can be gradually reduced based on the gradual reduction of the file size, and an example of the first score value range of the candidate files is shown in Table 1:

[0092] First score range First score <128MB 0 128MB ~ 144MB 1 144MB ~ 512MB 2 512MB ~ 1GB 6 1GB ~ 2GB 8 > 2GB 10

[0093] Table 1

[0094] When the target size is 128 MB, files smaller than the target size are small files, which are not processed and are assigned a first score value of 0. When the target size is 128 MB to 144 MB, it is determined that the file size does not exceed the target size much, and the file is assigned a first score value of 1. Similarly, as the file size of the candidate file gradually increases, the first score value also increases.

[0095] The specific numerical values in the table are used for illustrative purposes, and the embodiments of the present disclosure are not limited in this regard.

[0096] In the embodiments of the present disclosure, the contribution of the file size to the splitting value is quantified, so that the process of screening the second type of file is more scientific and reasonable. Larger file size may mean that more significant performance improvement and storage optimization effect can be brought after merging, so that this type of file is preferentially selected for splitting and merging, which helps to achieve better resource utilization and query performance.

[0097] S402, scoring the candidate files based on the update time of the candidate files to obtain a second score value of the candidate files.

[0098] The second score value is used to measure the necessity of splitting from the dimension of the update time of the file.

[0099] In some embodiments, the candidate files are scored based on the update time of the candidate files to obtain a second score value of the candidate files, which can be implemented specifically as follows: a second score value range corresponding to the update time of the candidate files is determined; a second preset value corresponding to the second score value range is obtained to obtain the second score value of the candidate files; wherein the smaller the update time is, the larger the corresponding second score value is.

[0100] In implementation, the smaller the update time is, the longer the update time of the file is from the present, i.e., the file is not updated for a long time, and the larger the second score is assigned to the file. The larger the update time is, the shorter the update time of the file is from the present, i.e., the file is frequently operated, and the smaller the second score is assigned to the file. Thus, the file that is stable for a long time will be stably retained after splitting, and the consumption of processing resources caused by frequent changes is reduced.

[0101] An example of the second score range of the candidate file is shown in Table 2.

[0102] Update time Second score <= 1 day 0 1 day ~ 7 days 1 7 days ~ 30 days 2 > 30 days 5

[0103] Table 2

[0104] In the table, the file with an update time less than or equal to 1 day is a frequently operated file, and the file is determined as an active file, and the score of the file is set to 0. The shorter the update time is, the larger the score is.

[0105] The specific numerical values in Table 2 are used for illustrative purposes, and the embodiments of the present disclosure are not limited thereto.

[0106] In the embodiments of the present disclosure, the file with the smaller update time is a file that is not updated for a long time, and the file is stable and less likely to be modified, and is suitable for merging. The scoring mechanism based on the update time helps to improve the timeliness and effectiveness of file merging.

[0107] It should be noted that the execution order of S401 and S402 is not limited.

[0108] S403, fusing the first score and the second score of the candidate file to obtain a split value of the candidate file.

[0109] In implementation, the sum of the first score and the second score can be used as the split value.

[0110] In addition, the first score and the second score can be weighted and summed to obtain the split value. In the case where the file size is considered more important, the first score can be assigned a larger weight, and the second score can be assigned a smaller weight.

[0111] S404, filtering out the candidate file with a split value greater than or equal to a preset score from the candidate file set to obtain a second type of file.

[0112] For example, the preset score can be selected based on the actual situation, and the embodiments of the present disclosure are not limited thereto.

[0113] In some embodiments, the identification of the candidate file with a split value less than the preset score is recorded, and in the next merging process, the candidate file can be used as the second type of file for subsequent splitting operation.

[0114] In the embodiments of the present disclosure, the second type of files screened out are more suitable for splitting and merging operations from the aspects of file size and update time, which can improve the stability of the processing result of large files and reasonably improve the query performance and management quality of the data lake.

[0115] In the embodiments of the present disclosure, the merging operation on the set of files to be merged can be implemented as follows:

[0116] In the case of meeting at least one of the following merging conditions, it is determined to perform a merging operation on the set of files to be merged:

[0117] 1) The first preset statistical indicator of the total number of files in the set of files to be merged meets the first merging condition.

[0118] The first preset statistical indicator can be the number of total files in the set of files to be merged, or the ratio of the number of newly added small files to the total number of files.

[0119] In the case where the first preset statistical indicator can be the number of total files in the set of files to be merged, the first merging condition is that the number of total files in the set of files to be merged is greater than the first quantity threshold. In implementation, in the case where the number of total files in the set of files to be merged is greater than the first quantity threshold, the merging operation is triggered.

[0120] In the case where the first preset statistical indicator can be the ratio of the number of newly added files in the set of files to be merged to the total number of files, the first merging condition is that the ratio of the number of newly added files in the set of files to be merged to the total number of files is greater than the first proportion. In implementation, in the case where the ratio of the number of newly added files in the set of files to be merged to the total number of files is greater than the first proportion, the merging operation is triggered.

[0121] In the embodiments of the present disclosure, the merging operation is triggered in the case where the number of small files is too large, which significantly reduces the storage and retrieval pressure of metadata, and at the same time, the merging of small files can also reduce the overhead of random reading.

[0122] 2) The second preset statistical indicator of the number of files of a target type in the set of files to be merged meets the second merging condition.

[0123] Among the foregoing, the target type is the deletion type. The second preset statistical indicator is the number of files of the target type or the ratio of the number of files of the target type to the total number of files.

[0124] In the case where the second preset statistical indicator is the number of files of the target type, the second merging condition is that the number of files of the target type is greater than the second quantity threshold, and in the case where the number of files of the target type exceeds the second quantity threshold, the merging operation is triggered.

[0125] In a case where the second preset statistical index is a ratio of the number of files of the target type to the total number of files, the second merging condition is that the ratio of the number of files of the target type to the total number of files is greater than a second proportion. In implementation, in a case where the ratio of the number of files of the target type to the total number of files in the file set to be merged is greater than the second proportion, the merging operation is triggered.

[0126] In the embodiments of the present disclosure, the second statistical index of the file of the target type is used to trigger the merging operation, so that the small files of the target type can be processed in time when there are too many small files, thereby reducing the consumption of storage resources and improving the query and management performance.

[0127] 3) A third preset statistical index of the data amount of the metadata file of the file set to be processed meets a third merging condition.

[0128] In implementation, the third merging condition can be that the data amount of the metadata file is greater than a third quantity threshold, in this case, the merging operation is triggered.

[0129] In the embodiments of the present disclosure, the third preset statistical index of the data amount of the metadata file is used to trigger the merging operation, so that the metadata with excessive quantity can be merged in time, and the speed of loading metadata in the query planning stage is improved as much as possible.

[0130] 4) A growth value of a target index of a query request load is greater than a preset baseline value.

[0131] The target index can include at least one of a calculation execution time length growth, a CPU (Central Processing Unit) utilization rate growth, and a calculation memory occupation rate growth.

[0132] In implementation, the query request load can be determined for a target query request. After each execution of the target query request, the historical execution is compared to obtain the above target index.

[0133] 1) The timestamps before and after the target query request are sampled to obtain the calculation execution time length, and when the execution time length exceeds a first proportion, such as 30%, of a first moving average, it is determined to be abnormal. The first moving average can be obtained by averaging the historical execution time lengths of the target query request.

[0134] 2) The CPU ticks before and after the target query request are sampled to obtain the CPU utilization rate, and when the CPU occupation exceeds a second proportion, such as 30%, of a second moving average, it is determined to be abnormal. The first moving average can be obtained by averaging the historical CPU utilization rates of the target query request.

[0135] 3. The memory of the JVM (Java Virtual Machine) before and after the query request is sampled to obtain the change in memory occupation, and the memory occupation exceeding the third moving average by a third proportion, such as 30%, is determined as an exception. The third moving average can be obtained by averaging the historical memory occupation values of the target query request.

[0136] Any of the first moving average, the second moving average, and the third moving average in the above can be the average of N times of historical collection. For example, the average of the first five calculation execution durations can be used as an example.

[0137] In some embodiments, when any two of the target indicators are abnormal, the merging strategy can be triggered. For example, the difference between the last value and the first value is compared with the corresponding moving average value, and when the execution duration increases significantly (such as doubled) or the CPU / memory consumption rises above a threshold, the merging is triggered, and all data is written to the database.

[0138] In the embodiments of the present disclosure, the target indicators based on the query request load are used as the conditions for triggering the merging operation, which can dynamically merge small files based on the query request load in a timely manner to improve the query efficiency.

[0139] In addition, when any of the above conditions is met, the machine load can also be considered to meet the computing resources required for merging, which can be implemented as: when the machine load is less than the task occupation threshold and any of the conditions 1-4 is met, the merging operation is performed. The machine load can include the machine memory and / or CPU task occupation. For example, the task occupation threshold can be 80%.

[0140] In the embodiments of the present disclosure, multiple merging conditions are introduced, so that the merging operation can be dynamically triggered according to the actual situation, and the intelligence and adaptability of the method are enhanced. Unnecessary operations are reduced when the number of files is small or no merging is required, thereby saving computing resources and time cost.

[0141] In some embodiments, the merging operation is performed on the set of files to be merged, such as Figure 5 As shown in the figure, the merging operation includes:

[0142] S501, based on the first grouping strategy, the to-be-allocated file in the set of files to be merged is allocated to the corresponding first target file group; the first grouping strategy includes: after the to-be-allocated file is allocated to the first target file group, the file size of the first target file group is less than or equal to the target size; and among the plurality of candidate file groups, the third difference between the remaining capacity of the first target file group and the file of the to-be-allocated file satisfies a preset condition.

[0143] The preset condition can be that the third difference is minimum. That is, the first target file group can not only accommodate the to-be-assigned file, but also the remaining capacity of the first target file group and the file size of the first file are closest.

[0144] In implementation, the existing candidate file groups are traversed, and the remaining capacities of the candidate file groups are compared with the file size of the to-be-assigned file, to filter out the candidate file group whose third difference meets the preset condition, and determine the candidate file group as the first target file group. It can also be understood that the candidate file group whose file size is closest to the to-be-assigned file is filtered out from the remaining capacities, and the candidate file group is determined as the first target file group.

[0145] The preset condition can be less than a preset gap, or the candidate file groups corresponding to the third difference are sorted, and the candidate file groups ranked in a preset ranking are determined as the first target file group. For example, the candidate file group ranked first can be determined as the first target file group.

[0146] For example, it is assumed that the remaining capacity of the candidate file group 1 is m, and the remaining capacity of the candidate file group 2 is n. In the case that the file size of the to-be-assigned file is k, if m > n > k > 0, it is found through comparison that the remaining capacity of the candidate file group 2 is closest to the file size of the to-be-assigned file, and the candidate file group 2 is determined as the first target file group.

[0147] S502, performing a merging operation on the first target file group to obtain a merged file.

[0148] In the embodiments of the present disclosure, the storage space can be fully utilized, and the frequent merging operation caused by the total file size in the file group being insufficient can be reduced. By reasonably distributing the files to different file groups, the file size of each file group after merging is as close to the target size as possible, thereby improving the storage efficiency, and being also conducive to subsequent query and management operations.

[0149] In some embodiments, the merged file can still have a small size problem, and therefore the file with a small size after merging can be merged again, and the first target file group is merged to obtain a merged file. Figure 6 As shown in the figure, the merged file includes:

[0150] S601, obtaining a first target file group with a merged file size smaller than a size lower limit, to obtain a to-be-optimized file of the first target file group; the size lower limit is smaller than the target size.

[0151] The to-be-optimized file of the first target file group can be a merged file obtained by merging multiple small files in the group, or can be multiple small files in the group, as long as the total file size in the group is smaller than the size lower limit.

[0152] In implementation, there can be file groups that are still small after merging. To reduce the occurrence of such file groups, a size lower limit can be set, and file groups below the size lower limit can be merged again later.

[0153] In implementation, the size lower limit can be represented by the following parameters:

[0154] min_bin_fill represents the minimum capacity requirement of the target file group, i.e., the size lower limit of the target file group. A value below this limit will be considered as not meeting the requirement. For example, it can be set to 100 MB.

[0155] In implementation, based on the aforementioned min_bin_fill, if the size of any file group is less than min_bin_fill, then the group is determined as the first target file group.

[0156] S602, match the to-be-optimized file with the merged files in the plurality of first target file groups to obtain a second target file group meeting a second grouping strategy; the second grouping strategy includes: after the to-be-optimized file is assigned to the second target file group, the file size of the second target file group is less than or equal to a size upper limit; the size upper limit is greater than the target size.

[0157] In implementation, to reduce the problem of excessively large files, a size upper limit can be set so that the size of each file group queried in the query process is appropriate, reducing the occurrence of file groups with excessively large sizes after merging.

[0158] The parameter of the size upper limit can be set as max_bin_overflow, representing the maximum excess of the merged files. For example, it can be set to 16 MB. In combination with the aforementioned ideal capacity of the target file group, it indicates that the size upper limit of the target capacity group is 144 MB.

[0159] For example, the to-be-optimized file 1 is t, and the merged file in the first target file group 1 is h. After comparison, it is found that (t+h) is less than the size upper limit of the target capacity group, i.e., the merging of the to-be-optimized file 1 and the first target file group 1 does not exceed the size upper limit. Therefore, the first target file group 1 can be determined as the second target file group.

[0160] S603, merge the to-be-optimized file with the second target file group to obtain a merged file of the second target file group.

[0161] Based on the aforementioned example, the to-be-optimized file 1 is merged into the second target file group, and a merged file of the second target file group is obtained.

[0162] In the embodiments of the present disclosure, by reassigning the first target file group with too small size to other suitable file groups for re-merging, the file sizes of all file groups can meet certain requirements as much as possible, thereby further improving the storage efficiency and query performance, reducing the problem of affecting the overall optimization effect due to too small file size, and improving the rationality and efficiency of the merged file.

[0163] In some embodiments, the file group with a file size not less than the size lower limit is directly subjected to a merging operation on the files in the group to facilitate subsequent user query.

[0164] For ease of understanding, the overall flowchart of the file intelligent merging method proposed in the embodiments of the present disclosure is shown in Figure 7 as follows, including:

[0165] S701, identifying first type files and second type files from a set of files to be processed.

[0166] S702, splitting the second type files according to a target size to obtain a plurality of sub-files, and screening sub-files smaller than the target size from the plurality of sub-files.

[0167] In the embodiments of the present disclosure, the related parameters can be flexibly set according to requirements. For example, for an Iceberg table, the parameters can be configured as shown in Table 3:

[0168]

[0169] Table 3

[0170] For “max-concurrent-file-group-rewrites”, the corresponding value is the number of threads, which is used to define the concurrent execution of the file merging operation in the embodiments of the present disclosure.

[0171] The value corresponding to “min-input-files” indicates that the number of files in the group does not exceed the value.

[0172] “target-file-size-bytes” represents the target size, and “min-file-size-bytes” represents the size of the file that needs to be merged based on the target size. If the size is less than the size, it is rewritten, that is, considered as a first type file, and then subjected to subsequent merging processing.

[0173] “max-file-size-bytes” represents the maximum size of the file. If the size is greater than the size, it indicates that it may need to be rewritten. The second type file can be screened from the large file greater than the size based on the screening method of the second type file described above, and then subjected to subsequent splitting processing and merging operation.

[0174] In addition, the files belonging to the target type also belong to the first type of files and will be subjected to subsequent merging processing. The target type is a deletion data file type, which specifically covers file types such as location-based deletion files, equality-based deletion files, manifest files, and manifest list files.

[0175] Among them, the deletion data file type is used to delete the entire data file.

[0176] Among them, the location-based deletion file is used to delete the file at a specified location; the location can be represented by a path.

[0177] Among them, the equality-based deletion file can be understood as filtering files according to predetermined equality conditions (such as time, capacity, reference state, etc.) and counting and deleting data that meets these conditions.

[0178] Among them, the manifest file records deletion markers when deleting data. These markers indicate which specific records in the data file need to be deleted.

[0179] Among them, the manifest list file defines the current snapshot of the table. After deleting data, Iceberg creates a new snapshot and updates the manifest list file to point to the new set of manifest files. This type of deletion file is used to process the manifest list file.

[0180] Since there can be multiple files under each deletion file type, the number of each deletion file can be counted to determine whether to trigger the file merging operation. As shown in Table 4, the expression of the statistical results is shown:

[0181] Serial number Name Explanation 1 deleted_data_files_count Number of deleted data files 2 deleted_position_delete_files_count Number of deleted position-based delete files 3 deleted_equality_delete_files_count Number of deleted equality-based delete files 4 deleted_manifest_files_count Number of deleted manifest files 5 deleted_manifest_lists_count Number of deleted manifest list files

[0182] Table 4

[0183] Among them, “deleted_data_files_count” in Table 4 represents the number of deletion files of the deletion data file type.

[0184] “deleted_position_delete_files_count” in Table 4 represents the number of location-based deletion files.

[0185] “deleted_equality_delete_files_count” in Table 4 represents the number of equality-based deletion files.

[0186] The "deleted_manifest_files_count" in sequence number 4 in Table 4 is the number of deleted files for the manifest file;

[0187] The manifest_lists in sequence number 5 in Table 4 is the number of deleted files for the manifest_list.

[0188] S703, merging the screened sub-file and the first type file into a to-be-merged file set;

[0189] S704, judging whether the to-be-merged file set meets the merging condition, if yes, executing S705; if not, executing S701 in the next period;

[0190] S705, judging whether the machine load is less than the task occupation threshold, if yes, executing S706; if not, executing S701;

[0191] S706, grouping the to-be-merged file set based on the first grouping strategy and the second grouping strategy, and executing the merging operation for each group in parallel to obtain a merged file;

[0192] S707, rewriting the disk based on the merged file and updating the metadata;

[0193] S708, performing an expiration operation on the snapshot file beyond the specified time range, updating the metadata, and ending the merging operation.

[0194] In summary, the file intelligent merging method proposed in the embodiments of the present disclosure has the following effects:

[0195] 1. Improving query performance

[0196] The problem of scanning a large number of small files during query is reduced, the I / O overhead is reduced, and the execution speed of SQL query engines such as Trino (an open source distributed SQL query engine), Presto (a distributed SQL query engine), ClickHouse (a high-performance columnar OLAP database), etc. is accelerated.

[0197] 2. Reducing storage cost

[0198] Reduce the storage request and metadata management overhead caused by too many small files in HDFS (Hadoop Distributed File System) / OSS (Object Storage Service) / GCS (Google Cloud Storage), improve storage space utilization, and reduce data lake storage costs.

[0199] 3. Optimize streaming data processing

[0200] Reduces the problem of small file accumulation when Flink (Apache Flink, an open source distributed stream processing framework), Kafka (Apache Kafka, a distributed stream data platform), and other streaming data are written to Iceberg, and can adapt to Flink+Iceberg to make streaming computing stable and efficient.

[0201] 4. Accelerate ETL (Extract, Transform, Load, standardized process of data extraction, transformation, and loading) / data warehouse task execution

[0202] Reduces the problem of too many small files affecting Spark (Apache Spark, a general distributed computing engine) / Hive (Apache Hive, a data warehouse tool) / Flink data conversion and merging operations, and improves the efficiency of Lakehouse batch processing tasks.

[0203] 5. Enhance enterprise data governance capabilities

[0204] Reduce Iceberg's excessive Snapshot by intelligent merging, optimize data management, and can be applied to data compliance, data audit, and lifecycle management requirements.

[0205] 6. Improve OLAP (Online Analytical Processing) performance

[0206] Reduces the problem of excessive metadata management burden when querying OLAP data lake.

[0207] Based on the same technical concept, the present disclosure also proposes a file intelligent merging device 800, as shown in Figure 8 , comprising:

[0208] An identification module 801 is configured to identify a first type of file and a second type of file from a set of files to be processed, the file size of the first type of file being less than a target size, and the file size of the second type of file being greater than the file size of the first type of file.

[0209] The splitting module 802 is configured to split the second type of file into a plurality of sub-files.

[0210] The determining module 803 is configured to determine a set of to-be-merged files based on the plurality of sub-files and the first type of file.

[0211] The merging module 804 is configured to perform a merging operation on the set of to-be-merged files.

[0212] In some embodiments, the identifying module comprises:

[0213] The first determining unit is configured to determine, for a first target file in the set of to-be-processed files, that the first target file belongs to the first type of file if the first target file is a target type file or meets a file size requirement.

[0214] The file size requirement comprises that a file size of the first target file is less than the target size, and a first difference degree between the file size of the first target file and the target size is greater than or equal to a first threshold value.

[0215] In some embodiments, the identifying module comprises:

[0216] The obtaining unit is configured to obtain, for a second target file in the set of to-be-processed files, a file size of the second target file.

[0217] The second determining unit is configured to determine, for a case where the file size of the second target file is greater than the target size, and a second difference degree between the file size of the second target file and the target size is greater than or equal to a second threshold value, the second target file as a candidate file, to obtain a set of candidate files.

[0218] The screening unit is configured to screen, from the set of candidate files, a candidate file that meets a splitting condition, to obtain the second type of file.

[0219] In some embodiments, the screening unit comprises:

[0220] The first scoring sub-unit is configured to score, based on a file size of a candidate file, the candidate file in the set of candidate files, to obtain a first score value of the candidate file; and

[0221] The second scoring sub-unit is configured to score, based on an update time of the candidate file, the candidate file, to obtain a second score value of the candidate file.

[0222] The fusion sub-unit is configured to fuse the first score value and the second score value of the candidate file, to obtain a splitting value of the candidate file.

[0223] The score screening subunit is configured to screen candidate files with a split value greater than or equal to a preset score from the candidate file set, and obtain the second type of files.

[0224] In some embodiments, the first scoring subunit is specifically configured to:

[0225] determine a first score range corresponding to a file size of the candidate file;

[0226] obtain a first preset value corresponding to the first score range, and obtain a first score of the candidate file;

[0227] wherein the larger the file size, the larger the first score.

[0228] In some embodiments, the second scoring subunit is specifically configured to:

[0229] determine a second score range corresponding to an update time of the candidate file;

[0230] obtain a second preset value corresponding to the second score range, and obtain a second score of the candidate file;

[0231] wherein the smaller the update time, the larger the second score.

[0232] In some embodiments, the merging module comprises:

[0233] In the case of meeting at least one of the following merging conditions, it is determined to perform a merging operation on the set of files to be merged:

[0234] 1) A first preset statistical indicator of the total number of files in the set of files to be merged meets a first merging condition;

[0235] 2) A second preset statistical indicator of the number of files of a target type in the set of files to be merged meets a second merging condition;

[0236] 3) A third preset statistical indicator of the data amount of the metadata file of the set of files to be processed meets a third merging condition;

[0237] 4) The growth value of a target indicator of the query request load is greater than a preset baseline value.

[0238] In some embodiments, the merging module comprises:

[0239] The allocation unit is configured to allocate, based on a first grouping policy, a to-be-allocated file in the set of to-be-merged files into a corresponding first target file group; the first grouping policy comprises: after the to-be-allocated file is allocated into the first target file group, a file size of the first target file group is less than or equal to the target size; among a plurality of candidate file groups, a third difference degree between a remaining capacity of the first target file group and a file size of the to-be-allocated file satisfies a preset condition;

[0240] The merging unit is configured to perform a merging operation on the first target file group to obtain a merged file.

[0241] In some embodiments, the merging unit comprises:

[0242] The obtaining subunit is configured to obtain a first target file group whose file size after merging is less than a size lower limit, to obtain a to-be-optimized file of the first target file group; the size lower limit is less than the target size.

[0243] The matching subunit is configured to match the to-be-optimized file with merged files in a plurality of first target file groups to obtain a second target file group that satisfies a second grouping policy; the second grouping policy comprises: after the to-be-optimized file is allocated into the second target file group, a file size of the second target file group is less than or equal to a size upper limit; the size upper limit is greater than the target size.

[0244] The merging subunit is configured to merge the to-be-optimized file with the second target file group to obtain a merged file of the second target file group.

[0245] The specific functions and examples of the modules and submodules of the apparatuses in the embodiments of the present disclosure are described above in the related description of the corresponding steps in the method embodiments, and will not be described here.

[0246] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0247] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0248] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0249] As shown in Figure 9 The device 900 includes a computing unit 901 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0250] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0251] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the file intelligent merging method. For example, in some embodiments, the file intelligent merging method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the file intelligent merging method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the file intelligent merging method by any other appropriate means, such as by means of firmware.

[0252] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0253] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0254] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0255] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0256] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0257] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0258] It should be understood that the various forms of flow shown above can be re-ordered, steps added or removed, etc. For example, the steps recited in the present disclosure can be performed in parallel, in series, in a different order, etc., so long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0259] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An intelligent file merging method, comprising: identifying a first type of file and a second type of file from a set of to-be-processed files, the file size of the first type of file being less than a target size, and the file size of the second type of file being greater than the file size of the first type of file; splitting the second type of file into a plurality of sub-files; determining a set of to-be-merged files based on the plurality of sub-files and the first type of file; performing a merging operation on the set of to-be-merged files.

2. The method of claim 1, wherein, Identifying the first type of file from the set of to-be-processed files, comprising: for a first target file in the set of to-be-processed files, if the first target file is a target type of file or meets a file size requirement, determining that the first target file belongs to the first type of file; the file size requirement comprises: the file size of the first target file is less than the target size, and a first difference between the file size of the first target file and the target size is greater than or equal to a first threshold.

3. The method of claim 1, wherein, Identifying the second type of file from the set of to-be-processed files, comprising: for a second target file in the set of to-be-processed files, obtaining the file size of the second target file; if the file size of the second target file is greater than the target size, and a second difference between the file size of the second target file and the target size is greater than or equal to a second threshold, determining the second target file as a candidate file to obtain a set of candidate files; screening a candidate file that meets a splitting condition from the set of candidate files to obtain the second type of file.

4. The method of claim 3, wherein, The screening of the candidate file that meets the splitting condition from the set of candidate files to obtain the second type of file, comprising: scoring the candidate file in the set of candidate files based on the file size of the candidate file to obtain a first score value of the candidate file; and scoring the candidate file based on the update time of the candidate file to obtain a second score value of the candidate file; fusing the first score value and the second score value of the candidate file to obtain a splitting value of the candidate file; screening a candidate file with a splitting value greater than or equal to a preset score value from the set of candidate files to obtain the second type of file.

5. The method of claim 4, wherein, The scoring of the candidate file in the set of candidate files based on the file size of the candidate file to obtain the first score value of the candidate file, comprising: determining a first score value range corresponding to the file size of the candidate file; obtaining a first preset value corresponding to the first score value range to obtain the first score value of the candidate file; wherein the larger the file size, the larger the corresponding first score value.

6. The method of claim 4, wherein, The scoring of the candidate file based on the update time of the candidate file to obtain the second score value of the candidate file, comprising: determining a second score value range corresponding to the update time of the candidate file; obtaining a second preset value corresponding to the second score value range to obtain the second score value of the candidate file; wherein the smaller the update time, the larger the corresponding second score value.

7. The method of any one of claims 1-6, wherein, The merging operation on the set of to-be-merged files, comprising: in the case that at least one of the following merging conditions is met, determining to perform a merging operation on the set of to-be-merged files: 1) a first preset statistical index of a total number of files in the set of files to be merged meets a first merging condition; 2) a second preset statistical index of a number of files of a target type in the set of files to be merged meets a second merging condition; 3) a third preset statistical index of a data volume of a metadata file of the set of files to be processed meets a third merging condition; 4) a growth value of a target index of a query request load is greater than a preset baseline value.

8. The method of claim 1 or 7, wherein, The merging operation on the set of files to be merged includes: based on a first grouping strategy, assigning a file to be assigned in the set of files to be merged to a corresponding first target file group; the first grouping strategy includes: after the file to be assigned is assigned to the first target file group, a file size of the first target file group is less than or equal to the target size; among a plurality of candidate file groups, a third difference between a remaining capacity of the first target file group and a file size of the file to be assigned meets a preset condition; performing a merging operation on the first target file group to obtain a merged file.

9. The method of claim 8, wherein, The merging operation on the first target file group to obtain a merged file includes: obtaining a first target file group whose file size after merging is less than a size lower limit, to obtain a file to be optimized of the first target file group; the size lower limit is less than the target size; matching the file to be optimized with a plurality of merged files in the first target file group to obtain a second target file group that meets a second grouping strategy; the second grouping strategy includes: after the file to be optimized is assigned to the second target file group, a file size of the second target file group is less than or equal to a size upper limit; the size upper limit is greater than the target size; merging the file to be optimized with the second target file group to obtain a merged file of the second target file group.

10. A file intelligent merging device, comprising: an identification module configured to identify a first type of file and a second type of file from a set of files to be processed, a file size of the first type of file being less than a target size, and a file size of the second type of file being greater than the file size of the first type of file; a splitting module configured to split the second type of file into a plurality of sub-files; a determination module configured to determine a set of files to be merged based on the plurality of sub-files and the first type of file; a merging module configured to perform a merging operation on the set of files to be merged.

11. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-9.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-9.

Images