Method, system, device and computer-readable medium for file merging

Abstract

The present invention discloses a method, system, device, and computer-readable medium for file merging, and relates to the field of computer technology. A specific implementation of the method includes: receiving a merge operation pushed by an initiating node, the merge operation including a computing node identifier, multiple file identifiers, and a merge file identifier, the initiating node being a computing node in a distributed file storage system that initiates the merge operation; creating a file merge task based on the computing node identifier, the multiple file identifiers, and the merge file identifier; placing the text merge task into a public execution queue, so that the computing nodes in the distributed file storage system, upon detecting changes in the public execution queue, update the execution queue according to the public execution queue, and execute file merging based on the execution queue. This implementation can reduce CPU and disk IO consumption and improve storage space utilization.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, device and computer-readable medium for merging files. Background Art

[0002] The LSM tree (Log-Structured Merge Tree) storage engine is a hierarchical, ordered, disk-oriented data structure. Its core concept is to fully utilize the fact that sequential batch writes on disks have much higher performance than random writes. This principle is used in design and optimization to achieve optimal write performance. Furthermore, through regular merges and reductions, invalid data is effectively removed, read paths are shortened, and disk space utilization is improved.

[0003] During the implementation of this invention, the inventors discovered that the existing technology has at least the following problems: Many databases with high write throughput requirements rely on LSM-Trees. However, merge operations are very CPU-intensive and disk I / O-intensive, especially during peak business hours, which reduces overall system throughput. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, system, device, and computer-readable medium for file merging, which can reduce CPU and disk IO consumption and improve storage space utilization.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for merging files is provided, comprising:

[0006] Receive a merge operation pushed by an initiating node, the merge operation including a computing node identifier, multiple file identifiers, and a merged file identifier, the initiating node being a computing node that initiates the merge operation in the distributed file storage system;

[0007] Creating a file merging task based on the computing node identifier, the multiple file identifiers, and the merged file identifier;

[0008] The text merging task is placed in a public execution queue so that the computing nodes in the distributed file storage system can update the execution queue according to the public execution queue when monitoring changes in the public execution queue, and perform file merging based on the execution queue.

[0009] The files corresponding to the multiple file identifiers are all files with a data volume smaller than a preset file size.

[0010] The receiving of the merge operation pushed by the initiating node includes:

[0011] Receive multiple merge operations pushed by multiple initiating nodes;

[0012] Placing the text merging task into a public execution queue includes:

[0013] determining validity of the plurality of merge operations based on the file identifier and the time when the merge operation was sent;

[0014] The valid merge operations are placed in the public execution queue according to the merge operation time.

[0015] The files corresponding to the multiple file identifiers belong to the same data partition and are adjacent.

[0016] When a computing node in the distributed file storage system monitors a change in the public execution queue, the computing node updates the execution queue according to the public execution queue, including:

[0017] The public execution queue is located on the coordination server, and the computing node in the distributed file storage system receives a notification message from the coordination server and monitors changes in the public execution queue;

[0018] The execution queue is updated according to the public execution queue.

[0019] The performing file merging based on the execution queue includes:

[0020] Other computing nodes in the distributed file storage system query the shared storage based on the execution queue to obtain the merged file, and load the merged file into the cache to perform file merging.

[0021] The performing file merging based on the execution queue includes:

[0022] The initiating node performs file merging in a cache based on the execution queue.

[0023] According to a second aspect of an embodiment of the present invention, a system for merging files is provided, including:

[0024] An initiating node, configured to push a merge operation, wherein the merge operation includes a computing node identifier, multiple file identifiers, and a merged file identifier. The initiating node is a computing node in the distributed file storage system that initiates the merge operation.

[0025] The coordination server receives the merge operation pushed by the initiating node, creates a file merge task based on the computing node identifier, the multiple file identifiers, and the merged file identifier, and places the text merge task into a public execution queue;

[0026] The computing node is configured to update the execution queue according to the public execution queue when monitoring changes in the public execution queue, and perform file merging based on the execution queue.

[0027] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for merging files, including:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method.

[0031] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0032] One embodiment of the above invention has the following advantages or beneficial effects: receiving a merge operation pushed by an initiating node, the merge operation including a computing node identifier, multiple file identifiers, and a merge file identifier, the initiating node being a computing node in a distributed file storage system that initiates the merge operation; creating a file merge task based on the computing node identifier, the multiple file identifiers, and the merge file identifier; placing the text merge task into a public execution queue so that the computing nodes in the distributed file storage system, upon monitoring changes in the public execution queue, update the execution queue according to the public execution queue and execute file merges based on the execution queue. In a distributed file storage system, computing nodes implement file merges through queues, thereby greatly reducing CPU and disk IO consumption and improving storage space utilization.

[0033] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0035] Figure 1 This is a schematic diagram of the main flow of a method for merging files according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of an application scenario of file merging according to an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of a process of processing multiple merge operations by a coordination server according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a process for updating an execution queue according to a public execution queue according to an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a computing node executing file merging according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the main structure of a file merging system according to an embodiment of the present invention;

[0041] Figure 7 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0042] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] Many databases with high write throughput requirements rely on LSM-Trees. File merging consumes significant CPU and disk I / O, impacting system throughput.

[0045] In order to solve the problem of large CPU and disk IO consumption by file merging, the following technical solution in the embodiment of the present invention can be adopted.

[0046] See also Figure 1 , Figure 1 This is a schematic diagram of the main process of the method for file merging according to an embodiment of the present invention. The computing nodes in the distributed file storage execute file merging through queues. Figure 1 As shown, it specifically includes the following steps: Figure 1 In the technical solution, the coordination server is the execution entity of each step.

[0047] S101. Receive a merge operation pushed by an initiating node, where the merge operation includes a computing node identifier, multiple file identifiers, and a merged file identifier. The initiating node is a computing node that initiates the merge operation in a distributed file storage system.

[0048] Distributed file storage systems distribute large amounts of data across different computing nodes, significantly reducing the risk of data loss. Distributed file storage systems offer redundancy, meaning failures on some computing nodes do not affect overall operation. Furthermore, even if data stored on a failed computer is corrupted, it can be restored by other computing nodes. Furthermore, computers outside the cluster can be added to the distributed file storage system and share data with minimal configuration, making it highly scalable.

[0049] See also Figure 2 , Figure 2 2 is a schematic diagram of an application scenario of file merging according to an embodiment of the present invention. Figure 2 It is a shared storage-based LSM engine database. The computing nodes are separated from the shared storage, and the computing nodes can be expanded or reduced horizontally in seconds.

[0050] Because shared storage is used, data does not need to be redistributed, and the storage nodes, i.e., the distributed file storage system, can be expanded infinitely. The coordination server provides distributed coordination task services, including distributed merging.

[0051] The coordination server is the manager of the cluster, monitoring the status of each node in the cluster and making appropriate decisions based on the feedback submitted by the nodes. Ultimately, it provides users with a simple and easy-to-use interface and a high-performance, stable system.

[0052] Figure 2 In the LSM architecture, instructions for reading and / or writing files are processed by compute nodes, which then perform read or write operations on files stored in shared storage. In other words, files are stored on shared storage, and every compute node can see them. Compute nodes initiate merge plans to perform file merges. The file merging process is called the LSM engine.

[0053] In an embodiment of the present invention, a computing node can initiate a merge operation after receiving an instruction to read and / or write a file. The computing node that initiates the merge operation is referred to as an initiating node. That is, the initiating node is the computing node that initiates the merge operation in the distributed file storage system. In a distributed file storage system, files are stored in shared storage. That is, files are stored in shared storage in the distributed file storage system.

[0054] In one embodiment of the present invention, an initiating node initiates a merge operation for files with small data volumes to execute the merge plan. That is, the files corresponding to the multiple file identifiers are all smaller than a preset file data volume. The preset file data volume can be preset based on actual application scenarios. In other words, in response to instructions to read and / or write files, the initiating node initiates a merge operation for files with smaller than the preset file data volume.

[0055] This is because, when performing a shared storage query, scanning a large number of small files significantly reduces query speed. Furthermore, to improve query speed, it's necessary to merge small files. In this embodiment of the present invention, a preset file size is used to measure small files. That is, files smaller than the preset file size are eligible for file merging.

[0056] In order to implement the merge operation, the initiating node needs to push the merge operation to the coordination server, and then the file merge is performed through the coordination server.

[0057] The following is an illustrative description of the parameters included in a merge operation. A merge operation includes a node identifier, multiple file identifiers, and a merged file identifier. The node identifier is the identifier of the initiating node. Multiple file identifiers refer to multiple file identifiers to be merged. The merged file identifier is the identifier of the merged file. As an example, the node identifier is CH1; the multiple file identifiers are 202001_0_0_0 and 202001_0_0_1; and the merged file identifier is 202001_0_1_1.

[0058] In the embodiment of the present invention, considering that the files are stored in a shared storage, in order to facilitate file merging, the files corresponding to the multiple file identifiers belong to the same data partition and are adjacent. In other words, the files to be merged are in the same data partition and are adjacent files.

[0059] Compute and storage are separated, making compute nodes stateless and shared storage scalable horizontally and infinitely. All compute nodes share the same data, and the data corresponding to each file is stored in the shared storage. The file directory format is partitionId_minBlock_maxBlock_level. PartitionId is the data partition identifier; minBlock is the minimum block identifier; maxBlock is the maximum block identifier; and level is the number of merges.

[0060] The text directory format stores user data in columns and includes some query-aiding data. Each time a user writes data, a new directory is generated. A background thread continuously merges files on disk to reduce storage overhead. Each merge generates a new file, and the old files are cleared by the background. To speed up queries, compute nodes cache files. Newly generated files must be added to the compute node cache before they are visible to users.

[0061] S102: Create a file merging task based on the computing node identifier, multiple file identifiers, and the merged file identifier.

[0062] The coordination server is used to receive the merge operation and then create a file merge task based on the computing node identifier, multiple file identifiers and the merge file identifier. It can be understood that the coordination server receives the merge operation sent by one or more initiating nodes.

[0063] When the coordination server receives a merge operation sent by an initiating node, it can directly create a file merge task based on the computing node identifier, multiple file identifiers and the merge file identifier, and put the text merge task into the public execution queue.

[0064] S103. Put the text merging task into the public execution queue, so that the computing nodes in the distributed file storage system can update the execution queue according to the public execution queue when monitoring the changes in the public execution queue, and perform file merging based on the execution queue.

[0065] The coordinating server receives merge operations from multiple initiating nodes. This means that multiple initiating nodes push multiple merge operations to the coordinating server in parallel. These merge operations must be placed in a common execution queue, and the coordinating server must process multiple merge operations.

[0066] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a coordination server processing multiple merge operations according to an embodiment of the present invention, which specifically includes the following steps:

[0067] S301: Determine the validity of multiple merge operations according to the file identifier and the time when the merge operation is sent.

[0068] When receiving multiple merge operations, the coordination server needs to determine the validity of the merge operations. Specifically, the coordination server determines the validity of the multiple merge operations based on the file identifier and the time when the merge operations were sent.

[0069] Specifically, the coordination server determines whether multiple merge operations conflict based on file identifiers. If the merge operations do not conflict, the multiple coordinated operations are determined to be valid. If the merge operations conflict, the validity of the merge operations can be determined based on the time the merge operations were sent.

[0070] As an example, the file identifier is an identifier of a file to be merged. If the text identifiers of multiple merge operations are the same, or the file identifiers of multiple merge operations correspond to overlapping files, it is determined that the multiple merge operations conflict.

[0071] In the case of determining whether a merge operation conflicts, two merge operations are used to determine whether there is a conflict. It should be noted that the merge operation conflict is due to the involvement of the same file.

[0072] For example, compute node 1 and compute node 2 initiate a merge plan in parallel. Compute node 1 merges file identifiers 201905_0_0_0 and 201905_1_1_0 into 201905_0_1_1. Compute node 2 merges file identifiers 201905_1_1_0 and 201905_3_3_0 into 201905_2_3_1.

[0073] Since the file identifier 201905_1_1_0 in merge operation 1 of computing node 1 is the same as the file identifier 201905_1_1_0 in merge operation 2 of computing node 2, it means that the file to be merged in merge operation 1 is the same as the file to be merged in merge operation 2, so merge operation 1 conflicts with merge operation 2.

[0074] In the case of a merge operation conflict, based on the merge operation time, the merge operation with an earlier merge operation time is determined as a valid merge operation; and the merge operation with a later merge operation time is determined as an invalid merge operation.

[0075] S302: Put the valid merge operations into the public execution queue according to the merge operation time.

[0076] Valid merge operations need to be executed sequentially according to the merge operation time. Specifically, the coordination server puts the valid merge operations into a public execution queue according to the merge operation time.

[0077] exist Figure 3 In the embodiment, the coordination server first determines the validity of multiple merge operations, and then puts the valid merge operations into a public execution queue.

[0078] To perform file merges, compute nodes in a distributed file storage system need to monitor the public execution queue for changes. Upon detecting changes, the compute node updates its execution queue based on the public execution queue, executing the file merge based on the execution queue. The public execution queue is used to record the text merge tasks that need to be executed. The execution queue is the queue in which the compute node itself performs the text merge.

[0079] It can be understood that for the computing nodes in the distributed file storage system, when they detect changes in the public execution queue, they will update the changed content to their own execution queue, and then execute file merges in the computing node to ensure the consistency of the files stored in the computing node and the files stored in the shared storage.

[0080] See also Figure 4 , Figure 4 This is a flow chart of updating an execution queue according to a public execution queue according to an embodiment of the present invention. Specifically, it includes the following steps:

[0081] S401: The public execution queue is located on a coordination server. When a computing node in the distributed file storage system receives a notification message from the coordination server, it monitors changes in the public execution queue.

[0082] To coordinate compute nodes, a public execution queue resides on a coordination server. Compute nodes in the distributed file storage system monitor changes to the public execution queue upon receiving notifications from the coordination server. In other words, when a change occurs in the public execution queue, each compute node is notified of the change by monitoring the event and receiving notifications from the coordination server.

[0083] S402: Update the execution queue according to the public execution queue.

[0084] The computing nodes that monitor the changes in the public execution queue update the execution queue according to the public execution queue to ensure the consistency of the stored files and the shared storage.

[0085] exist Figure 4 In an embodiment of the present invention, a computing node in a distributed file storage system monitors changes in a public execution queue and updates the execution queue to ensure consistency between its own stored files and the shared storage.

[0086] In one embodiment of the present invention, during the file merge process, computing nodes in the distributed file storage system are divided into two categories: initiating nodes and non-initiating nodes. An initiating node is a computing node that initiates the merge operation. A non-initiating node is a node that does not initiate the merge.

[0087] For the initiating node and the non-initiating node, the way of performing file merging based on the execution queue is different.

[0088] For the initiating node, the file merge is performed in the cache based on the execution queue. This is because the initiating node can perform the file merge in its own cache after being processed by the coordination server.

[0089] For non-initiating nodes, that is, other computing nodes in the distributed file storage system, shared storage is queried based on the execution queue, and the merged files are loaded into the cache to perform file merging.

[0090] The non-initiating node does not store the merged file in its cache. The merged file is stored in shared storage. The non-initiating node needs to load the merged file into its own cache to perform the file merge. In other words, other computing nodes in the distributed file storage system query the shared storage based on the execution queue to obtain the merged file and load the merged file into the cache to perform the file merge.

[0091] It should be noted that after the merge operation is performed on the shared storage, the merged file is included in the shared storage.

[0092] In this embodiment of the present invention, a single merge plan is executed by only one compute node, the initiating node, to prevent conflicts. The compute node that does not execute the merge operation waits for the initiating node to complete the execution. It then queries the shared storage based on the execution queue to synchronize data, updates its own cache, and completes the file merge.

[0093] In the above embodiment, a merge operation pushed by an initiating node is received, the merge operation including a computing node identifier, multiple file identifiers, and a merge file identifier, and the initiating node is a computing node in a distributed file storage system that initiates the merge operation; a file merge task is created based on the computing node identifier, the multiple file identifiers, and the merge file identifier; the text merge task is placed in a public execution queue, so that when the computing nodes in the distributed file storage system monitor changes in the public execution queue, they update the execution queue according to the public execution queue and execute file merges based on the execution queue. In a distributed file storage system, computing nodes implement file merges through queues, which can greatly reduce CPU and disk IO consumption and improve storage space utilization.

[0094] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a computing node executing file merging according to an embodiment of the present invention. Figure 5 There are two computing nodes involved, computing node 1 and computing node 2.

[0095] Compute nodes 1 and 2 concurrently initiate a merge plan. The merge plan includes a merge operation. Specifically, the distribution log includes the parameters involved in the merge operation. The merge operation includes a node identifier, a start file identifier, an end file identifier, and a merge file identifier.

[0096] As an example, the node identifier of computing node 1 is CH1; the starting file identifier is 201905_0_0_0; the ending file identifier is 201905_1_1_0; and the merged file identifier is 201905_0_1_1. The node identifier of computing node 2 is CH2; the starting file identifier is 201905_2_2_0; the ending file identifier is 201905_3_3_0; and the merged file identifier is 201905_2_3_1.

[0097] After coordination by the coordination server, the merge operation of computing node 1 is performed first. Computing node 1 is the initiating node, and computing node 2 is the non-initiating node.

[0098] Compute Node 1 and Compute Node 2 monitor the public execution queue. If they detect changes in the public execution queue, they update the execution queue based on the public execution queue. This is recorded in the response log.

[0099] The file merging process differs for initiating and non-initiating nodes. Compute node 1 performs the merge in the cache; compute node 2 loads the merged file into the shared cache. After the merge operation on compute node 1 is complete, a similar merge operation is performed on compute node 2.

[0100] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the main structure of a file merging system according to an embodiment of the present invention. The file merging system can implement a file merging method, such as Figure 6 As shown, the file merging system specifically includes:

[0101] An initiating node 601 is used to push a merge operation, wherein the merge operation includes a computing node identifier, multiple file identifiers, and a merged file identifier. The initiating node is a computing node that initiates the merge operation in the distributed file storage system.

[0102] The coordination server 602 receives the merge operation pushed by the initiating node, creates a file merge task based on the computing node identifier, the multiple file identifiers, and the merged file identifier, and places the text merge task into a public execution queue;

[0103] The computing node 603 is configured to update the execution queue according to the public execution queue when a change in the public execution queue is detected, and perform file merging based on the execution queue.

[0104] It should be noted that the initiating node 601 is a type of computing node.

[0105] In one embodiment of the present invention, the files corresponding to the multiple file identifiers are all files with a data size smaller than a preset file size.

[0106] In one embodiment of the present invention, the coordination server 602 is specifically used to receive multiple merge operations pushed by multiple initiating nodes; and, based on the file identifier and the time of sending the merge operation, determine the validity of the multiple merge operations; and put the valid merge operations into a public execution queue according to the merge operation time.

[0107] In one embodiment of the present invention, the files corresponding to the multiple file identifiers belong to the same data partition and are adjacent.

[0108] In one embodiment of the present invention, the computing node 603 is specifically used for the public execution queue located on the coordination server, and upon receiving a notification message from the coordination server, monitors changes in the public execution queue and updates the execution queue according to the public execution queue.

[0109] In one embodiment of the present invention, the computing node 603 is specifically configured to query the shared storage based on the execution queue to obtain the merged file, and load the merged file into the cache to perform file merging.

[0110] In one embodiment of the present invention, the initiating node 601 is specifically configured to execute file merging in the cache based on the execution queue.

[0111] Figure 7 An exemplary system architecture 700 is shown to which the file merging method or file merging system according to an embodiment of the present invention may be applied.

[0112] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0113] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0114] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0115] Server 705 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 701, 702, and 703. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (for example, target push information and product information—for example only) to the terminal device.

[0116] It should be noted that the file merging method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the file merging system is generally set in the server 705 .

[0117] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0118] Reference below Figure 8 , which shows a schematic structural diagram of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0119] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the system 800 are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0120] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0121] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are performed.

[0122] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The modules described may also be provided in a processor. For example, they may be described as: a processor including an initiating node, a coordinating server, and a computing node. The names of these modules do not, in some cases, constitute a limitation on the module itself. For example, a computing node may also be described as "used to monitor changes in the public execution queue, update the execution queue according to the public execution queue, and perform file merging based on the execution queue."

[0125] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0126] Receive a merge operation pushed by an initiating node, the merge operation including a computing node identifier, multiple file identifiers, and a merged file identifier, the initiating node being a computing node that initiates the merge operation in the distributed file storage system;

[0127] Creating a file merging task based on the computing node identifier, the multiple file identifiers, and the merged file identifier;

[0128] The text merging task is placed in a public execution queue so that the computing nodes in the distributed file storage system can update the execution queue according to the public execution queue when monitoring changes in the public execution queue, and perform file merging based on the execution queue.

[0129] According to the technical solution of an embodiment of the present invention, a merge operation pushed by an initiating node is received, the merge operation includes a computing node identifier, multiple file identifiers, and a merge file identifier, and the initiating node is a computing node that initiates the merge operation in a distributed file storage system; a file merge task is created based on the computing node identifier, the multiple file identifiers, and the merge file identifier; the text merge task is placed in a public execution queue, so that the computing nodes in the distributed file storage system, when monitoring changes in the public execution queue, update the execution queue according to the public execution queue, and execute file merges based on the execution queue. In a distributed file storage system, computing nodes implement file merges through queues, thereby greatly reducing CPU and disk IO consumption and improving storage space utilization.

[0130] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for merging files, characterized in that: include: Receiving multiple merge operations pushed by multiple initiating nodes, the merge operations including a computing node identifier, multiple file identifiers, and a merged file identifier, the initiating node being a computing node that initiates the merge operation in the distributed file storage system; Creating a file merging task based on the computing node identifier, the multiple file identifiers, and the merged file identifier; determining validity of the plurality of merge operations based on the file identifier and the time when the merge operation was sent; The valid merge operations are placed in the public execution queue according to the merge operation time, so that the computing nodes in the distributed file storage system can update the execution queue according to the public execution queue when they monitor the changes in the public execution queue, and perform file merges based on the execution queue. The execution queue is the queue for the computing node itself to perform text merges.

2. The method for merging files according to claim 1, wherein: The files corresponding to the multiple file identifiers are all files with a data volume smaller than a preset file size.

3. The method for merging files according to claim 1, wherein: The files corresponding to the multiple file identifiers belong to the same data partition and are adjacent.

4. The method for merging files according to claim 1, wherein: When a computing node in the distributed file storage system monitors a change in the public execution queue, the computing node updates the execution queue according to the public execution queue, including: The public execution queue is located on the coordination server, and the computing node in the distributed file storage system receives a notification message from the coordination server and monitors changes in the public execution queue; The execution queue is updated according to the public execution queue.

5. The method for merging files according to claim 1, wherein: The performing file merging based on the execution queue includes: Other computing nodes in the distributed file storage system query the shared storage based on the execution queue to obtain the merged file, and load the merged file into the cache to perform file merging.

6. The method for merging files according to claim 1, wherein: The performing file merging based on the execution queue includes: The initiating node performs file merging in a cache based on the execution queue.

7. A file merging system, characterized in that: include: An initiating node, configured to push a merge operation, wherein the merge operation includes a computing node identifier, multiple file identifiers, and a merged file identifier. The initiating node is a computing node in the distributed file storage system that initiates the merge operation. The coordination server receives multiple merge operations pushed by multiple initiating nodes, creates a file merge task based on the computing node identifier, the multiple file identifiers, and the merged file identifier, and determines the validity of the multiple merge operations based on the file identifier and the time when the merge operation was sent; Put the valid merge operations into the public execution queue according to the merge operation time; The computing node is used to update the execution queue according to the public execution queue when monitoring changes in the public execution queue, and perform file merging based on the execution queue. The execution queue is the queue for the computing node itself to perform text merging.

8. An electronic device for merging files, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images