Data processing method, system, electronic device and computer storage medium

Abstract

An embodiment of the present application provides a data processing method, system, electronic device, and computer storage medium, which are applied to a database system. The database system includes a shared storage unit, a master node, and multiple data processing nodes. The multiple data processing nodes share data files stored in the shared storage unit. The data processing method applied to the master node includes: querying multiple data processing nodes for metadata corresponding to data files accessed by them to obtain global metadata; calling an interface of the shared storage unit to obtain file metadata corresponding to data files stored in the shared storage unit; comparing the global metadata with the file metadata to determine data files that exist in the file metadata but not in the global metadata as files to be recycled; and recycling the files to be recycled determined in the shared storage unit.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data processing method, system, electronic device, and computer storage medium. Background Art

[0002] In the data processing nodes of the database, data files are generally named after the file node number filenodeoid of the table. However, in the shared storage unit, the universal unique identifier uuid is used to store data files. When the data processing node writes data files to the shared storage unit, several data files corresponding to multiple file node numbers filenodeoid are generally written to one data file in the shared storage unit, and the corresponding directory is not saved. As a result, the corresponding data files in the shared storage unit cannot be deleted by deleting the file node number filenodeoid of the table.

[0003] In view of this, a technical problem that needs to be solved urgently in the prior art is how to recycle files in a shared storage unit. Summary of the Invention

[0004] In view of this, an embodiment of the present application provides a data processing solution to at least partially solve the above-mentioned problem.

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, which is applied to a master node of a database, the database system including a shared storage unit, a master node, and a data processing node, the data processing nodes sharing data files stored in the shared storage unit, the method including: querying the multiple data processing nodes for metadata corresponding to the data files they access to obtain global metadata; calling an interface of the shared storage unit to obtain file metadata corresponding to the data files stored in the shared storage unit; comparing the global metadata with the file metadata to determine, as files to be recycled, data files that exist in the file metadata but not in the global metadata; and recycling the files to be recycled determined in the shared storage unit.

[0006] According to a second aspect of an embodiment of the present application, a database system is provided, including a shared storage unit, a master node, and multiple data processing nodes, wherein the multiple data processing nodes share data files stored in the shared storage unit, wherein the master node is used to query the multiple data processing nodes for metadata corresponding to the data files it accesses to obtain global metadata; call an interface of the shared storage unit to obtain file metadata corresponding to the data files stored in the shared storage unit; compare the global metadata with the file metadata, and determine that the data files that exist in the file metadata but do not exist in the global metadata are files to be recycled; and recycle the files to be recycled determined in the shared storage unit.

[0007] According to the third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0009] According to the data processing solution provided by the embodiment of the present application, global metadata and file metadata are obtained through the master node, so as to accurately identify data files that exist in the file metadata but not in the global metadata as files to be recovered, so that data files that do not exist in the data processing nodes (that is, data files that have been deleted by all data processing nodes that have operated the data files) but are still left on the shared storage unit can be recovered from the shared storage unit. There is no need to perform complex processing, no need to make complex judgments based on the deletion operations of the data processing nodes before recovering the data files on the shared storage unit, and no need to add other version numbers for maintaining data files. The recovery process is simple and efficient. The metadata is directly used to determine the files to be recovered, which ensures the accuracy of the determined files to be recovered, and thus ensures the accuracy of the recovery processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1 A schematic diagram of an exemplary system applicable to the data processing method of an embodiment of the present application;

[0012] Figure 2 is a structural block diagram of a database system according to an embodiment of the present application;

[0013] Figure 3 is a flowchart of the steps of a data processing method according to an embodiment of the present application;

[0014] Figure 4A is a flowchart of another data processing method according to an embodiment of the present application;

[0015] Figure 4B A program diagram of a master node of a database system according to an embodiment of the present application;

[0016] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0018] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0019] Figure 1 An exemplary system for the data processing method of the embodiment of the present application is shown. Figure 1 As shown, the system may include a cloud service 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in the figure is for multiple user devices.

[0020] The cloud server 102 may be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 may perform any suitable function. For example, in some embodiments, the cloud server 102 may include the database system described below.

[0021] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0022] The user device 106 may include any one or more user devices suitable for operating the database system. For example, the user device 106 may operate a data processing node, retrieve a data file from a shared storage unit and modify it, and store the modified data file in the shared storage unit. In some embodiments, the user device 106 may include any suitable type of device. For example, in some embodiments, the user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0023] See also Figure 2 , shows a database system, which includes a shared storage unit 201, a master node 202, and multiple data processing nodes 203. The multiple data processing nodes 203 share data files stored in the shared storage unit.

[0024] It should be noted that the database system in the embodiment of the present application can be any form of database system, and this embodiment is not limited to this. For example, the database system can be a cloud native database system, and the corresponding shared storage unit is a shared storage unit on the cloud.

[0025] The master node 202 is used to query the multiple data processing nodes 203 for metadata corresponding to the data files it accesses to obtain global metadata; call the interface of the shared storage unit 201 to obtain file metadata corresponding to the data files stored in the shared storage unit 201; compare the global metadata with the file metadata, and determine that the data files that exist in the file metadata but do not exist in the global metadata are to be recycled; and recycle the to-be-recycled files determined in the shared storage unit.

[0026] Generally, multiple data processing nodes 203 can share the data files stored in the shared storage unit 201 .

[0027] Specifically, multiple data processing nodes 203 can receive data processing requests, and obtain corresponding data files from the shared storage unit 201 according to the data processing requests, store the data files in the local storage space of each data processing node 203, and perform corresponding data processing operations based on the data files.

[0028] Multiple data processing nodes 203 can obtain the same data file and perform corresponding data processing operations. During the data processing operations, the data processing node 203 will locally maintain the metadata of the corresponding data file. After receiving an operation request to delete a data file, the data processing node 203 can delete the corresponding data file locally from the data processing node 203. However, since a data file may exist on multiple data processing nodes 203, deleting the data file in the shared storage unit based on the deletion operation of the data processing node 203 may cause deletion errors.

[0029] Therefore, in this embodiment, the master node queries the multiple data processing nodes 203 for metadata corresponding to the data files they access, and thus obtains metadata of all data files being stored by the data processing nodes 203, thereby obtaining corresponding global metadata, which records all data files that have not been deleted by the data processing nodes 203.

[0030] In addition, by calling the interface of the shared storage unit, the file metadata corresponding to the data file stored in the shared storage unit can be obtained. By comparing the global metadata and the file metadata, the data file that exists in the file metadata but does not exist in the global metadata can be determined as a file to be recycled, that is, the file that is still stored in the shared storage unit but has not been edited by the data processing node 203 is regarded as a file to be recycled; the file to be recycled determined in the shared storage unit is recycled. The solution provided by this embodiment obtains the global metadata and the file metadata through the master node, thereby accurately locating the file to be recycled that exists in the file metadata but does not exist in the global metadata, so that the data file that does not exist in the data processing node (that is, the data file that has been deleted by all the data processing nodes that have operated on the data file) but still remains on the shared storage unit can be recycled from the shared storage unit without complicated processing, without complex judgment based on the deletion operation of the data processing node before recycling the data file on the shared storage unit, and without adding other methods for maintaining the version number of the data file. The recycling process is simple and efficient. The direct use of metadata to determine the file to be recycled ensures the accuracy of the determined file to be recycled, thereby ensuring the accuracy of the recycling process.

[0031] Optionally, in this embodiment, the master node is configured to initiate a file cleanup task, which, through the file cleanup task, continues to query the multiple data processing nodes for metadata corresponding to the data files they access, obtaining global metadata, and subsequently determining the file metadata, determining the files to be recycled, and performing recycling processing. By initiating the file cleanup task and performing the above operations through the file cleanup task, the master node can control the timing of recycling processing by controlling the timing of initiating the file cleanup task, making the recycling process more flexible.

[0032] Optionally, in this embodiment, a scheduling process is started by the master node, and the scheduling process is used to determine whether there is a running file cleaning process at intervals of a preset time. If there is no running file cleaning process, a new file cleaning process is started to start the file cleaning task, wherein the preset time is determined based on the deletion progress of the data files by the data processing node.

[0033] Optionally, in this embodiment, the file cleaning task is used to determine the modification time corresponding to each of the files to be recycled in the shared storage unit; and to obtain the start time of each transaction operation being executed in the database system; and to recycle data files whose modification time is earlier than the start time of each transaction operation, wherein the transaction operation is an operation that can change the data files stored in the shared storage unit.

[0034] The following is an illustrative example of the solution of this application through a specific method flow, and the following method is executed by the master node.

[0035] See also Figure 3 , shows a flow chart of a data processing method, as shown in the figure, which includes:

[0036] S301: Query the multiple data processing nodes for metadata corresponding to the data files they access, and obtain global metadata.

[0037] S302: Call the interface of the shared storage unit to obtain file metadata corresponding to the data file stored in the shared storage unit.

[0038] S303: Compare the global metadata with the file metadata to determine that the data file that exists in the file metadata but does not exist in the global metadata is a file to be recycled.

[0039] It should be noted that, because multiple data files in a data processing node are written into a single data file in the shared storage unit, the file marked as "to be received" may be part of the data files in the shared storage unit. If the global metadata and file metadata use table names to record file information, the file specifically identified as "to be received" may be the table name.

[0040] S304: Recycle the to-be-recycled files determined in the shared storage unit.

[0041] The specific recycling process may be to delete the file to be recycled from the shared storage unit. Of course, other recycling processes are also within the scope of protection of this application.

[0042] The solution provided by this embodiment obtains global metadata and file metadata through the master node, thereby accurately locating the to-be-recovered files that exist in the file metadata but not in the global metadata, so that data files that do not exist in the data processing nodes (that is, data files that have been deleted by all data processing nodes that have operated the data files) but are still left on the shared storage unit can be recovered from the shared storage unit without complicated processing, without the need for complicated judgment based on the deletion operation of the data processing node to recycle the data files on the shared storage unit, and without the need to add other version numbers for maintaining the data files. The recovery process is simple and efficient. The metadata is directly used to determine the to-be-recovered files, which ensures the accuracy of the determined to-be-recovered files, and thus ensures the accuracy of the recovery processing process.

[0043] See also Figure 4A, shows a flow chart of another data processing method, as shown in the figure, which includes:

[0044] S401: Initiate a file cleanup task through the master node.

[0045] Optionally, in this embodiment, a scheduling process is initiated by the master node, and a predetermined time interval is used to determine whether a file cleaning process is currently running. The predetermined time interval is determined based on the deletion progress of the data files by the data processing node. If no file cleaning process is currently running, a new file cleaning process is initiated to initiate the file cleaning task. This ensures that if a file cleaning process worker is already running, a restarted file cleaning process worker will fail, thereby avoiding parallel operations on data files in the shared storage unit.

[0046] S402: query the multiple data processing nodes through the file cleaning task for metadata corresponding to the data files accessed by them, and obtain multiple metadata corresponding to the multiple data processing nodes.

[0047] S403: Deduplication and sorting of the plurality of metadata are performed through the file cleaning task to obtain the global metadata.

[0048] In this embodiment, since a data file may be processed by multiple data processing nodes, multiple metadata obtained from multiple data processing nodes are aggregated, and there may be duplicate data files. In this embodiment, the data files in the multiple metadata can be deduplicated and sorted to obtain global metadata without duplication of data files.

[0049] S404: Call the interface of the shared storage unit through the file cleaning task to obtain file metadata corresponding to the data file stored in the shared storage unit.

[0050] S405 : Compare the global metadata and the file metadata through the file cleaning task to determine data files that exist in the file metadata but not in the global metadata as files to be recycled.

[0051] Specifically, the comparison may be made between table names in the global metadata and the file metadata, etc., which is not limited in this embodiment.

[0052] S406: Determine the modification time corresponding to each of the files to be recycled in the shared storage unit through the file cleaning task.

[0053] S407: Obtain the start time of each transaction operation being executed in the database system through the file cleaning task, wherein the transaction operation is an operation that can change the data file stored in the shared storage unit.

[0054] When archiving files, there are write transactions. When the write transaction reaches the storage limit in the cache, an Object Storage Service (OSS) file will be immediately generated and the OSS file will be stored in the shared storage unit. At this time, the write transaction has not been committed yet, and other transactions cannot access the OSS file. However, the file metadata obtained by calling the API interface of the shared storage unit includes these newly added data files. If the data files that exist in the file metadata but not in the global metadata are directly determined as files to be recycled and recycled, other transactions will be unable to read these data files after they are committed.

[0055] Therefore, in this embodiment, the start time of each transaction operation being executed in the database system is determined, and in subsequent steps, data files whose modification time is earlier than the start time of each transaction operation are recycled to avoid recycling data files that have just been stored in the file storage system but have not been acquired by other transactions.

[0056] S408: Recycling data files whose modification time is earlier than the start time of each transaction operation through the file cleaning task.

[0057] Specifically, the status of the to-be-reclaimed files determined in the shared storage unit can be modified to an archive status through the file cleaning process; and data files that have been in the archive status for a time greater than a preset archive time can be deleted through the file cleaning process.

[0058] See also Figure 4BA database management program master can be run in the master node. The database management program can be started by a client, and the client can set a preset duration for starting a file cleaning process. The master can start a scheduling process SCHEDULER and synchronize a count value int corresponding to the number of file cleaning process workers to the scheduling process. The scheduling process can request the master to simultaneously start (start) multiple file cleaning process workers corresponding to the count value int according to the preset duration. The multiple file cleaning process workers can respectively query the metadata corresponding to the data files they access from the multiple data processing nodes to obtain global metadata; call the interface of the shared storage unit to obtain file metadata corresponding to the data files stored in the shared storage unit; compare the global metadata with the file metadata, and determine that the data files that exist in the file metadata but do not exist in the global metadata are files to be recycled; call the archive program archive to change the status of the files to be recycled to the archive status, and call the cleanup program cleanup to delete the files to be recycled that have been in the archive status for longer than the preset archive time.

[0059] The solution provided by this embodiment determines the start time of each transaction operation being executed in the database system, and recycles the data files whose modification time is earlier than the start time of the transaction operation, so as to avoid the recycling of data files that have just been stored in the file storage system but have not been acquired by other transactions; and when it is determined by calling a process that there is no running file cleaning process, a new file cleaning process is started according to a preset duration, which can ensure that if there is already a running file cleaning process worker, the restarted file cleaning process worker will fail, and after the previous file cleaning process is closed, a new file cleaning process can be started at an interval of preset duration to perform a new round of data file recycling processing again, so as to try to recycle data files in a timely manner.

[0060] Reference Figure 5 , shows a structural diagram of an electronic device according to the fifth embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0061] like Figure 5 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0062] in:

[0063] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0064] The communication interface 504 is used to communicate with other electronic devices or servers.

[0065] The processor 502 is configured to execute the program 510 , and specifically may execute the relevant steps in the above-mentioned data processing method embodiment.

[0066] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0067] The processor 502 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0068] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0069] The program 510 can be specifically used to enable the processor 502 to execute operations corresponding to the data processing method described in any of the aforementioned method embodiments.

[0070] Specifically, the program 510 can be used to enable the processor 502 to perform the following steps: query the metadata corresponding to the data files accessed by the multiple data processing nodes to obtain global metadata; call the interface of the shared storage unit to obtain the file metadata corresponding to the data files stored in the shared storage unit; compare the global metadata and the file metadata to determine the data files that exist in the file metadata but not in the global metadata as files to be recycled; and recycle the files to be recycled determined in the shared storage unit.

[0071] Optionally, in any embodiment of the present application, the program 510 can also be used to enable the processor 502 to perform the following steps: starting a file cleaning task through the master node, and continuing to query the metadata corresponding to the data files accessed by the multiple data processing nodes through the file cleaning task to obtain global metadata.

[0072] Optionally, in any embodiment of the present application, the file cleaning task is started through the master node, including: starting a scheduling process through the master node, judging whether there is a running file cleaning process through a preset time interval of the scheduling process, and the preset time interval is determined according to the deletion progress of the data processing node in deleting the data file; if there is no running file cleaning process, starting a new file cleaning process to start the file cleaning task.

[0073] Optionally, in any embodiment of the present application, the recycling of the to-be-reclaimed files determined in the shared storage unit includes: determining, through the running file cleanup process, the modification time corresponding to each of the to-be-reclaimed files in the shared storage unit; obtaining the start time of each transaction operation being executed in the database system, wherein the transaction operation is an operation that can change the data file stored in the shared storage unit; and recycling the to-be-reclaimed files whose modification time is earlier than the start time of each transaction operation.

[0074] Optionally, in any embodiment of the present application, querying the metadata corresponding to the data files accessed by the multiple data processing nodes to obtain global metadata includes: querying the metadata corresponding to the data files accessed by the multiple data processing nodes through the file cleaning task to obtain multiple metadata corresponding to the multiple data processing nodes; deduplicating and sorting the multiple metadata to obtain the global metadata.

[0075] Optionally, in any embodiment of the present application, the recycling of the files to be recycled determined in the shared storage unit includes: modifying the status of the files to be recycled determined in the shared storage unit to an archive status; and deleting the files to be recycled that have been in the archive status for a time greater than a preset archive time.

[0076] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0077] An embodiment of the present application further provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the operation corresponding to any data processing method in the above-mentioned multiple method embodiments is implemented.

[0078] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any data processing method in the above-mentioned multiple method embodiments.

[0079] An embodiment of the present application also provides a database system, including a shared storage unit, a master node, and multiple data processing nodes, wherein the multiple data processing nodes share data files stored in the shared storage unit, wherein the master node is used to query the multiple data processing nodes for metadata corresponding to the data files it accesses to obtain global metadata; call an interface of the shared storage unit to obtain file metadata corresponding to the data files stored in the shared storage unit; compare the global metadata with the file metadata, and determine that the data files that exist in the file metadata but do not exist in the global metadata are files to be recycled; and recycle the files to be recycled determined in the shared storage unit.

[0080] Optionally, in any embodiment of the present application, the master node is used to start a file cleaning task to continue to execute the step of querying the metadata corresponding to the data files accessed by the multiple data processing nodes through the file cleaning task to obtain global metadata.

[0081] Optionally, in any embodiment of the present application, the master node is used to start a scheduling process, and the scheduling process is used to determine whether there is a running file cleaning process at intervals of a preset time. If there is no running file cleaning process, a new file cleaning process is started to start the file cleaning task, wherein the preset time is determined based on the deletion progress of the data file by the data processing node.

[0082] Optionally, in any embodiment of the present application, the file cleaning task is used to determine the modification time corresponding to each of the files to be recycled in the shared storage unit; and to obtain the start time of each transaction operation being executed in the database system; and to recycle the files to be recycled whose modification time is earlier than the start time of each transaction operation, wherein the transaction operation is an operation that can change the data files stored in the shared storage unit.

[0083] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0084] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0085] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0086] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A data processing method, applied to a master node of a database system, wherein the database system includes a shared storage unit, a master node, and multiple data processing nodes, the method comprising: querying the plurality of data processing nodes for metadata corresponding to the data files accessed by the nodes to obtain global metadata; Calling the interface of the shared storage unit to obtain file metadata corresponding to the data file stored in the shared storage unit; Comparing the global metadata with the file metadata, determining data files that exist in the file metadata but not in the global metadata as files to be recycled, wherein the files to be recycled represent files stored in the shared storage unit but not edited by the multiple data processing nodes, or data files that have been deleted by all data processing nodes that have operated on the data files but still remain on the shared storage unit; Recycling the to-be-reclaimed files determined in the shared storage unit specifically includes: Determining, through a file cleanup task, modification times corresponding to the files to be recycled in the shared storage unit; Obtaining the start time of each transaction operation being executed in the database system, wherein the transaction operation is an operation that can change the data file stored in the shared storage unit; The to-be-reclaimed files whose modification time is earlier than the start time of each transaction operation are recycled.

2. The method according to claim 1, wherein The method further comprises: A file cleaning task is started through the master node, so as to continue to query the metadata corresponding to the data files accessed by the multiple data processing nodes through the file cleaning task to obtain the global metadata.

3. The method according to claim 2, wherein: The starting of the file cleaning task by the master node includes: Initiate a scheduling process through the master node, and determine whether there is a running file cleaning process according to a preset time interval of the scheduling process, wherein the preset time interval is determined according to the deletion progress of the data file by the data processing node; If there is no file cleaning process running, a new file cleaning process is started to start the file cleaning task.

4. The method according to any one of claims 1 to 3, wherein: The step of querying the plurality of data processing nodes for metadata corresponding to the data files accessed by the plurality of data processing nodes to obtain global metadata includes: querying the plurality of data processing nodes for metadata corresponding to the data files accessed by the plurality of data processing nodes, and obtaining a plurality of metadata corresponding to the plurality of data processing nodes; The multiple metadata are deduplicated and sorted to obtain the global metadata.

5. The method according to any one of claims 1 to 3, wherein: The recycling of the to-be-reclaimed files determined in the shared storage unit includes: The status of the to-be-reclaimed file determined in the shared storage unit is modified to an archive status; and the to-be-reclaimed file that has been in the archive status for a time greater than a preset archive time is deleted.

6. A database system comprising a shared storage unit, a master node, and a plurality of data processing nodes, wherein the plurality of data processing nodes share data files stored in the shared storage unit; The master node is used to query the multiple data processing nodes for metadata corresponding to the data files it accesses, to obtain global metadata; calling an interface of the shared storage unit to obtain file metadata corresponding to a data file stored in the shared storage unit; comparing the global metadata with the file metadata, and determining as a to-be-reclaimed file a data file that exists in the file metadata but does not exist in the global metadata, wherein the to-be-reclaimed file represents a file that is stored in the shared storage unit but has not been edited by the plurality of data processing nodes, or a data file that has been deleted by all data processing nodes that have operated on the data file but still remains on the shared storage unit; Recycling the to-be-reclaimed files determined in the shared storage unit specifically includes: Determining, through a file cleanup task, modification times corresponding to the files to be recycled in the shared storage unit; Obtaining the start time of each transaction operation being executed in the database system, wherein the transaction operation is an operation that can change the data file stored in the shared storage unit; The to-be-reclaimed files whose modification time is earlier than the start time of each transaction operation are recycled.

7. The system according to claim 6, wherein: The master node is used to start a file cleaning task to continue to execute the step of querying the multiple data processing nodes for metadata corresponding to the data files they access through the file cleaning task to obtain global metadata.

8. The system according to claim 7, wherein: The master node is used to start a scheduling process, and the scheduling process is used to determine whether there is a running file cleaning process at intervals of a preset time. If there is no running file cleaning process, a new file cleaning process is started to start the file cleaning task, wherein the preset time is determined according to the deletion progress of the data file by the data processing node.

9. The system according to claim 7, wherein: The file cleaning task is used to determine the modification time corresponding to each of the files to be recycled in the shared storage unit; and to obtain the start time of each transaction operation being executed in the database system; The method is further configured to recycle the to-be-reclaimed files whose modification time is earlier than the start time of each transaction operation, wherein the transaction operation is an operation that can change the data files stored in the shared storage unit.

10. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the data processing method according to any one of claims 1 to 5.

11. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images