Data processing method and device and computing equipment

Abstract

The invention discloses a data processing method, device and equipment, and relates to the technical field of storage. The method comprises the steps that after a server side of the distributed storage system determines first data according to a to-be-deleted persistent layer log PLOG, physical address information of the first data can be sent to a storage device, and the first data are stored in the storage device in an appending writing mode by the server side; and writing the business data in the target logic block address twice and persistently storing the business data in at least one to-be-recycled data block in the storage equipment for the first time. When the storage device carries out garbage collection, the first data in the valid data can be identified based on the physical address information of the first data, the first data is migrated to the first data block, and the valid data except the first data are migrated to the second data block. According to the embodiment of the invention, the server side of the distributed storage system enables the storage device to independently migrate the service junk data to one data block, data migration operation does not need to be carried out again during junk collection, and the expenditure of junk collection operation can be reduced.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method, apparatus and computing device. Background Art

[0002] Append only refers to a data writing mode, which is characterized by the fact that new data can only be appended to existing data, and existing data cannot be directly modified or deleted. When a distributed storage system writes data in append only mode, the write operation can only add new data to the end of existing data, and overwriting or deleting existing data is not allowed. The distributed storage system adopts the append only mode, which can effectively improve the writing performance and data reliability. Since the append only mode does not overwrite or delete existing data, a large amount of garbage data is generated in the distributed storage system. If the distributed storage system does not clear the garbage data in time, it will occupy a large amount of storage space, affecting the performance and stability of the distributed storage system. Summary of the invention

[0003] In order to solve the above-mentioned problems, a data processing method is provided in an embodiment of the present application. During the garbage collection process of the memory, the business garbage data in the valid data stored in the data block is migrated to a new data block, and the remaining valid data is migrated to another new data block. This makes it unnecessary to perform the data migration operation again when clearing the business garbage data, thereby reducing the overhead of the garbage collection operation.

[0004] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a server of a distributed storage system, and the server is used to access a storage device. The method includes: determining first data based on at least one persistence layer log (PLOG) to be deleted in the server, wherein the first data is business data that is first persistently stored in at least one data block to be recycled in the storage device and written twice to a target logical block address by the server in an append write manner; sending physical address information of the first data to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block when performing garbage collection, and to relocate second data in the at least one data block to be recycled to a second data block.

[0006] In this implementation, after the server of the distributed storage system determines the business garbage data according to the PLOG to be deleted, the physical address information of the business garbage data can be sent to the storage device. When the storage device performs garbage collection, the business garbage data in the valid data can be identified based on the physical address information of the business garbage data, and the business garbage data can be migrated to a data block. The server of the distributed storage system allows the storage device to migrate the business garbage data to a data block separately, so that when clearing the business garbage data, there is no need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0007] In one embodiment, before sending the physical address information of the first data to the storage device, the method includes: determining a PLOG list based on the at least one PLOG to be deleted, the PLOG list including each PLOG identifier to be deleted written in the first data and the offset of the first data in each PLOG to be deleted; converting the PLOG list into a logical block address (logical block address, LBA) list, the LBA list is used to indicate the physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and the offset address of the first data in each LBA.

[0008] In this embodiment, the address information recognizable by the storage device is generally a physical address, so the server of the distributed storage system can convert the PLOG list of logical block addresses including business garbage data into an LBA list of physical addresses including business garbage data, so that the storage device can receive the recognizable physical address and realize the migration of the business garbage data to a data block separately.

[0009] In one implementation, sending the physical address information of the first data to the storage device includes: sending the physical address information of the first data to the storage device through a target interface of the storage device.

[0010] In one embodiment, the storage device is a solid state drive (SSD), and a controller of the SSD is used to relocate the first data to the first data block and relocate the second data to the second data block when performing garbage collection.

[0011] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a storage device of a distributed storage system, wherein the storage device includes multiple data blocks, and the method includes: receiving physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server based on at least one persistent layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server writing twice to a target logical block address in an append write manner; determining at least one data block to be recovered among the multiple data blocks based on the physical address information of the first data; relocating the first data to the first data block; relocating the second data in the at least one data block to be recovered to the second data block; and erasing the at least one data block to be recovered.

[0012] In this implementation, after receiving the physical address information of the business garbage data, the storage device identifies the business garbage data in the valid data. When the storage device performs garbage collection, the business garbage data is migrated to one data block, and the valid data other than the business garbage data is migrated to another data block, so that when the storage device clears the business garbage data, it does not need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0013] In one embodiment, determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data includes: determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0014] In one implementation, the storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block, and to relocate the second data to the second data block.

[0015] In a third aspect, an embodiment of the present application provides a data processing device, comprising: a first processing unit, for determining first data based on at least one persistent layer log PLOG to be deleted in a server of a distributed storage system, wherein the first data is business data that is first persistently stored in at least one data block to be recycled in a storage device and written twice to a target logical block address by the server in an append write manner; a transceiver unit, for sending physical address information of the first data to the storage device, wherein the physical address information is used by the storage device to relocate the first data to the first data block when performing garbage collection, and to relocate the second data in the at least one data block to be recycled to the second data block.

[0016] In one embodiment, it also includes a second processing unit, which is used to: determine a PLOG list based on the at least one PLOG to be deleted, the PLOG list including each PLOG identifier to be deleted written in the first data and the offset of the first data in each PLOG to be deleted; convert the PLOG list into a logical block address LBA list, the LBA list is used to indicate the physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and the offset address of the first data in each LBA.

[0017] In one implementation, the transceiver unit is specifically configured to send the physical address information of the first data to the storage device via a target interface of the storage device.

[0018] In one embodiment, the storage device is a solid state drive (SSD), and a controller of the SSD is used to relocate the first data to the first data block and relocate the second data to the second data block when performing garbage collection.

[0019] In a fourth aspect, an embodiment of the present application provides a data processing device, comprising: a transceiver unit, for receiving physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server based on at least one persistent layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server twice writing into a target logical block address in an append write manner; a processing unit, for determining at least one data block to be recovered among the multiple data blocks based on the physical address information of the first data; the processing unit, for relocating the first data to the first data block; the processing unit, for relocating the second data in the at least one data block to be recovered to the second data block; the processing unit, for erasing the at least one data block to be recovered.

[0020] In one implementation, the processing unit is specifically configured to determine the at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0021] In one implementation, the storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block, and to relocate the second data to the second data block.

[0022] In a fifth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes various possible implementation embodiments of the first aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes various possible implementation embodiments of the second aspect.

[0024] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes various possible implementations of the first aspect.

[0025] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes various possible implementations of the second aspect.

[0026] In a ninth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computing device, the computing device implements each possible implementation embodiment of the first aspect.

[0027] In the tenth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computing device, the computing device implements each possible implementation embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0029] Figure 1 A schematic diagram of the structure of a distributed storage system provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0032] Figure 4A schematic diagram of the architecture of a storage device provided in an embodiment of the present application;

[0033] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the data processing process in a data block provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0038] Fig.10 A schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0039] Fig.11 A schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0042] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0043] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0044] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0045] A distributed storage system is a system that stores data on multiple nodes. Each node can be an independent computer, server or other storage device. Multiple nodes can be connected together through a network to form a storage system. Common distributed storage systems include Hadoop distributed file system (HDFS), Google file system (GFS), open source HBase (Apache HBase), open source Cassandra (Apache Cassandra), etc.

[0046] Figure 1 Schematic diagram of a distributed storage system provided in an embodiment of the present application. Figure 1 As shown, the distributed storage system 100 includes a client 110, a server 120 and a storage device 130. The number of the client 110, the server 120 and the storage device 130 is not limited to Figure 1 Quantity shown.

[0047] The client 110 may be a terminal device such as a laptop computer, a desktop computer, a tablet computer, a smart phone, etc. The client 110 may receive data input by a user and forward the data to the server 120 to write the data, or receive a command input by a user and forward the command to the server 120 to read the data. Typically, the client 110 may be installed with an application (APP). The user may log in to the APP account to use the functions of the distributed storage system 100.

[0048] The server 120 can be a server, data center or other device. The server 120 is used to manage and maintain the index structure of data. The index structure is a data structure used to speed up the retrieval of data. The index structure records the metadata information of the data in the storage device 130. The metadata information includes the file name, file path, location of the data block, creation time, etc. In the process of writing data, after receiving the write data, the server 120 is responsible for recording and updating the metadata information of the data to track the status and location of the data. In the process of reading data, after receiving the read request, the server 120 locates the logical block address of the data through the index structure according to the index information of the read request to read the data.

[0049] The server 120 is also responsible for the actual data storage and processing. In the process of writing data, after receiving the data, the server 120 writes the data into the physical page of the storage device 130. The server 120 is responsible for managing and allocating available physical pages, determining the location where the data is written, and writing the data into the storage device 130. In the process of reading data, the server 120 finds the physical location of the stored data according to the logical block address, and reads the corresponding data block or page from the storage device 130. The server 120 returns the read data to the client 110.

[0050] The storage device 130 can be an independent memory or a medium for storing data in the server 120, which is generally a memory with certain processing functions such as a hard disk drive (HDD), SSD, memory stick, etc. In the process of writing data, the storage device 130 receives a write request provided by the server 120 and writes the data to the corresponding physical page. In the process of reading data, the storage device 130 receives a read request provided by the server 120 and transmits the data to the server 120 for processing by quickly accessing the physical storage space.

[0051] The server 120 writes data to the storage device 130 in an append write mode, specifically:

[0052] Assume that the server 120 triggers a data write request for the first time, and the data write request includes data A and the LBA corresponding to data A, assuming that LBA=0. In response to the data write request, the server 120 creates a log file corresponding to the client 110, and generates a record A in the log file, and record A includes data A and LBA=0. If the server 120 triggers a data write request for the second time, the data write request includes data B and the LBA corresponding to data B, and LBA=0. It can be seen that the server 120 modifies the data written to LBA=0 from the original data A to data B. At this time, the server 120 generates a record B in the log file again in response to the second data write request, and record B includes data B and LBA=0.

[0053] The server 120 can pre-set the size of a log file (e.g., 100MB). When a log file is full, a new log file will be created. The log file is just an intermediate tool, and the content written in each log file is generally stored in the storage device 130. Therefore, the server 120 can set the data block based on the storage space of the storage device 130, and pre-set the size of the data block (e.g., 10MB). When the amount of data written to a log file reaches the size of a data block, a data block is formed, and the server 120 can store the data block in different storage devices 130.

[0054] When the server 120 writes data A to LBA=0 based on the client 110's data write request, the log file stores record A, that is, the mapping relationship between data A and data A and LBA=0. When the server 120 writes data B to LBA=0 based on the client 110's data write request, the log file stores record B, that is, the mapping relationship between data B and data B and LBA=0. At this time, the server 120 sets record A as business junk data and sets record B as business valid data.

[0055] Normally, the storage device 130 can identify its own data as valid data or garbage data. In the process of recycling garbage data, the storage device 130 migrates the valid data in the data block to a new data block, and then clears all the data in the original data block to recycle the garbage data in the data block. However, the storage device 130 cannot identify the business garbage data and business valid data recorded by the server 120, and can only treat the business garbage data and business valid data as valid data and migrate them to the new data block. If the server 120 needs to delete the business garbage data, it is necessary to instruct the storage device 130 to migrate the business valid data in the new data block to another new data block again, and then clear all the data in the new data block again. It can be seen from this that the garbage collection operation of the related art has a relatively large overhead.

[0056] In order to solve the defects of the garbage collection operation of the related art, the embodiment of the present application provides a data processing method and server. After the server 120 determines the business garbage data according to the PLOG to be deleted, the server 120 sends the physical address information of the business garbage data to the storage device 130. When the storage device 130 performs garbage collection, it can identify the business garbage data in the valid data based on the physical address information of the business garbage data, and migrate the business garbage data to a data block. The server 120 allows the storage device 130 to migrate the business garbage data to a data block separately, so that when clearing the business garbage data, there is no need to perform the data migration operation again, which can reduce the overhead of the storage device 130 performing the garbage collection operation.

[0057] Figure 2 Schematic diagram of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the data processing method is executed by the server 120 of the distributed storage system 100, and the specific implementation process is as follows:

[0058] Step S201 : the server 120 determines first data according to at least one persistence layer log PLOG to be deleted of the server 120 .

[0059] The first data is business garbage data, that is, when the server 120 writes data in an append write mode, the first of the two writes to the target logical block address is persistently stored in at least one data block to be recycled in the storage device. Normally, when the server 120 loses data, multiple write requests arrive at the same time, or rewrites after a write failure, the server 120 may assign the same logical block address to two or more write requests. When the server 120 detects that the same logical block address is written twice, the data written for the first time can be set as business garbage data, and the data written for the second time can be set as business valid data.

[0060] like Figure 3 As shown, the server 120 may include a software part 121 and a hardware part 122. The software part 121 may be divided into a metadata layer 1211 and a persistence layer 1212 based on the execution function. The metadata layer 1211 and the persistence layer 1212 may be executed by a processor in one server or by processors in different servers. The hardware part 122 may include N interfaces. N is a positive integer greater than or equal to 1. The N interfaces may be interfaces in one server connected to different storage devices 130 or interfaces in different servers connected to different storage devices 130.

[0061] The metadata layer 1211 is used to store and manage metadata information of data, such as file name, file size, creation time, etc. In the process of reading data, the metadata layer 1211 provides the location and index information of the data so that the server 120 can quickly locate and retrieve the data. In the process of writing data, the metadata layer 1211 is responsible for recording and updating the metadata of the data to track the status and location of the data.

[0062] The metadata layer 1211 records the metadata information of a data in PLOG. PLOG is an incremental log in a database system, which is used to record modification operations on the database. PLOG records the operations performed by each transaction, such as insert, update, delete, etc. The content recorded by PLOG will be stored in an orderly manner on a persistent medium, so that when the system fails or recovers, PLOG can be used to recover data and ensure consistency. In an embodiment of the present application, the metadata layer 1211 stores multiple PLOGs, and each PLOG records an identifier and offset of a data. When the server 120 writes business garbage data to the storage device 130, it can record and update the metadata information of the written data to track the status and location of the written data. After the server 120 completes the write operation, the metadata layer 1211 can record the identifier and offset of the business garbage data in the logical block address to the PLOG to form a PLOG of the business garbage data.

[0063] When the metadata layer 1211 performs garbage collection on multiple business garbage data, the PLOGs corresponding to the multiple business garbage data can be sorted into a PLOG list, and then the PLOG list can be forwarded to the persistence layer 1212. The PLOG list is a list composed of multiple PLOGs, each of which records the identifier and offset of a business garbage data. The metadata layer 1211 sorts the multiple PLOGs into a PLOG list and sends them to the persistence layer 1212 together to improve the efficiency of garbage collection by the storage device 130.

[0064] In step S202 , the server 120 sends the physical address information of the first data to the storage device 130 .

[0065] The persistence layer 1212 is used to store data in a permanent medium to ensure the reliability and persistence of the data. When the server 120 writes data, the persistence layer 1212 stores the data in the persistent medium to ensure that the data is not lost or damaged in abnormal situations such as system shutdown. When the server 120 reads data, the persistence layer 1212 can provide an interface for reading and writing data stored on the persistent medium so that the application can access and operate the data.

[0066] In the embodiment of the present application, after receiving the PLOG list, the persistence layer 1212 can obtain the identification and offset of each business garbage data on the logical block address from each PLOG in the PLOG list. The persistence layer 1212 can determine the logical block address of each business garbage data according to the identification and offset of each business garbage data on the logical block address. The persistence layer 1212 can obtain the physical address of each business garbage data according to the mapping relationship between the logical block address and the physical address. Since the service end 120 cannot overwrite or delete the existing data when writing data in an append write mode, the data written to the same logical block address for the second time cannot be located at the physical address corresponding to the logical block address, but at the physical address behind the physical address corresponding to the logical block address. Therefore, the physical address corresponding to the logical block address is the physical address where the business garbage data is stored. In an example, the logical block address written to data A and data B is the same. When the memory writes data A, data A will be written to the physical address corresponding to the logical block address. When the memory writes data B, data B will be written to the physical address behind the physical address corresponding to the logical block address.

[0067] After the persistence layer 1212 obtains the physical address of each business garbage data, it can determine the LBA identifier corresponding to the physical address of each business garbage data and the offset in each LBA, and record the LBA identifier of each business garbage data and the offset in each LBA in the LBA list. Since the storage device 130 can only recognize the physical address, the storage device 130 can send the physical address of the business garbage data to the storage device 130 through the LBA list, so that the storage device 130 can identify the location where the business garbage data is stored, and realize accurate positioning and processing of the business garbage data.

[0068] The persistence layer 1212 can cache the physical addresses of the connected storage devices 130 , and after obtaining the LBA list, can determine the object to which the LBA list is sent. The persistence layer 1212 can call the interface of the target storage device 130 to send the LBA list to the target storage device 130 .

[0069] In the embodiment of the present application, after the server of the distributed storage system determines the business garbage data according to the PLOG to be deleted, the physical address information of the business garbage data can be sent to the storage device. When the storage device performs garbage collection, the business garbage data in the valid data can be identified based on the physical address information of the business garbage data, and the business garbage data can be migrated to a data block. The server of the distributed storage system allows the storage device to migrate the business garbage data to a data block separately, so that when clearing the business garbage data, there is no need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0070] Figure 4 Schematic diagram of the architecture of the storage device provided in the embodiment of the present application. Figure 4 As shown, the storage device 130 may include a main controller 131, multiple flash memory chips 132, multiple storage class memory (SCM) chips 133 and a dynamic random access memory (DRAM) chip 134.

[0071] The flash memory chip 132 is the main storage medium of the storage device 130 and is used to store data persistently. The flash memory chip 132 uses non-volatile storage technology and can retain data after power failure.

[0072] SCM chip 133 is a new storage technology between traditional DRAM and flash memory. The function of SCM chip 133 is to provide non-volatile storage and lower access latency. SCM chip 133 can be used as a cache or main storage medium to speed up the reading and writing of data.

[0073] The DRAM chip 134 is used as a cache for the main controller 131 to temporarily store frequently accessed data and metadata. The DRAM chip 134 can provide fast read and write speeds to alleviate the speed difference between the main memory and the flash memory chip and improve overall performance.

[0074] The main controller 131 can cache the received data to the flash memory chip 132, the SCM chip 133 or other storage chips. After receiving the read request, the main controller 131 can read the target data in the target storage chip. The main controller 131 can perform garbage collection on the storage data in each storage chip, as well as other functions. The main controller 131 may include a host interface 1311, a processor 1312 and a plurality of channel controllers. The host interface 1311 establishes a communication connection with an external device for receiving various instructions and writing data. The channel controller is a bridge between the processor 1312 and each storage chip, responsible for managing the reading, writing and erasing operations of data on the chip, and providing a data transmission interface with the host system.

[0075] After receiving the physical address information of the business garbage data through the host interface 1311, the processor 1312 identifies the business garbage data in the valid data. When the processor 1312 performs garbage collection, the business garbage data is migrated to a data block, and the valid data other than the business garbage data is migrated to another data block, so that when the processor 1312 clears the business garbage data, it does not need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0076] Figure 5 FIG. 1 is a flow chart of another data processing method provided in an embodiment of the present application. Figure 5 As shown, the data processing method is executed by the processor 1312 in the storage device 130 of the distributed storage system 100, and the specific implementation process is as follows:

[0077] In step S501 , the processor 1312 receives physical address information of first data sent by the server 120 .

[0078] The first data refers to business junk data. In the storage device 130, business junk data refers to data in the memory that is no longer needed and can be safely deleted. The physical address information is an LBA list, which includes the LBA identifier of each business junk data and the offset in each LBA.

[0079] Step S502: The processor 1312 determines at least one data block to be recycled among a plurality of data blocks according to the physical address information of the first data.

[0080] In step S503 , the processor 1312 moves the first data to the first data block.

[0081] In step S504 , the processor 1312 relocates the second data in the at least one data block to be recycled to the second data block.

[0082] Step S505: the processor 1312 erases at least one data block to be recycled.

[0083] Normally, when the processor 1312 performs garbage collection, it can identify the valid data stored in each data block, and migrate the valid data in the data block to a new data block, and then clear all the data in the original data block to realize the recovery of the garbage data in the data block. In the embodiment of the present application, after the processor 1312 receives the LBA list, it can obtain each physical address from the LBA list. The processor 1312 can determine the data block where each physical address is located according to each physical address, as the target data block. When the processor 1312 performs garbage collection, the data stored in the physical address in each target data block is used as business garbage data, and the business garbage data is migrated to a new data block, and other valid data except the business garbage data in each target data block is migrated to another new data block. Finally, the processor 1312 erases each target data block to release the storage space of each target data block.

[0084] The life cycle of business garbage data may change and become garbage data. In the next garbage collection cycle, since the life cycle of business garbage data in the data block is different, the proportion of garbage data in each data block will also be different. Therefore, when the processor 1312 performs garbage collection, it gives priority to selecting data blocks with a high proportion of garbage data for garbage collection, which can reduce the load of the memory migration of valid data, so as to improve the garbage collection efficiency and increase the service life of the memory.

[0085] The processor 1312 stores multiple data blocks, and the data type, proportion of different types of data, attributes of the data, and other information stored in each data block may be different. The processor 1312 can configure different garbage collection strategies for different data blocks to improve the efficiency of garbage collection in the memory. The garbage collection strategy may include delayed garbage collection, active garbage collection, mixed garbage collection, pre-fetched garbage collection, block garbage collection, and other strategies.

[0086] The delayed garbage collection strategy refers to a strategy in which the memory performs garbage collection during idle time, which does not affect the user's read and write operations. The active garbage collection strategy refers to a strategy in which the memory performs garbage collection immediately when data is written, which can improve the performance of the memory. The hybrid garbage collection strategy combines the advantages of the delayed garbage collection and active garbage collection strategies, and can dynamically adjust the garbage collection strategy according to the load of the memory. The prefetch garbage collection strategy refers to a strategy in which the memory prefetches garbage data that may be generated during idle time, which can reduce the garbage collection time and the impact on users. The block garbage collection strategy refers to a strategy in which valid data in multiple data blocks is organized into one data block, which can reduce the amount of garbage data to improve the performance of the memory.

[0087] Optionally, the processor 1312 may perform garbage collection in advance before receiving the LBA list. When the processor 1312 performs garbage collection, it may identify the valid data in each target data block in advance and prepare to migrate the valid data to a new data block. After receiving the LBA list, the processor 1312 may directly obtain the business garbage data from the valid data based on the physical address in the LBA list, migrate the business garbage data to a new data block, and migrate other valid data except the business garbage data in each target data block to another new data block, thereby accelerating the speed of the processor 1312 performing garbage collection to shorten the latency of the storage device 130 in processing data.

[0088] like Figure 6 As shown, at time T1, after the processor 1312 receives the physical address information of 14 business garbage data, it can identify the target data blocks including the physical address information of the 14 business garbage data based on the physical address information of the 14 business garbage data, which are data block 1, data block 2, data block 3, data block 4, data block 5 and data block 6.

[0089] At time T2, when the processor 1312 performs garbage collection, it can migrate the business garbage data from the 6 data blocks according to the physical address information of the 14 business garbage data, and migrate the 14 business garbage data to data block 7 (which can be migrated to multiple data blocks). Then, the processor 1312 migrates the valid data in the 6 data blocks to data block 8 (which can be migrated to multiple data blocks). Finally, the processor 1312 erases data block 1, data block 2, data block 3, data block 4, data block 5 and data block 6 to release the storage space of the 6 target data blocks.

[0090] At time T3, some of the 14 service junk data in data block 7 may be transformed into junk data. Some of the 11 valid data in data block 8 may be transformed into service junk data and some into junk data.

[0091] In the embodiment of the present application, after receiving the physical address information of the business garbage data, the processor 1312 identifies the business garbage data in the valid data. When the processor 1312 performs garbage collection, the business garbage data is migrated to a data block, and the valid data other than the business garbage data is migrated to another data block, so that when the processor 1312 clears the business garbage data, it does not need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0092] Figure 7 Schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 7 As shown, the data processing device 700 can be divided into a transceiver unit 710, a first processing unit 720 and a second processing unit 730 according to the execution function. The functions executed by each unit of the data processing device 700 are as follows:

[0093] The first processing unit 720 is used to determine the first data according to at least one persistence layer log PLOG to be deleted in the server of the distributed storage system. The first data is the business data that is first persistently stored in at least one data block to be recovered in the storage device by the server in an append write manner and written twice in the target logical block address. The transceiver unit 710 is used to send the physical address information of the first data to the storage device. The physical address information is used by the storage device to relocate the first data to the first data block when performing garbage collection, and to relocate the second data in at least one data block to be recovered to the second data block.

[0094] In one embodiment, the second processing unit 730 is further included to determine a PLOG list according to at least one PLOG to be deleted. The PLOG list includes each PLOG identifier to be deleted in which the first data is written and the offset of the first data in each PLOG to be deleted. The second processing unit 730 is also used to convert the PLOG list into a logical block address LBA list. The LBA list is used to indicate the physical address information of the first data. The LBA list includes each LBA identifier storing the first data and the offset address of the first data in each LBA.

[0095] In one implementation, the transceiver unit 710 is specifically configured to send the physical address information of the first data to the storage device through a target interface of the storage device.

[0096] In one implementation, the storage device is a solid-state drive (SSD), and a controller of the SSD is used to relocate the first data to a first data block and relocate the second data to a second data block when performing garbage collection.

[0097] Figure 8FIG. 1 is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application. Figure 8 As shown, the data processing device 800 can be divided into a transceiver unit 810 and a processing unit 820 according to the execution function. The functions executed by each unit of the data processing device 800 are as follows:

[0098] The transceiver unit 810 is used to receive the physical address information of the first data sent by the server of the distributed storage system. The physical address information is determined by the server based on at least one persistent layer log PLOG to be deleted by the server. The first data is the business data that is first persistently stored in the storage device by the server twice writing the target logical block address in an append write manner. The processing unit 820 is used to determine at least one data block to be recovered from multiple data blocks based on the physical address information of the first data. The processing unit 820 is used to relocate the first data to the first data block. The processing unit 820 is used to relocate the second data in at least one data block to be recovered to the second data block. The processing unit 820 is used to erase at least one data block to be recovered.

[0099] In one implementation, the processing unit 820 is specifically configured to determine at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0100] In one implementation, the storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to a first data block and to relocate the second data to a second data block.

[0101] Fig. 9 Schematic diagram of a computing device provided in an embodiment of the present application. Fig. 9 As shown, the computing device 900 includes a bus 910, a processor 920, a memory 930, and a communication interface 940. The processor 920, the memory 930, and the communication interface 940 communicate with each other through the bus 910. The computing device 900 can be a server, a computer, a storage system, etc. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0102] The bus 910 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9The bus 910 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 910 may include a path for transmitting information between various components of the computing device 900 (for example, the processor 920, the memory 930, and the communication interface 940).

[0103] The processor 920 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0104] The memory 930 may include a volatile memory, such as a random access memory (RAM), or a nonvolatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or a SSD.

[0105] The memory 930 stores executable program codes, and the processor 920 executes the executable program codes to respectively implement the functions of the transceiver unit 710, the first processing unit 720, and the second processing unit 730 of the aforementioned data processing device 700, thereby implementing the data processing method. That is, the memory 930 stores instructions for executing the data processing method.

[0106] Alternatively, the memory 930 stores executable codes, and the processor 920 executes the executable codes to respectively implement the functions of the transceiver unit 810 and the processing unit 820 of the aforementioned data processing device 800, thereby implementing the data processing method. That is, the memory 930 stores instructions for executing the data processing method. Fig. 9 It is merely exemplarily shown that the memory 930 stores program codes for implementing the functions of the aforementioned transceiver unit 710 , the first processing unit 720 , and the second processing unit 730 .

[0107] The communication interface 940 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0108] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0109] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 930 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method.

[0110] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the data processing method.

[0111] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data processing device 700 or the data processing device 800. In other words, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more modules in the transceiver unit 710, the first processing unit 720, and the second processing unit 730, or implement the functions of one or more modules in the transceiver unit 810 and the processing unit 820.

[0112] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, two computing devices are computing device 900A and computing device 900B respectively, which are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 930 in the computing device 900A stores instructions for executing the functions of some modules in the data processing device 700 or the data processing device 800. Fig.11 In the example, the memory 930 in the computing device 900A stores instructions for executing the functions of the transceiver unit 710 and the first processing unit 720. Meanwhile, the memory 930 in the computing device 900B stores instructions for executing the functions of the second processing unit 730. Fig.11 The connection method between the computing device clusters shown may be that considering that the data processing method provided in the present application requires a large amount of data storage, the functions implemented by the second processing unit 730 are considered to be executed by the computing device 100B.

[0113] It should be understood that Fig.11 The functions of the computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900.

[0114] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.10 and Fig.11 The connection mode of the computing device cluster is different in that the memory 930 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method.

[0115] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 900 may jointly execute instructions for executing the data processing method.

[0116] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions for executing partial functions of the computing device 900. That is, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more devices in the data processing device 700 or the data processing device 800.

[0117] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0118] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a server of a distributed storage system, and the server is used to access a storage device. The method includes: Determine, according to at least one persistence layer log PLOG to be deleted in the server, first data, where the first data is business data that is first persistently stored in at least one data block to be reclaimed in the storage device and written twice by the server in an append write manner to a target logical block address; The physical address information of the first data is sent to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block and relocate the second data in the at least one data block to be reclaimed to a second data block when performing garbage collection.

2. The method according to claim 1, characterized in that Before sending the physical address information of the first data to the storage device, the method includes: Determine a PLOG list according to the at least one PLOG to be deleted, wherein the PLOG list includes an identifier of each PLOG to be deleted written in the first data and an offset of the first data in each PLOG to be deleted; The PLOG list is converted into a logical block address LBA list, where the LBA list is used to indicate physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and an offset address of the first data in each LBA.

3. The method according to claim 1 or 2, characterized in that: The sending the physical address information of the first data to the storage device includes: The physical address information of the first data is sent to the storage device through a target interface of the storage device.

4. The method according to any one of claims 1 to 3, characterized in that: The storage device is a solid state drive (SSD), and a controller of the SSD is used for relocating the first data to the first data block and relocating the second data to the second data block when performing garbage collection.

5. A data processing method, characterized in that: The method is applied to a storage device of a distributed storage system, the storage device includes a plurality of data blocks, and the method includes: Receive physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server according to at least one persistence layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server twice writing into a target logical block address in an append write manner; Determine at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data; Migrating the first data to a first data block; relocating the second data in the at least one data block to be recycled to the second data block; The at least one data block to be recycled is erased.

6. The method according to claim 5, characterized in that The step of determining at least one data block to be recycled among the plurality of data blocks according to the physical address information of the first data includes: At least one data block to be recycled among the multiple data blocks is determined according to the physical address information of the first data and the amount of garbage data of the multiple data blocks.

7. The method according to claim 5 or 6, characterized in that: The storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block and to relocate the second data to the second data block.

8. A data processing device, characterized in that: include: A first processing unit is configured to determine first data according to at least one persistent layer log PLOG to be deleted in a server of the distributed storage system, wherein the first data is business data that is first persistently stored in at least one data block to be recycled in a storage device and written twice by the server in an append write manner to a target logical block address; A transceiver unit is used to send the physical address information of the first data to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block when performing garbage collection, and to relocate the second data in the at least one data block to be recycled to a second data block.

9. The device according to claim 8, characterized in that Also included is a second processing unit, configured to: Determine a PLOG list according to the at least one PLOG to be deleted, wherein the PLOG list includes an identifier of each PLOG to be deleted written in the first data and an offset of the first data in each PLOG to be deleted; The PLOG list is converted into a logical block address LBA list, where the LBA list is used to indicate physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and an offset address of the first data in each LBA.

10. The device according to claim 8 or 9, characterized in that The transceiver unit is specifically used for: The physical address information of the first data is sent to the storage device through a target interface of the storage device.

11. The device according to any one of claims 8 to 10, characterized in that The storage device is a solid state drive (SSD), and a controller of the SSD is used for relocating the first data to the first data block and relocating the second data to the second data block when performing garbage collection.

12. A data processing device, characterized in that: include: A transceiver unit, configured to receive physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server according to at least one persistence layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server twice writing into a target logical block address in an append write manner; a processing unit, configured to determine at least one data block to be recycled among the plurality of data blocks according to the physical address information of the first data; The processing unit is further used to relocate the first data to a first data block; The processing unit is further configured to relocate the second data in the at least one data block to be recycled to the second data block; The processing unit is further used to erase the at least one data block to be recycled.

13. The device according to claim 12, characterized in that The processing unit is specifically used for: At least one data block to be recycled among the multiple data blocks is determined according to the physical address information of the first data and the amount of garbage data of the multiple data blocks.

14. The device according to claim 12 or 13, characterized in that The storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block and to relocate the second data to the second data block.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to claims 1-4.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to claims 5-7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device performs the method as claimed in any one of claims 1 to 4.

18. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device performs the method as claimed in any one of claims 5 to 7.

19. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computing device, cause the computing device to implement the method of any one of claims 1 to 4.

20. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computing device, cause the computing device to implement the method of any one of claims 5 to 7.

Images