Data processing method and apparatus, and computing device

Abstract

A data processing method and apparatus, and a device, which relate to the technical field of storage. The method comprises: after determining first data on the basis of a persistence layer log (PLOG) to be deleted, a server of a distributed storage system being able to send physical address information of the first data to a storage device, wherein the first data is service data which is written into a target logical block address twice by the server in an append-only manner and is persistently stored the first time in at least one data block to be collected in the storage device. During garbage collection, the storage device can identify the first data from valid data on the basis of the physical address information of the first data, migrate the first data to a first data block, and migrate valid data other than the first data to a second data block. The server of the distributed storage system makes the storage device separately migrate service garbage data to a data block, and during garbage collection, it is unnecessary to perform a data migration operation again, and therefore the overheads of a garbage collection operation can be reduced.

Description

Data processing method, device and computing equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 17, 2023, with application number 202310876406.5 and application name “A method, device and other equipment for garbage recycling”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 29, 2023, with application number 202311610069.1 and application name “A data processing method, device and computing equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data processing method, apparatus, and computing device. Background Art

[0003] Append-only writing refers to a data writing mode characterized by the fact that new data can only be appended to existing data, and existing data cannot be directly modified or deleted. When a distributed storage system writes data using append-only writing, the write operation can only add new data to the end of existing data; overwriting or deleting existing data is not allowed. Distributed storage systems using append-only writing can effectively improve write performance and data reliability. However, because append-only writing does not overwrite or delete existing data, it can lead to a large amount of garbage data in the distributed storage system. If the distributed storage system does not promptly remove garbage data, it will occupy a large amount of storage space, affecting the performance and stability of the distributed storage system.

[0004] Summary of the Invention

[0005] In order to solve the above problems, a data processing method is provided in an embodiment of the present application. During the garbage collection process of the memory, the business garbage data in the valid data stored in the data block is migrated to a new data block, and the remaining valid data is migrated to another new data block. When clearing the business garbage data, there is no need to perform the data migration operation again, which can reduce the overhead of the garbage collection operation.

[0006] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0007] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a server of a distributed storage system, and the server is used to access a storage device. The method includes: determining first data based on at least one persistence layer log (PLOG) to be deleted in the server, wherein the first data is business data that is first persistently stored in at least one data block to be recycled in the storage device by the server in an append-write manner and written twice to a target logical block address; sending physical address information of the first data to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block when performing garbage collection, and to relocate the second data in the at least one data block to be recycled to a second data block.

[0008] In this embodiment, after the distributed storage system server identifies the business junk data based on the PLOG to be deleted, it can send the physical address information of the business junk data to the storage device. When performing garbage collection, the storage device can use this physical address information to identify the business junk data within valid data and migrate the business junk data to a data block. The distributed storage system server instructs the storage device to migrate the business junk data to a separate data block. This eliminates the need for a further data migration operation when clearing the business junk data, thus reducing the overhead of garbage collection operations.

[0009] In one embodiment, before sending the physical address information of the first data to the storage device, the method includes: determining a PLOG list based on the at least one PLOG to be deleted, the PLOG list including each PLOG identifier to be deleted written with the first data and the offset of the first data in each PLOG to be deleted; converting the PLOG list into a logical block address (LBA) list, the LBA list being used to indicate the physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and the offset address of the first data in each LBA.

[0010] In this embodiment, the address information recognizable by the storage device is generally a physical address, so the server of the distributed storage system can convert the PLOG list of logical block addresses including business garbage data into an LBA list of physical addresses including business garbage data, so that the storage device can receive the recognizable physical address and realize the migration of business garbage data into a data block separately.

[0011] In one embodiment, sending the physical address information of the first data to the storage device includes: sending the physical address information of the first data to the storage device through a target interface of the storage device.

[0012] In one embodiment, the storage device is a solid state drive (SSD), and a controller of the SSD is configured to relocate the first data to the first data block and relocate the second data to the second data block when performing garbage collection.

[0013] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a storage device of a distributed storage system, wherein the storage device includes multiple data blocks, and the method includes: receiving physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server based on at least one persistence layer log PLOG to be deleted of the server, wherein the first data is the business data that is first persistently stored in the storage device by the server in an append-write manner and written twice to the target logical block address; determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data; relocating the first data to the first data block; relocating the second data in the at least one data block to be recycled to the second data block; and erasing the at least one data block to be recycled.

[0014] In this embodiment, after receiving the physical address information of the business junk data, the storage device identifies the business junk data within the valid data. When performing garbage collection, the storage device migrates the business junk data to one data block and migrates the valid data other than the business junk data to another data block. This eliminates the need for the storage device to perform data migration operations again when clearing the business junk data, thereby reducing the overhead of garbage collection operations.

[0015] In one embodiment, determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data includes: determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0016] In one embodiment, the storage device is a solid-state drive (SSD), and a controller of the solid-state drive is configured to relocate the first data to the first data block and relocate the second data to the second data block.

[0017] In a third aspect, an embodiment of the present application provides a data processing device, comprising: a first processing unit, for determining first data based on at least one persistence layer log PLOG to be deleted in a server of a distributed storage system, wherein the first data is business data that is first persistently stored in at least one data block to be recycled in a storage device by the server in an append-write manner and written twice to a target logical block address; a transceiver unit, for sending physical address information of the first data to the storage device, wherein the physical address information is used by the storage device to relocate the first data to the first data block when performing garbage collection, and to relocate the second data in the at least one data block to be recycled to the second data block.

[0018] In one embodiment, a second processing unit is further included, which is used to: determine a PLOG list based on the at least one PLOG to be deleted, wherein the PLOG list includes each PLOG identifier to be deleted written with the first data and the offset of the first data in each PLOG to be deleted; convert the PLOG list into a logical block address LBA list, wherein the LBA list is used to indicate the physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and the offset address of the first data in each LBA.

[0019] In one embodiment, the transceiver unit is specifically configured to send the physical address information of the first data to the storage device through a target interface of the storage device.

[0020] In one embodiment, the storage device is a solid-state drive (SSD), and a controller of the SSD is configured to relocate the first data to the first data block and relocate the second data to the second data block when performing garbage collection.

[0021] In a fourth aspect, an embodiment of the present application provides a data processing device, comprising: a transceiver unit for receiving physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server based on at least one persistence layer log PLOG to be deleted by the server, wherein the first data is the business data that is first persistently stored in the storage device by the server in an append-write manner and written twice to the target logical block address; a processing unit for determining at least one data block to be recycled among the multiple data blocks based on the physical address information of the first data; the processing unit for relocating the first data to the first data block; the processing unit for relocating the second data in the at least one data block to be recycled to the second data block; and the processing unit for erasing the at least one data block to be recycled.

[0022] In one embodiment, the processing unit is specifically configured to determine the at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0023] In one embodiment, the storage device is a solid-state drive (SSD), and a controller of the solid-state drive is configured to relocate the first data to the first data block and relocate the second data to the second data block.

[0024] In a fifth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the various possible implementation embodiments of the first aspect.

[0025] In a sixth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the various possible implementation embodiments of the second aspect.

[0026] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes each possible implementation of the first aspect.

[0027] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes each possible implementation of the second aspect.

[0028] In the ninth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, which, when executed by a computing device, enable the computing device to implement various possible implementation embodiments of the first aspect.

[0029] In the tenth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, which, when executed by a computing device, enable the computing device to implement each possible implementation embodiment of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0031] FIG1 is a schematic diagram of the structure of a distributed storage system provided in an embodiment of the present application;

[0032] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0033] FIG3 is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of the architecture of a storage device provided in an embodiment of the present application;

[0035] FIG5 is a flow chart of another data processing method provided in an embodiment of the present application;

[0036] FIG6 is a schematic diagram of a data processing process in a data block provided in an embodiment of the present application;

[0037] FIG7 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0038] FIG8 is a schematic structural diagram of another data processing device provided in an embodiment of the present application;

[0039] FIG9 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0040] FIG10 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0041] FIG11 is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0043] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0044] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0045] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0046] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0047] A distributed storage system stores data across multiple nodes. Each node can be an independent computer, server, or other storage device. Multiple nodes can be connected via a network to form a storage system. Common distributed storage systems include the Hadoop Distributed File System (HDFS), the Google File System (GFS), the open-source HBase (Apache HBase), and the open-source Cassandra (Apache Cassandra).

[0048] Figure 1 is a schematic diagram of the structure of a distributed storage system provided in an embodiment of the present application. As shown in Figure 1, distributed storage system 100 includes a client 110, a server 120, and a storage device 130. The number of clients 110, servers 120, and storage devices 130 is not limited to the number shown in Figure 1.

[0049] The client 110 can be a terminal device such as a laptop, desktop computer, tablet computer, or smartphone. The client 110 can receive data input by the user and forward the data to the server 120 for writing data, or receive instructions input by the user and forward the instructions to the server 120 for reading data. Typically, the client 110 can have an application (APP) installed. The user can log in to the APP account to use the functions of the distributed storage system 100.

[0050] The server 120 can be a server, data center, or other device. The server 120 is responsible for managing and maintaining the data index structure. An index structure is a data structure used to accelerate data retrieval. The index structure records metadata information about the data in the storage device 130. This metadata information includes the file name, file path, data block location, creation time, and so on. During the data writing process, upon receiving the write data, the server 120 is responsible for recording and updating the data metadata information to track the data's status and location. During the data reading process, upon receiving a read request, the server 120 uses the index structure to locate the logical block address of the data based on the index information in the read request, and then reads the data.

[0051] The server 120 is also responsible for actual data storage and processing. When writing data, the server 120 receives the data and writes it to the physical pages of the storage device 130. The server 120 manages and allocates available physical pages, determines the data write location, and writes the data to the storage device 130. When reading data, the server 120 finds the physical location of the stored data based on the logical block address and reads the corresponding data block or page from the storage device 130. The server 120 returns the read data to the client 110.

[0052] Storage device 130 can be an independent memory device or a data storage medium within server 120, typically a hard disk drive (HDD), SSD, memory stick, or other memory device with certain processing capabilities. When writing data, storage device 130 receives a write request from server 120 and writes the data to the corresponding physical page. When reading data, storage device 130 receives a read request from server 120 and quickly accesses physical storage space to transfer the data to server 120 for processing.

[0053] The server 120 writes data to the storage device 130 in an append-write mode, specifically:

[0054] Assume that the server 120 triggers a data write request for the first time. The data write request includes data A and the LBA corresponding to data A. Assume that LBA=0. In response to the data write request, the server 120 creates a log file corresponding to the client 110 and generates a record A in the log file. Record A includes data A and LBA=0. If the server 120 triggers a data write request for the second time, the data write request includes data B and the LBA corresponding to data B. LBA=0. It can be seen that the server 120 modifies the data written to LBA=0 from the original data A to data B. At this time, the server 120 generates a record B in the log file again in response to the second data write request. Record B includes data B and LBA=0.

[0055] The server 120 can pre-set the size of a log file (for example, 100MB). When a log file is full, a new log file will be created. The log file is just an intermediate tool, and the content written in each log file is generally stored in the storage device 130. Therefore, the server 120 can set the data block based on the storage space of the storage device 130 and pre-set the size of the data block (for example, 10MB). When the amount of data written to a log file reaches the size of a data block, a data block is formed, and the server 120 can store the data block in different storage devices 130.

[0056] When the server 120 receives a data write request from the client 110 to write data A at LBA = 0, the log file stores record A, i.e., the mapping relationship between data A and data A and LBA = 0. When the server 120 receives a data write request from the client 110 to write data B at LBA = 0, the log file stores record B, i.e., the mapping relationship between data B and data B and LBA = 0. In this case, the server 120 sets record A as business junk data and record B as business valid data.

[0057] Normally, the storage device 130 can identify its own data as valid data or garbage data. In the process of recycling garbage data, the storage device 130 migrates the valid data in the data block to a new data block, and then clears all the data in the original data block to recycle the garbage data in the data block. However, the storage device 130 cannot identify the business garbage data and business valid data recorded by the server 120, and can only treat the business garbage data and business valid data as valid data and migrate them to the new data block. If the server 120 needs to delete the business garbage data, it needs to instruct the storage device 130 to migrate the business valid data in the new data block to another new data block again, and then clear all the data in the new data block again. It can be seen from this that the overhead of the garbage collection operation of the related art is relatively large.

[0058] In order to solve the defects of the garbage collection operation of the related art, the embodiment of the present application provides a data processing method and server. After the server 120 determines the business garbage data based on the PLOG to be deleted, the server 120 sends the physical address information of the business garbage data to the storage device 130. When the storage device 130 performs garbage collection, it can identify the business garbage data in the valid data based on the physical address information of the business garbage data and migrate the business garbage data to a data block. The server 120 allows the storage device 130 to migrate the business garbage data to a data block separately, so that when clearing the business garbage data, there is no need to perform the data migration operation again, which can reduce the overhead of the storage device 130 performing the garbage collection operation.

[0059] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG2 , the data processing method is executed by the server 120 of the distributed storage system 100, and the specific implementation process is as follows:

[0060] In step S201 , the server 120 determines first data according to at least one persistence layer log PLOG to be deleted of the server 120 .

[0061] The first data is business garbage data, that is, when the server 120 writes data in an append-write mode, the first of the two writes to the target logical block address is persistently stored in at least one data block to be recycled in the storage device. Under normal circumstances, when the server 120 loses data, multiple write requests arrive at the same time, or rewrites after a write failure, the server 120 may allocate the same logical block address to two or more write requests. When the server 120 detects that data is written twice to the same logical block address, it can set the data written for the first time as business garbage data and the data written for the second time as business valid data.

[0062] As shown in Figure 3, the server 120 may include a software portion 121 and a hardware portion 122. The software portion 121 may be divided into a metadata layer 1211 and a persistence layer 1212 based on execution functions. The metadata layer 1211 and the persistence layer 1212 may be executed by a processor in a single server or by processors in different servers. The hardware portion 122 may include N interfaces. N is a positive integer greater than or equal to 1. The N interfaces may be interfaces connected to different storage devices 130 in a single server or interfaces connected to different storage devices 130 in different servers.

[0063] The metadata layer 1211 is used to store and manage data metadata, such as file name, file size, and creation time. When reading data, the metadata layer 1211 provides data location and index information, enabling the server 120 to quickly locate and retrieve data. When writing data, the metadata layer 1211 is responsible for recording and updating data metadata to track the data's status and location.

[0064] The metadata layer 1211 records the metadata information of a data in PLOG. PLOG is an incremental log in a database system, which is used to record modification operations on the database. PLOG records the operations performed by each transaction, such as insert, update, delete, etc. The content recorded by PLOG will be stored in an orderly manner on a persistent medium so that when the system fails or recovers, PLOG can be used to recover data and ensure consistency. In an embodiment of the present application, the metadata layer 1211 stores multiple PLOGs, and each PLOG records an identifier and offset of a data. When the server 120 writes business garbage data to the storage device 130, it can record and update the metadata information of the written data to track the status and location of the written data. After the server 120 completes the write operation, the metadata layer 1211 can record the identifier and offset of the business garbage data in the logical block address to the PLOG to form a PLOG of the business garbage data.

[0065] When performing garbage collection on multiple pieces of service garbage data, metadata layer 1211 can organize the PLOGs corresponding to the multiple pieces of service garbage data into a PLOG list and then forward the PLOG list to persistence layer 1212. A PLOG list is a list of multiple PLOGs, each of which records the identifier and offset of a piece of service garbage data. Metadata layer 1211 organizes the multiple PLOGs into a PLOG list and sends it to persistence layer 1212 to improve garbage collection efficiency on storage device 130.

[0066] In step S202 , the server 120 sends the physical address information of the first data to the storage device 130 .

[0067] The persistence layer 1212 is used to store data on permanent media to ensure data reliability and durability. When the server 120 writes data, the persistence layer 1212 stores the data on the persistent media to ensure that the data is not lost or damaged in abnormal situations such as system shutdown. When the server 120 reads data, the persistence layer 1212 provides an interface for reading and writing data stored on the persistent media, allowing applications to access and manipulate the data.

[0068] In an embodiment of the present application, after receiving the PLOG list, the persistence layer 1212 can obtain the identifier and offset of each piece of business junk data at the logical block address from each PLOG in the PLOG list. The persistence layer 1212 can determine the logical block address of each piece of business junk data based on the identifier and offset at the logical block address. The persistence layer 1212 can also obtain the physical address of each piece of business junk data based on the mapping relationship between the logical block address and the physical address. Because the server 120 cannot overwrite or delete existing data when writing data in an append-write mode, the data written to the same logical block address a second time cannot be located at the physical address corresponding to the logical block address, but rather at the physical address following the physical address corresponding to the logical block address. Therefore, the physical address corresponding to the logical block address is the physical address where the business junk data is stored. In one example, data A and data B are written to the same logical block address. When the memory writes data A, data A is written to the physical address corresponding to the logical block address. When the memory writes data B, data B is written to the physical address following the physical address corresponding to the logical block address.

[0069] After obtaining the physical address of each piece of business junk data, persistence layer 1212 can determine the LBA identifier corresponding to the physical address of each piece of business junk data and the offset within each LBA, and record the LBA identifier and offset within each LBA of each piece of business junk data in the LBA list. Since storage device 130 can only recognize physical addresses, storage device 130 can send the physical address of the business junk data to storage device 130 via the LBA list, allowing storage device 130 to identify the location where the business junk data is stored, thereby accurately locating and processing the business junk data.

[0070] The persistence layer 1212 can cache the physical addresses of each connected storage device 130 , and after obtaining the LBA list, can determine the object to which the LBA list is sent. The persistence layer 1212 can call the interface of the target storage device 130 to send the LBA list to the target storage device 130 .

[0071] In an embodiment of the present application, after the server of the distributed storage system identifies the business junk data based on the PLOG to be deleted, it can send the physical address information of the business junk data to the storage device. When the storage device performs garbage collection, it can identify the business junk data from the valid data based on the physical address information of the business junk data and migrate the business junk data to a data block. The server of the distributed storage system instructs the storage device to migrate the business junk data to a separate data block. This eliminates the need to perform a data migration operation again when clearing the business junk data, thereby reducing the overhead of garbage collection operations.

[0072] FIG4 is a schematic diagram of the architecture of a storage device provided in an embodiment of the present application. As shown in FIG4 , the storage device 130 may include a main controller 131, multiple flash memory chips 132, multiple storage class memory (SCM) chips 133, and dynamic random access memory (DRAM) chips 134.

[0073] The flash memory chip 132 is the main storage medium of the storage device 130 and is used to store data persistently. The flash memory chip 132 uses non-volatile storage technology and can retain data after power failure.

[0074] SCM chip 133 is a new storage technology between traditional DRAM and flash memory. It provides non-volatile storage and lower access latency. It can serve as a cache or primary storage medium, accelerating data read and write speeds.

[0075] DRAM chip 134 is used as a cache for main controller 131 to temporarily store frequently accessed data and metadata. DRAM chip 134 can provide fast read and write speeds to alleviate the speed difference between main memory and flash memory chips and improve overall performance.

[0076] The main controller 131 can cache the received data to the flash memory chip 132, the SCM chip 133 or other memory chips. After receiving a read request, the main controller 131 can read the target data in the target memory chip. The main controller 131 can perform garbage collection on the stored data in each memory chip, as well as other functions. The main controller 131 may include a host interface 1311, a processor 1312 and multiple channel controllers. The host interface 1311 establishes a communication connection with an external device for receiving various instructions and writing data. The channel controller is a bridge between the processor 1312 and each memory chip, responsible for managing the reading, writing and erasing operations of data on the chip, and providing a data transmission interface with the host system.

[0077] After receiving the physical address information of the service garbage data through the host interface 1311, the processor 1312 identifies the service garbage data in the valid data. When performing garbage collection, the processor 1312 migrates the service garbage data to one data block and migrates the valid data other than the service garbage data to another data block. This eliminates the need for the processor 1312 to perform data migration again when clearing the service garbage data, thereby reducing the overhead of the garbage collection operation.

[0078] FIG5 is a flow chart of another data processing method provided in an embodiment of the present application. As shown in FIG5 , the data processing method is executed by the processor 1312 in the storage device 130 of the distributed storage system 100. The specific implementation process is as follows:

[0079] In step S501 , the processor 1312 receives physical address information of first data sent by the server 120 .

[0080] The first data refers to business junk data. In the storage device 130, business junk data refers to data in the memory that is no longer needed and can be safely deleted. The physical address information is an LBA list, which includes the LBA identifier of each business junk data and the offset in each LBA.

[0081] In step S502 , the processor 1312 determines at least one data block to be recycled from among the plurality of data blocks according to the physical address information of the first data.

[0082] In step S503 , the processor 1312 moves the first data to the first data block.

[0083] In step S504 , the processor 1312 relocates the second data in the at least one data block to be recycled to the second data block.

[0084] In step S505 , the processor 1312 erases at least one data block to be recycled.

[0085] Normally, when the processor 1312 performs garbage collection, it can identify the valid data stored in each data block, migrate the valid data in the data block to a new data block, and then clear all data in the original data block to recycle the garbage data in the data block. In an embodiment of the present application, after the processor 1312 receives the LBA list, it can obtain each physical address from the LBA list. Based on each physical address, the processor 1312 can determine the data block where each physical address is located as the target data block. When the processor 1312 performs garbage collection, the data stored in the physical address of each target data block is used as business garbage data, and the business garbage data is migrated to a new data block, and other valid data in each target data block except the business garbage data is migrated to another new data block. Finally, the processor 1312 erases each target data block to release the storage space of each target data block.

[0086] The lifecycle of business garbage data may change, turning it into garbage data. During the next garbage collection cycle, because the lifecycles of business garbage data in data blocks vary, the percentage of garbage data in each data block will also vary. Therefore, when processor 1312 performs garbage collection, it prioritizes data blocks with a high percentage of garbage data for garbage collection. This can reduce the load on the memory to migrate valid data, thereby improving garbage collection efficiency and extending the lifespan of the memory.

[0087] Processor 1312 stores multiple data blocks. Each data block may contain different information, including the type of data stored, the proportion of different data types, and data attributes. Processor 1312 can configure different garbage collection strategies for different data blocks to improve the efficiency of memory garbage collection. Garbage collection strategies include delayed garbage collection, active garbage collection, mixed garbage collection, prefetch garbage collection, and block garbage collection.

[0088] The delayed garbage collection strategy refers to a strategy in which the memory performs garbage collection during idle time, which does not affect the user's read and write operations. The active garbage collection strategy refers to a strategy in which the memory performs garbage collection immediately when data is written, which can improve the performance of the memory. The hybrid garbage collection strategy combines the advantages of the delayed garbage collection and active garbage collection strategies, and can dynamically adjust the garbage collection strategy according to the load of the memory. The prefetch garbage collection strategy refers to a strategy in which the memory prefetches garbage data that may be generated during idle time, which can reduce the time of garbage collection and the impact on users. The block garbage collection strategy refers to a strategy in which valid data from multiple data blocks is organized into one data block, which can reduce the amount of garbage data and improve the performance of the memory.

[0089] Optionally, the processor 1312 may perform garbage collection operations in advance before receiving the LBA list. When performing garbage collection, the processor 1312 may identify valid data in each target data block in advance and prepare to migrate the valid data to a new data block. After receiving the LBA list, the processor 1312 may directly obtain business garbage data from the valid data based on the physical address in the LBA list, migrate the business garbage data to a new data block, and migrate all valid data other than the business garbage data in each target data block to another new data block, thereby accelerating the speed of garbage collection performed by the processor 1312 and shortening the latency of data processing by the storage device 130.

[0090] As shown in Figure 6, at time T1, after the processor 1312 receives the physical address information of 14 business garbage data, it can identify the target data blocks including the physical address information of the 14 business garbage data based on the physical address information of the 14 business garbage data, which are data block 1, data block 2, data block 3, data block 4, data block 5 and data block 6.

[0091] At time T2, when processor 1312 performs garbage collection, it can migrate the business garbage data from the six data blocks based on the physical address information of the 14 business garbage data, and migrate the 14 business garbage data to data block 7 (which can be migrated to multiple data blocks). Then, processor 1312 migrates the valid data in the six data blocks to data block 8 (which can be migrated to multiple data blocks). Finally, processor 1312 erases data blocks 1, data blocks 2, data blocks 3, data blocks 4, data blocks 5, and data blocks 6 to free up storage space in the six target data blocks.

[0092] At time T3, some of the 14 service junk data in data block 7 may be converted into junk data. Some of the 11 valid data in data block 8 may be converted into service junk data and some into junk data.

[0093] In the embodiment of the present application, after receiving the physical address information of the service junk data, the processor 1312 identifies the service junk data in the valid data. When performing garbage collection, the processor 1312 migrates the service junk data to one data block and migrates the valid data other than the service junk data to another data block. This eliminates the need for the processor 1312 to perform data migration again when clearing the service junk data, thereby reducing the overhead of the garbage collection operation.

[0094] FIG7 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG7 , the data processing device 700 can be divided into a transceiver unit 710, a first processing unit 720, and a second processing unit 730 according to the execution function. The functions performed by each unit of the data processing device 700 are as follows:

[0095] The first processing unit 720 is used to determine the first data based on at least one persistent layer log PLOG to be deleted in the server of the distributed storage system. The first data is business data that is first persistently stored in at least one data block to be recycled in the storage device by writing it twice in the target logical block address in an append-write manner. The transceiver unit 710 is used to send the physical address information of the first data to the storage device. The physical address information is used by the storage device to relocate the first data to the first data block when performing garbage collection, and to relocate the second data in at least one data block to be recycled to the second data block.

[0096] In one embodiment, the system further includes a second processing unit 730 configured to determine a PLOG list based on at least one PLOG to be deleted. The PLOG list includes an identifier of each PLOG to be deleted where the first data is written and an offset of the first data within each PLOG to be deleted. The second processing unit 730 is further configured to convert the PLOG list into a logical block address (LBA) list. The LBA list indicates physical address information of the first data. The LBA list includes an identifier of each LBA storing the first data and an offset address of the first data within each LBA.

[0097] In one embodiment, the transceiver unit 710 is specifically configured to send the physical address information of the first data to the storage device through a target interface of the storage device.

[0098] In one embodiment, the storage device is a solid-state drive (SSD), and a controller of the SSD is configured to relocate the first data to a first data block and relocate the second data to a second data block when performing garbage collection.

[0099] FIG8 is a schematic diagram of the structure of another data processing device provided in an embodiment of the present application. As shown in FIG8 , the data processing device 800 can be divided into a transceiver unit 810 and a processing unit 820 according to the execution function. The functions performed by each unit of the data processing device 800 are as follows:

[0100] The transceiver unit 810 is used to receive the physical address information of the first data sent by the server of the distributed storage system. The physical address information is determined by the server based on at least one persistence layer log PLOG to be deleted by the server. The first data is the business data that is first persistently stored in the storage device by the server twice in the target logical block address in an append-write manner. The processing unit 820 is used to determine at least one data block to be recycled among multiple data blocks based on the physical address information of the first data. The processing unit 820 is used to relocate the first data to the first data block. The processing unit 820 is used to relocate the second data in at least one data block to be recycled to the second data block. The processing unit 820 is used to erase at least one data block to be recycled.

[0101] In one embodiment, the processing unit 820 is specifically configured to determine at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data and the amount of garbage data in the multiple data blocks.

[0102] In one embodiment, the storage device is a solid-state drive (SSD), and a controller of the solid-state drive is configured to relocate the first data to a first data block and relocate the second data to a second data block.

[0103] Figure 9 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 9, computing device 900 includes a bus 910, a processor 920, a memory 930, and a communication interface 940. Processor 920, memory 930, and communication interface 940 communicate with each other via bus 910. Computing device 900 can be a server, a computer, a storage system, etc. It should be understood that this application does not limit the number of processors and memories in computing device 900.

[0104] Bus 910 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG9 shows a single bus line, but this does not imply a single bus or type of bus. Bus 910 may include a path for transmitting information between various components of computing device 900 (e.g., processor 920, memory 930, and communication interface 940).

[0105] The processor 920 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0106] The memory 930 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or an SSD.

[0107] The memory 930 stores executable program code. The processor 920 executes the executable program code to implement the functions of the transceiver unit 710, the first processing unit 720, and the second processing unit 730 of the data processing device 700, thereby implementing the data processing method. In other words, the memory 930 stores instructions for executing the data processing method.

[0108] Alternatively, the memory 930 stores executable code, and the processor 920 executes the executable code to implement the functions of the transceiver unit 810 and the processing unit 820 of the aforementioned data processing device 800, thereby implementing the data processing method. In other words, the memory 930 stores instructions for executing the data processing method. Figure 9 only exemplifies that the memory 930 stores program code that implements the functions of the aforementioned transceiver unit 710, the first processing unit 720, and the second processing unit 730.

[0109] The communication interface 940 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0110] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0111] As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 930 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method.

[0112] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the data processing method.

[0113] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data processing device 700 or the data processing device 800. In other words, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more modules in the transceiver unit 710, the first processing unit 720, and the second processing unit 730, or implement the functions of one or more modules in the transceiver unit 810 and the processing unit 820.

[0114] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN), a local area network (LAN), or the like. Figure 11 illustrates one possible implementation. As shown in Figure 11 , two computing devices, computing device 900A and computing device 900B, are connected via a network. Specifically, the connection to the network is achieved via a communication interface within each computing device. In this type of possible implementation, memory 930 within computing device 900A stores instructions for executing the functions of some modules within data processing apparatus 700 or data processing apparatus 800. Figure 11 uses the example of memory 930 within computing device 900A storing instructions for executing the functions of transceiver unit 710 and first processing unit 720. Simultaneously, memory 930 within computing device 900B stores instructions for executing the functions of second processing unit 730. The connection method between the computing device clusters shown in Figure 11 may be based on the consideration that the data processing method provided herein requires large amounts of data storage, and therefore considers the functions performed by second processing unit 730 to be performed by computing device 100B.

[0115] It should be understood that the functionality of the computing device 900A shown in FIG11 may also be implemented by multiple computing devices 900. Similarly, the functionality of the computing device 900B may also be implemented by multiple computing devices 900.

[0116] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 10 and 11. However, the memory 930 in one or more computing devices 900 in this computing device cluster can store the same instructions for executing the data processing method.

[0117] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the data processing method.

[0118] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions for executing partial functions of the computing device 900. In other words, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more devices in the data processing device 700 or the data processing device 800.

[0119] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0120] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a server of a distributed storage system, and the server is used to access a storage device. The method includes: Determine, according to at least one persistence layer log PLOG to be deleted in the server, first data, where the first data is business data that is first persistently stored in at least one data block to be reclaimed in the storage device and written twice by the server in an append write manner to a target logical block address; The physical address information of the first data is sent to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block and relocate the second data in the at least one data block to be reclaimed to a second data block when performing garbage collection.

2. The method according to claim 1, characterized in that Before sending the physical address information of the first data to the storage device, the method includes: Determine a PLOG list according to the at least one PLOG to be deleted, wherein the PLOG list includes an identifier of each PLOG to be deleted written in the first data and an offset of the first data in each PLOG to be deleted; The PLOG list is converted into a logical block address LBA list, where the LBA list is used to indicate physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and an offset address of the first data in each LBA.

3. The method according to claim 1 or 2, characterized in that: The sending the physical address information of the first data to the storage device includes: The physical address information of the first data is sent to the storage device through a target interface of the storage device.

4. The method according to any one of claims 1 to 3, characterized in that: The storage device is a solid state drive (SSD), and a controller of the SSD is used for relocating the first data to the first data block and relocating the second data to the second data block when performing garbage collection.

5. A data processing method, characterized in that: The method is applied to a storage device of a distributed storage system, the storage device includes a plurality of data blocks, and the method includes: Receive physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server according to at least one persistence layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server twice writing into a target logical block address in an append write manner; Determine at least one data block to be recycled among the multiple data blocks according to the physical address information of the first data; Migrating the first data to a first data block; relocating the second data in the at least one data block to be recycled to the second data block; The at least one data block to be recycled is erased.

6. The method according to claim 5, characterized in that The step of determining at least one data block to be recycled among the plurality of data blocks according to the physical address information of the first data includes: At least one data block to be recycled among the multiple data blocks is determined according to the physical address information of the first data and the amount of garbage data of the multiple data blocks.

7. The method according to claim 5 or 6, characterized in that: The storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block and to relocate the second data to the second data block.

8. A data processing device, characterized in that: include: A first processing unit is configured to determine first data according to at least one persistent layer log PLOG to be deleted in a server of the distributed storage system, wherein the first data is business data that is first persistently stored in at least one data block to be reclaimed in a storage device and written twice by the server in an append write manner to a target logical block address; A transceiver unit is used to send the physical address information of the first data to the storage device, and the physical address information is used by the storage device to relocate the first data to a first data block when performing garbage collection, and to relocate the second data in the at least one data block to be recycled to a second data block.

9. The device according to claim 8, characterized in that Also included is a second processing unit, configured to: Determine a PLOG list according to the at least one PLOG to be deleted, wherein the PLOG list includes an identifier of each PLOG to be deleted written in the first data and an offset of the first data in each PLOG to be deleted; The PLOG list is converted into a logical block address LBA list, where the LBA list is used to indicate physical address information of the first data, wherein the LBA list includes each LBA identifier storing the first data and an offset address of the first data in each LBA.

10. The device according to claim 8 or 9, characterized in that The transceiver unit is specifically used for: The physical address information of the first data is sent to the storage device through a target interface of the storage device.

11. The device according to any one of claims 8 to 10, characterized in that The storage device is a solid state drive (SSD), and a controller of the SSD is used for relocating the first data to the first data block and relocating the second data to the second data block when performing garbage collection.

12. A data processing device, characterized in that: include: A transceiver unit, configured to receive physical address information of first data sent by a server of the distributed storage system, wherein the physical address information is determined by the server according to at least one persistence layer log PLOG to be deleted of the server, wherein the first data is business data that is first persistently stored in the storage device by the server twice writing into a target logical block address in an append write manner; a processing unit, configured to determine at least one data block to be recycled among the plurality of data blocks according to the physical address information of the first data; The processing unit is further used to relocate the first data to a first data block; The processing unit is further configured to relocate the second data in the at least one data block to be recycled to the second data block; The processing unit is further used to erase the at least one data block to be recycled.

13. The device according to claim 12, characterized in that The processing unit is specifically used for: At least one data block to be recycled among the multiple data blocks is determined according to the physical address information of the first data and the amount of garbage data of the multiple data blocks.

14. The device according to claim 12 or 13, characterized in that The storage device is a solid state drive (SSD), and a controller of the solid state drive is used to relocate the first data to the first data block and to relocate the second data to the second data block.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to claims 1-4.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to claims 5-7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device performs the method as claimed in any one of claims 1 to 4.

18. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device performs the method as claimed in any one of claims 5 to 7.

19. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computing device, cause the computing device to implement the method of any one of claims 1 to 4.

20. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computing device, cause the computing device to implement the method of any one of claims 5 to 7.

Images